Jane
Have you considered Ann d/o Richard and Ann as John James mother? In the 1861 census Ann is 24 and single and living with Richard and Ann as is John age 2 G/S. If John was illeg it would seem to make sense he was living with his mother and grand parents. Do you have info that would rule Ann out as a possible Mom. I had no luck finding Ann after 1861.
Carol