Are you saying that one son was on the vessel with their mother, as well as the two girls? From what you wrote before, I thought only Thomas came to Australia. If so, that leaves two sons in the IOM, one of whom you know was Andrew.
If Thomas had a tattoo which included "brother James", he looks like a good bet, especially with a possible father James. What was his age given in the other censuses? If his age differs from the one given in 1881, that would explain how he could have been born in Ireland when a possible sister was bapt earlier in the IOM. Or hers could have been a late bapt?
Was Sarah bapt 1826, daughter of Michael LYNCHY and Sarah Mullen, one of the daughters who travelled with Sarah, and does that age correspond to Australian records?
It would be interesting to know who was looking after James in the IOM, if he was the third son b.c.1827.