Lawence is a rare christian name on the IOM. I ran the name on the 1901 census and only came up with 24 people. Of these 7 were born off the Island. Of the remaining 17 two wre Quilliams one Lawrence Quilliam 26 b. Douglas now married to Eleanor and the other a three year old in Arbory. I think that it is safe to say, despite the age variations, that he is the one in the 1881 and 1891 census.
I think that you have correctly identified Louisa in 1881.
James's age of 26 looks like an an aberration.
John James Quilliam is on the IGI christened on December 22 1872 in Braddan, son of James and Elizabeth. He seems to have been omitted from the 1881 census.
What is surprising is that Catherine A, James and Emily from the 1881 census are neither on the IGI, Brian's burials nor any subsequent census. Possibly they were in foster homes or adopted in England. I had understood that adoptions on the IOM wer discouraged.