don't forget until the advent of cheap communication with England and good roads there was little brick used on the Island - one architectural historian says that few houses built before 1850 would use stone quarried or lifted from fields from further than a quarter of a mile away.
Many older Manx houses are effectively large drystone walls rendered to keep damp out - my own Peel flat is in a c.1900 3 storey house that from outside appears to be brick built but this is merely a curtain wall attached to a 18inch thick rubble filled dual wall built from small irregular pieces of peel sandstone from the quarry about 100 yards away. The men responsible for this highly skilled occupation would be masons
The Brick was brought over from North wales as ballast in the returning lead ore boats. Douglas having easier access to Steam boats used brick a little more than other other towns