Looking at the 1950s map certainly Nos. 5 and 3 were quite large houses.
The 1834 map shows the retreat and a few houses, with no houses on Sutton Street whatsoever. I wouldn't be surprised if these had survived until the 1950s, looking at the dilapidated condition of the houses in the pictures...