Well I started looking on the 1841, and no numbers but the Cadbury family were living in Bull Street. Edcartermo has proved that they still had a foothold in Bull Street in 1869, but in 1851 most were in Islington Row and in 1861 Wheeleys Lane.
My gut feeling is that the garden might have been Cadbury's in 1841 - but that is probably guided by the fact that they were non-conformists and non-conformists liked classical shapes, etc, as they were deemed less 'showy'. I might be way out here, but I would like to think that it was a Cadbury garden! Now back to earth, dinner to cook!