The numbering in directories seems to have altered slightly , but I think we can definitely say no 110 is the one marked so in the map. Ward st is on the far left. To the left of no 110 are licensed premises, which was a beerhouse. In the 1888 directory it is listed it as 108, and in some later directories as 106. It disappears from Kellys between 1921 & 1932.
In addition to the earlier reason given for him not living on the premises there is the possibility that the owners of the pub lived there, but employed a manager to run it. If he was next door then that would be quite easy. what was the name of your grandfather. He might be listed in other directories