One interesting possibility I've come across while looking at Edward's children is that one of his sons, Alfred, may have ended up living in Kenilworth (1911).
If it is him he was listed as a billiard scorer at the Grand Hotel (1871), then a barman (1881) and finally a PH manager (1891 & 1901). Also in 1901 and 1911 he is listed with a Caroline Oliver Morton (nee Horsfall) and in 1891 as a servant of her sister.
Their family came from Kenilworth and I've been trying to see if there is connection between them and Esther Oliver but nothing so far, and as I'm not really sure if it is the correct Alfred (he doesn't marry) it may just be one of those strange coincidences.
One other thing is that when Caroline and her sister die the man listed in their probate is a John Wilkinson.