It's tricky, I'm feeling my way with this family too! I have a working tree for them, but a lot is still unconfirmed. I think there could be 14 siblings of which my gt gt grandfather Matthew may be one, but still haven't absolutely proved it.
Samuel was born c1836, so can't be your Annie's brother. However he did have a brother called Thomas, who could be the grandfather of your Annie. This Thomas was married to a Mary Ann, but I haven't found them after 1861 yet. There are also a couple of other brothers I have yet to trace, who might be her grandfather.
Samuel's parents were called John and Ann, but I don't have definite ages/places of birth for them. We cannot find the family at all in 1841, but Gypsies/travellers weren't recorded on that census so that's probably why.