Annabel, I have found your Joseph Prior in 1871 living in Baldwin Street, Harborne:
Joseph Prior, head, married, 25, Pattern Maker, Inniskillen.
Mary A Prior, wife, married, 23, Welland Worcs.
Joseph A Prior, son, 1 Welland Worcs.
William D Prior, son, 1 month, Smethwick Staffs
In 1881 they are living at 141 Crocketts Road, Handsworth
In 1891 living at 46 Wright Road, Saltley (civil parish of Aston) with his family.
Joseph Prior, head, married, 44, Engineers Pattern Maker, Ireland.
Mary A Prior, wife, married, 42, Welland Worcs.
William D Prior, son, unmarried, 20, Toolmaker, (Smethwick?) Staffs.
Emma G Prior, daughter, unmarried, 15, Scholar, Handsworth Staffs.
Mary A Prior, daughter, 11, Scholar, Handsworth Staffs.
Catherine B Prior, daughter, 6, Scholar, Handsworth Staffs.
Joseph H Prior, son, 4, Saltley Warwicks.
William Cox, nephew, 10, Scholar, Castlemorton Worcs.
Hope this helps

Judy