Reading a book on fishing folk of the East Coast of Scotland recently it is said that the name derives from 'Guernsey' (presumably where the style of garment originated and probably also without a zip) as mentioned above. It was said that fishermen who had drowned could be recognised by the patterns on their Ganzey. I have certainly heard the term used in Brum but really cannot recall when or by whom.
Click on this - A Google page with quite a bit of reading matter on the subject.
A 'gansey' is sweater made from wool still containing its natural oils; and knitted on four needles.....a 'fishermans' gansey' etc.