I know this is a bit late but it might save someone else a search.
My great grandfather, James TAYLOR, was told that he was found on a doorstep (we will never know whether he was told the truth about this) and he was brought up in the Asylum for the Infant Poor where the staff gave him the name "James TAYLOR". As far as I can tell, he was admitted to the institution in 1835.
He told my late grandmother that when he was approaching the age of ten years, they asked him what sort job he would like to be trained for and he replied "a tailor". (I don't know how much choice he had in this!). Accordingly he was trained in tailoring and was a self-employed tailor for the rest of his life. (He died in 1901).
I'm afraid that there is very little information available about the institution and none of the minute books or registers have survived. William Hutton's 1836 "History of Birmingham" contains the following brief note:-
"The Asylum for the Infant Poor, established in Summer Lane in 1797, is conducted by a committee of guardians and overseers. The manufacture of pins, straw-plait, lace, &c., is carried on for the purpose of employing the children, whose labour produces a profit to the parish. There is a bath, garden, play-ground, school, and chapel connected with this institution. There are usually from two hundred to two hundred and fifty children in this parish family."
The 1841 Census contains the usual list of staff and inmates - HO107/1141 Folio 66 - but that is about your lot! I have yet to find a sketch of the place and it appears that it closed when the new Birmingham Workhouse opened in 1852.