I looked around for info on this for a long time and couldn't find any clear info.
One group of people say there is no problem with doing what you are doing, that it creates no problems whatsoever, and keeps things simpler.
One group says that you need to steep in 1.5-2 qts per pound of steeping grains to avoid tannin extraction. Tannins cause an astringent, dry/mouth puckering flavor in your beer and are extracted from grain husk when your steeping water is too hot or the pH is off. This group of people feels that if you have too much water for a small amount of grains, you won't affect the pH of the water enough to avoid tannin extraction.
Personally I steep in a small volume in a side pot, and use the 30 min or so to start heating up my actual 4 gallon brew kettle. Saves a little time because I don't have to hold my brew kettle at 155 for thirty minutes and then start bringing that whole mass up to boil.