If you put the starter in the fridge...well, the point of putting the starter in the fridge is to compact the yeast on the bottom so you can poor off the wort (and leave the yeast). If you left the 'stuff' (which, I'm sorry to say, was the yeast cake...) than you may want to think about pitching some dry yeast. The fermentation still may occur...but you really, really under pitched. I usually just take my starter off the stirplate when I start my brew. By the time I'm ready to pitch the yeast has settled out enough I can decant most of the wort. Some just do as you stated in the beginning of your post....just stir and pour in the whole thing.