Yeah, should be just fine. I've left yeast cakes in the fridge for a week or more before pitching, and still had violent takeoff! Just make sure you take the starter out of the fridge about 4 hours before you pitch, in order to bring it up to room temp.
You might also consider tempering the starter by adding a bit of your freshly cooled wort to the starter flask, giving it a good shake, and allowing the trub to settle before pitching the top of the slurry. That way you've re-roused the yeast, as well as brought the temperature up closer to what they'll be swimming in when they hit the wort.
