Just to put your mind at rest... My 'normal' process for starters has them going into the fridge ~24-30 hours before pitching time. I let them stay in the fridge until about 4-8 hours before I'll be pitching (I transport them to my brew buddy's in a cooler). I take the flask out of the cooler and put it onto the dining room table to start to warm up. About 10-20 minutes before I'll be pitching the yeast, I decant most of the spent starter and get the yeast into suspension. I make sure the stirbar isn't going to end up in the fermenter too. After I've oxygenated the beer wort, I pour the slurry into the fermenter. It then goes into the basement for the duration.
I typically get active fermenation sign in under 12 hours this way. I do try to get the yeast slurry within about 10-15F of the chilled wort. I also try to get my wort to the low end, or below that, for the yeast fermenting temperature. I've had it at 50-55F before, for a yeast that likes it in the low 60's. No issue with that, since it fermented very well. It ended up in about the middle of it's temperature range, maybe a little lower.
If you want to see how we (my brew-buddy and I) do things, let me know. I'll be brewing with him probably one more time before I move, and [hopefully] can brew at home again. Not sure where I'll be moving to just yet. We currently brew at his house in Sudbury.