That should be fine. Usually when people refrigerate starters they let them sit for about 3 days anyways. You'll want to take the yeast out of the fridge at the beginning of your brew day to let it come up to about room temperature. Just before pitching, decant the liquid and pitch the yeast.
I don't think you need to do this, but if you're worried about your yeast being fresh, you could take it out of the fridge the night before your brew day and let it come up to room temperature. Then at the beginning of your brew day, decant the beer and add some 1.040 wort to get the yeast going. Then pitch that into the beer. You won't be growing many cells, but they'll be getting "woken" up.
