It should be totally fine. What you could do is put it somewhere cool for a couple days, decant it a couple days ahead of your brew and add a liter of wort back to it. Get it rolling again. I have starters around all the time for 5-10 days.
Often, when I brew I will actually start a new starter, from my starter before pitching the yeast in my beer. Make a nice starter, get 2 liters of new wort cooled, decant a bit of yeast into the new starter, pitch the rest into my brew. Let that starter go for a week or two until I want to use it, decant, add a liter of wort a couple days out, and it is ready to roll.
I have also left starters in the fridge for a couple weeks when something came up.
Never had a problem doing this.