I assume by "cake" you mean krausen?
You are 100% fine. If you let your starter go for too long, the yeast will begin true fermentation (anaerobic respiration, where they make alcohol) and that doesn't do your starter any good. The goal of a starter is for the yeast to make babies not alcohol! I personally think it's best to wait 24 - 48 hours then cold crash your starter, then you can decant off all that nasty oxidized starter wort, allow the slurry to warm to room temp, put in a small amount of LME/DME wort (like 500mL) give it a shake and put it back on the stir plate for about 4 hours before pitching. think of it like the wyeast smack packs, it gets your yeast all woken up and ready to go just prior to pitching, and eliminates the chance of off flavors from starter wort.