I just find cleaning them is easier if I don't wait between batches. I HAVE done what you're asking about though, and not had any problems. But usually I open the keg, see the remainder of the yeast/sediment at the bottom and think..."Meh, may as well just clean it now" and it usually takes very little time or effort. I just rinse it down with hot water, dump the sediment, put another gallon of PBW mix in, shake it a few times, charge it, run it through the tap, rinse, put a gallon of starsan in, charge it, run it through the tap and it's clean sanitized and full. It probably only takes me a bit longer then typing this post actually took
First Brew was thanksgiving 2011, I'm at 103 batches and counting (as of 11/11/2014), and ran out of room in my signature to list them all.