Yeast is alive thing, it reproduces. Thing is, less cells you have the more time yeast spends reproducing instead of fermenting, which may lead to off flavors or, in worst case scenario, contamination if another microbe manages to take a hold before yeast gets strong enough.