For all my beers, I wait until signs of fermentation is complete (yeast has dropped, no air lock activity, etc.) This usually takes about 5-7 days for an ale and 10-14 days for a lager. I then take a gravity reading and it is usually where it is needed to be. If I am doing a d-rest (lagers or some english strains), I will do so at this time. The beer only needs a few days after fermentation to "clean up" after itself. Once that is done, the beer will only degrade quicker (oxidation, yeast dieing, etc.) If you are drinking the beer fast, it won't be a problem, but if you are going to store it for a while it could be. Keep in mind, this works for me because I pitch the correct amount of healthy yeast and ferment in a temperature controlled freezer. If you don't control temp or have bad yeast health, then the beer may benefit from extra conditioning.