Its not about getting adverse flavors so much as not getting the character from the yeast that you want. The yeast imparts a lot of its character during the aerobic reproductive phase and pitching onto a full cake means the yeast will not produce a good portion of the esters that we treasure.
I agree with you on planning my brewing on the available yeast though. I love harvesting yeast and having copious quantities of slurry to pitch when I do a batch. This is also another benefit of harvesting and cleansing though in that you get multiple containers of pitchable yeast from a single cake.