By no means do I consider this conclusive (especially considering the many other factors that went into this), but the one time I pitched onto a yeast cake, i could swear the beer had a strange band aid, burnt rubber sort of flavor and smell to it.
I say "other factors" because that particular beer was an experimental American Wheat that I did spur of the moment using leftover ingredients from a number of other kits.
In other words, the off flavor could have been from something else.
EDIT: I should also mention everyone else said they liked it, so it could also have just been me. I'll probably pitch onto a yeast cake again eventually, just to see if I get similar results. But for now, washing yeast and starters all the way.