If you dump the trub into the fermenter, then that trub can impart off flavors.
On the other hand, the trub also acts as a yeast nutrient, which will increase the yeast cell count and make the yeasts happier so they produce less off flavors. In other words, it probably doesn't make any noticeable difference.
If you really want to reduce the trub in the fermenter, you can whirlpool after cooling, and siphon into the fermenter instead of pouring. This will leave more trub in the kettle.