I always figured that it had more to do with clarity and haze than taste.
Removing at least the hot break will get you clearer beer out the end.
But the trub contains nutrients that the yeast use, don't forget. So removing it or leaving it will have a (minor) impact on flavor.
Me? I don't bother with removing the break. Too much work and it involves sanitizing extra pieces of equipment for me since I don't have a bottom drain on my pot.
Do it if you want, even if it's just for S&Gs, but I wouldn't worry about it.