Hot break and cold break do contribute to off flavors in beer, but they would be lower on your list to go after compared to sanitation issues, etc. Many homebrewers never worry about them at all.
It will improve your beers if you eliminate them but, as I mentioned, there are usually a lot...