You shouldn't, a lot of it will settle out. Some brewers think that added mineral content may even help fermentation in some capacity. Either way, shouldn't be a problem
What kind of kettle finings are you adding and how much?
Pay attention to the kettle finings quantity; more is not better; more just creates more and fluffier break material and "floating brains" if you go too crazy.
you can put all of that in your fermentor and it won't make hurt your beer at all. had you used a plate chiller all of that would have been in the fermentor since the break would have happened in the chiller. commercial breweries all use some kind of plate chiller that puts the break material in the fermentor.
I have a counterflow chiller, so I have no way to filter out the cold break as it goes into the fermenter, or at least most of it. It's ok, and the break material does compact down with the trub and the beer is very clear coming out of the fermenter into the keg.