No, it's fine to leave all that stuff in there.
In time, the yeast will settle out to the bottom of the carboy, and eventually the beer will clarify and all the sediment will be on the bottom, too. So when you rack the beer off to the bottling bucket (assuming you're bottling), you can just...