After ice bath wort chilling. Darker "sludge" question

In my northern brewer extract kits, it says not to transfer any "sludge" from the bottom of your brew pot after chilling your wort. I don't ever have any "sludge" per say.... its just a darker colored wort mixture with pieces of my hop pellets in it (this is just the extract and hops settling after lowering to a different temperature) . All of my beer has turned out pretty damn good, but my question is... should I be leaving the last 4-5 ounces of darker wort in the bottom of the pot, and not transfering it to my primary? Is it going to make that fine line of good vs great beer?

Thanks

I just transfer it all, maybe strain the larger chunks out only to make transferring easier later on, all these "pieces" are good yeast food and in my opinion make the beer more complex.

You can dump it all in if you want, or strain it, or siphon it and leave the break material in the brew pot. It will all settle to the bottom of the primary in time and you will leave it behind when transfer to bottle. The only time you might want to consider leaving the break material in the pot is if you are making a very light beer, like a pilsner of light lager.