Yes, it's fine. Like you said, just be careful when racking out of the fermenter.
Cold crashing (if you can) helps drop the debris too
"Brewers make wort, yeast make beer."
"Brewing beer is neither complicated nor expensive. It's the responsibility of the brewer to make it as complicated and expensive as their spouse & budget will allow."