Let's assume you bottle, since cold crashing for kegging is a given either before or after transferring it to the keg.
Chilling your fermentation vessel causes thermal contraction of the liquid beer and gas-filled head space. The negative pressure pulls in air, including the dreaded oxygen. Oxygen causes oxidation of various beer components, which is bad for beer flavor, generally speaking.
For this same reason, it's better to NOT use a secondary vessel. The beer picks up oxygen during the transfer and even more from the headspace in the secondary vessel.
After fermentation completes, it does not need to stay refrigerated. Leave it at room temp for as much time as you feel is needed to let the yeast drop (days, weeks, months).
Even without clearing in the fermenter, it will clear in the bottle or keg regardless, so all you're really doing is reducing a little sediment in the package.
Skipping secondary is something everyone who's followed kit instructions needs to unlearn. Anything that increases oxygen exposure post-fermentation is a bad thing. Same goes for cold-crashing without a system or device to prevent air ingress.
Cheers