I've been no-chilling since the beginning of this year. You have to adjust your hop schedule, but that's the biggest issue and it's pretty easy to overcome with a little experimentation (and who doesn't love a little brewing experimentation?). For malt-forward beers, it's just a matter of accounting for a little bit extra of a bittering contribution. For hop-forward beers, you also need to keep in mind that the wort will stay above 180 for much longer than with a quick chill. That means you'll want to add your late kettle hops later, sometimes even after flameout but while the temp is still about 180. For your whirlpool/hopstand hops, you wait until the wort temp comes down below 180 (no additional risk of infection here, since the temp is still well above flash-pasteurization temperature).
As far as clarity goes, cold break still forms, it just forms more slowly. I cold crash my beers, which I'm sure also helps prevent chill haze, but I can't say that I've had any issues with chill haze in my cold crashed beers vs. the beers I made using my IC.
Contamination issues are easy to mitigate. If you ferment in buckets, you can transfer wort that is at boiling temps without hurting the integrity of the plastic - #2 food grade buckets are rated to 250F. If you use glass, that's not an option, but you can seal the wort in your kettle overnight with some aluminum foil or plastic wrap around the lid, then transfer and pitch your yeast like you would normally do.
All-in-all, it's a different approach that has its own pros and cons, so whether you do it or not is entirely a personal preference.