I leave the top off of the pot for the first 10-15 and try to get it under 140° then add the lid, and I move the pot as much as I can in the ice bath, whether it is spinning it or rocking the pot slightly to get the wort to move in a circular motion. At about 120° I pour 1 gallon of refrigerated PUR filtered water into the fermenter and then pour in the 3-4 gallons of wort, then top-off whats needed with more PUR filtered water, I usually hit near my pitching temp. Aerate, I splash and churn when pouring wort into the fermenter, but I give it a good stir with my brew spoon before and after pitching the yeast.
I would think that the lid off would be okay if your indoors and your home is relatively clean and dry such as those with A/C. Those with evaporative coolers (humidity) and those outside probably need to be much more carefull not just with debris, but outdoor nasties. I would think there are more nasties in the humid air. (When I was younger we had an evaporative cooler, and it seemed like our bread molded quickly and often. Now that I own a home with A/C it is much dryer and temperatures fluctuate less and my bread will literally get hard and stale before it gets moldy.)
I think infections are harder to come by than many think, especially if you are pitching the proper amount of yeast, healthy yeast would normally take over the nasties. Just santizie every piece of equipment before you use it. IMHO, given you aren't living in filthy conditions, I think your chances of a random nasty floating into your beer (that can survive the yeasties) while the lid is off is possible, but not plausable. Yes, it can happen...more than likely not. My opinion from my experiences.