I frequently leave my wort in the kettle (covered) on the counter for 2-3 hours while we have dinner and put the kids to bed, then start the boil. It's usually still hot enough that it's uncomfortable to touch when I get back to it, so it's not even a particularly welcoming environment for most of the lag period.
Plus, the mash is hot and long enough to pretty well sanitize the wort at the start of the process. It's going to take a while for something to find its way in and get a foothold, even if the wort does enter the temperature danger zone. (I guess this is more true for BIAB systems where there's very little contact between wort and surfaces that haven't been heated.)