My guess is that it will be totally fine. I'm sure there is a chance of infection, but it is very small. The pressure inside the fermenter from the CO2 will keep any oxygen from coming inside. So unless a bee flew in through the hole and dropped off some wild yeast everything will be perfectly fine.
Watch me leap through this hoop!
On Tap: Rocketsauce IPA, Party Ride Wheat
Fermenting: Meet Your Maker Stout, Big Ring DIPA