My guess is that it will be totally fine. I'm sure there is a chance of infection, but it is very small. The pressure inside the fermenter from the CO2 will keep any oxygen from coming inside. So unless a bee flew in through the hole and dropped off some wild yeast everything will be perfectly fine.
Watch me leap through this hoop!
Brewing PM for 5 years
Finally upgrading to AG in 2016!