Usually when a room is infected it means there's a large source of something growing in that area. having said that I would agree that if proper sanitation is followed is should be a non-issue.
Having said that I worked for a biotech company that had a lab infected with mold. It wasn't "cleanroom sterile" but it was cleaned daily. Needless to say enough people were getting contamination to the point where they stripped everything out of the lab and treated every surface with a spore-killing cleaning agent (something like peroxide and acetic acid). After that there weren't any issues.
Unless you're fermenter is sitting open next to something growing I think you should be fine. I've been dealing with micro-organisms far longer than I've been brewing. While sanitation is definitely important, the few basic steps should be enough to keep everything clean.
Prepare for Ludicrous Speed! Keezer Build