Well theoretically, since you have a sanitized vessel to begin with, fresh yeast, and fresh sanitary wort....as long as you don't open the thing all the time...you should still have a sanitary vessel. Even though you have lots of crud, it is not contaminated with potential infectious bugs. In reality I think the truth is more like this...you have a dominant strain of organism that is quite ready and able to push out anything in its path to establish itself. Once those yeast begin to feed and produce alcohol (the beer pH drops as well) this creates an even more harsh environment for infections. This process happens all the time and is sucessfully used in dry salami, cheese making and composting to name a few. So even though there are tons of things floating around on dust particles, etc etc the ones that do make it (unless strong enough) are usually not capable of establishing themselves and die off. Now the big problems that we usually see with infection come from bugs that hang around that are specifically designed to eat your wort, those are the ones that can cause problems because they can tolerate all that stuff (including hops). If these are left over from a brew date (say you didn't clean and sanitize) or stuck in a bottle that doesn't get sanitized then you can have problems. But coming from a primary that was just used to ferment in, there should be very little problem as long as you take care.