I'll be brewing outside, and I'm not going to buy into the "dont mill where you boil" stuff.
The same can be said for hot-side aeration, leaving beer on the yeast cake will give you off flavors, doing full boils with an extract kit will mess up your hop utilization, ect. Obviously, there is a shred of truth in all of it, but with proper technique any off flavors can be avoided. There will always be some room for mistakes, and you can still have great beer. If you accidently splash some hot wort around, it's not immediately cardboard-beer. Leaving the beer on the yeast cake for 5 or 6 weeks instead of 4 wont give you dead yeast-beer.
This is a good thread:
http://www.homebrewtalk.com/f36/bacteria-grain-dust-105517/index3.html
Anybody who says they've had an infection due to grain dust is usually reluctant to describe their chrushing/brewing technique, where as people who crush/boil in the same room can describe exactly what they do(and it usualy involves good sanitation and common sense).
As others have mentioned, of course there is tons of bacteria in grain, but I bet theres a lot more floating around outside while im brewing, or in my garage if I brew in there.
Keeping a clean, sanitized fermentor closed and sealed untill the moment before you siphon wort, clean tools, and a clean CFC will mitiagte almost all risk of infection.
On my first all grain I actually milled my grains directly into my fermentor, then just rinsed it with clean water and a good rinse with Star-san before racking. Great beer! Will I do that again? No, but it definately didnt hurt anything.
Sorry for the rant on bacteria, but im a fan of RDWHAHB.