pjj2ba, I think thats a very good Idea to be mindful of where your contamination sources are, like dust from grains and wind.Wearing gloves and washing up well after milling will reduce your risk of bringing some thing over from the grains to the cooled wort. And like you I don't wear gloves when I brew, though the Bilchmann brewing gloves I've always wanted to get a pair to protect my hands during brewing.
I also work in a lab, and work using sterile techniques, like the OP. But one thing that always gets me, is that when people who have not been acquainted with sterile techniques used for working with biologicals tend to think of the latex glove, and other pieces of personal safety equipment as a shield from contaminating their work.
Gloves are only more sanitary than your hands when they are first put on and before you touch anything. It always fun to watch some one walking around the lab with their gloves on, and go straight in the the hood to do tissue culture work with out changing gloves or spraying them down with 70% EtOH, and the next day they are confused why some thing other than their cells is growing in their flasks.
I think my point here is for all the people who are thinking about incorporating gloves into their brew day. Gloves are to protect you not your work. If you use them, change them often. be mindful of what you touch and when you are touching it, and wash your hands and arms frequently, just like you do with out gloves...right. Determine when and where your most sensitive steps are in the brewing process. ie, post boil, pitching, racking, packaging and dispensing (if you are kegging) and be extra diligent at these points. Incorporate a mandatory hand washing after the boil, and before pitching.