As long as you don't touch the sanitized surfaces that need to remain sanitized, it shouldn't matter.
Now, I'm not saying you should have dirty hands from yard work or cleaning out the gutters, etc. or dripping dirt everywhere.
Clean hands and wrists, washed with soap and rinsed with or dunked into clean water ought to be sufficient. Also consider the clothes you're wearing, such as loose sleeves, anything that could touch those sanitized surfaces or spread dirt over them.
I use a half-size bucket with 3 gallons of Starsan. I dunk my hands and wrists in it regularly. I also use smallish (~8"x 8") cotton washcloths in my Starsan bucket for mopping surfaces, such as insides of fermentation buckets, lids etc. I like the thick foam it produces that clings to the surface. I like to submerge smaller items, jugs, jars, hoses, tubing, etc. too for at least 3 minutes. Larger items get mopped and/or sprayed.
When tinkering with yeast, I shut the AC/heat pump off...