No worries man. I've had hop pellets fall on the floor. I have dog hair all over the house despite my efforts. Your yeast is likely to win the war with germs.
I find that threatening my child with the loss of limbs usually keeps him away from my wort.
On a serious note though, I have my son help with the basics. I know you have 3 but maybe you can ask the oldest to help you and give him/her a buck or two when the brewing is over if he/she will keep the other two out of your hair while you brew.