Any bacteria/etc would have to work their way through 6+ feet of beer line, through the disconnect, and down through the dip tube to get into my kegs, and this is all while fighting the occasional dispensing which would flush out all the beer in the line, and the fact that the fridge is nice and cold which slows bacterial growth way down.
In short, I'm not worried. I clean my beer lines occasionally for the purpose of removing any nasty buildup that might cause off flavors, not because I'm concerned about infection.