Something is amiss in your sanitizing process, whether it is in the starter or the process of going into the fermenter you are not being careful somewhere along the line.
What is your cleaning an sanitizing process and what product are you using. I have had starters in the fridge for a week covered with foil and have never experienced an infection in almost 350 gallons of beer.
You can take your current starter out of the fridge and let it sit at room temp for a couple days and see if something starts to grow to see if it's infected.
Nothing Left to do but smile and drink beer.....
The Commune Brewing Company-Perfecting the "art" of beer since 2010