If that was really mold, you've got issues with sanitation. I've left batches in primary for up to 3 weeks without ever getting infections (despite blowing the stoppers a few times and cleaning beer off of the ceiling).
Is it possible you pitched the yeast while the wort was still too hot? (above 80 degrees) Or, maybe the dry yeast just wasn't good for some reason. Did you re-hydrate it? If so, was it re-hydrated with sterile water? I used to boil a cup or 2 of water when I start my brew, then let it cool. By the time I'm working on cooling my wort, I pitch the yeast in the dish of clean water and let it go to town. When the wort is under 80 degrees, I dump the yeast water-bowl on top.
Recently, I switched to a dry yeast that doesn't require activation (just dump the dry packet on top) and it still seems to work just fine.
What method were you using?