I used to use soap and hot water, never had a problem. Now I use PBW (available online or at most HBS). No scrubbing involved, just let it soak and rinse well. Then use the STARSAN. This is more important if using plastic better bottles, you don't want to use any abrasive brush or scrubber on those. I don't think using soap would cause you to grow mold on your last two batches.
Brewers make wort, Yeast make beer!!!!