My buckets always smell like the previous batch and has had no ill effect. You might not want to scrub the buckets with a carboy brush though as any scratches left in the buckets can harbor bacteria. I just use oxiclean and a sponge to clean mine and that seems to do the job fine.