>>I ask because I usually don't scrub the inside, just make sure every square inch of the inside gets hit with sanitizer at the end. Any hard gunk that needs removing I usually do with my hand.
I don't think thats good enough.
I have a sponge that I only use for cleaning inside fermentors. I clean them with dish soap and water. I scrub a few times with the soap and water. I sanitize both before and after using (with the same sanitizing solution).
If you leave any dirt, no matter how small, the Star San can't get at whats in it (if anything is there or settles in there). I think many home brewers don't clean adequately. That doesn't guarantee an infection, but it can leave one more vulnerable.
If you have an infection, it's possible you can kill it off with multiple treatments of sanitizer after a hard cleaning.
(something like an ounce of bleach + an ounce of vinegar in 1 gallon of water, also using Star San and Iodophor later on)
https://www.homebrewtalk.com/f11/charlie-talley-five-star-chemicals-notes-brewcasts-58928/
But you still run the risk that you did not eliminate the infection. Bacteria can form biofilms that are resistant to sanitizers and hard to clean. Hospitals have infections because the bacteria/spores are resilient.
If you have a 75% chance of eliminating the infection, and a 25% chance you get it again, is it worth it?
.75 * $16 (bucket cost) = $12 savings
vs
.25 * 40 (kit cost + S&H) = $10 loss of materials PLUS several hours of your time.
Several hours of my time are worth more to me than a few bucks.
It's a harder choice if you have to toss a bottling bucket, 2 fermenting buckets, auto siphon, filler, tubing. If you aren't sure where the infection is.