I do the same thing. I'm generally good about rinsing my bottles after I pour, but I found that if I didn't dry them well, I'd sometimes still get stuff to grow in there. Then that would dry by the time it came to reuse the bottle and it was stuck on like glue. My solution has been to rinse and then set the bottles upside down over a paper towel. Keeps everything clean enough that a simple sanitization is good enough for refilling.I am kind of anal retentive about it, but I like to hold my bottles up to the light and look at them before I sanitize each one. I do sometimes catch ones that have crap dried in them even though I am careful about rinsing them out immediately.
check out the jet bottle washer thing. sprayes the crap out of the bottles and probably wastes less water since it cuts off when you dont have a bottle on it. makes for fast n easy rinsing