When I was saving bottles I'd pour a glass, immediately fill the bottle with hot water and rinse, fill it half way 3 or 4 more times, shaking vigorously for a second or 2 each time, fill it all the way up with hot water and finally, when I had enough, I'd fill the sink with hot water, submerge the full bottles and let them soak overnight.
The next morning the labels come right off and any leftover glue or paper comes off easy with a green dish scrubby.
The ones that won't come off easily the next morning get recycled.
I perform the same fill and shake ritual with my labelless bottles after pouring and fill them with hot water to soak for an hour or more before dumping out and one more quick rinse.
I've never washed a bottle and never had an issue, but I perform this rinsing ritual without fail.
Washing is probably required for bottles that aren't immediately and thoroughly rinsed.