I pulled one of my kegs apart tonight just out of curiosity and HOLY CRAP there was some nasty stuff in there. I usually just spray them out real good with a hose and hit them with my carboy brush, but from now on rest assured I will be pulling the fittings and tubes and cleaning them after every fill.
I clean and replace o-rings every time (except lid)--After spending the time to brew, monitor fermentation, etc. it's totally worth it to spend the extra time and 10 cents to ensure a clean keg. Same goes with beer line--flush with BLC between every beer.
I rinse and sanitize after each keg blows. I completely pull it apart every 3 or 4 beers. I haven't had an issue yet. I do flush the lines with BLC after every keg though. I also disassemble the tap and soak it in BLC for 15 mins every couple kegs as well.