I don't sanitize my growlers, just like they don't sanitize them when I buy them full. I mean, they're clean. You just grab a clean one, fill with a piece of tubing to reduce foaming, and pop the lid on. Just like at the brewpub. It takes all of 3 minutes.
It's nice if they're cold to reduce foaming, but not necessary. I have the piece of tubing right next to my kegerator, with the other supplies, and just fill as needed. I also fill Grolsch bottles as needed without sanitizing, since they are going to be drunk right away, or in the fridge until they're consume. Usually, for a weekend jaunt to the cottage, I take a couple of growlers and a few Grolsch bottles since I only have one keg out there. I don't sanitize any of them, but I do store the beer in the fridge until I drink them.
For actually storing beer bottled from the keg, sanitizing and doing it "right" is imperative.