I've done 5 batches and also never done a starter. Really wish I had on one of them, though: I'm dealing with a possible stuck fermentation. That was a 1.086 OG beer, though; for the average gravity ones it's always fermented out fine.
Another aspect of this is that even if you get a good healthy fermentation without a starter, I guess there's some thought that underpitching can result in "stressed" yeast and the development of off-flavors as a result. I don't have nearly the experience to give any thoughts on this, I'm just sharing what I've heard.