IMO, going all grain, when done smart, only adds your mash time to the length of the batch time. So if you mash for an hour, you're only looking at that much more time spent brewing.
Also, not sure where you're getting the cost difference from, but I can typically brew a 5 gallon batch for 1/3-1/4 the cost of going with extracts. Of course, I'm also washing/harvesting my yeast, but I'm just talking about the other ingredients here. My extract batches were in the area of $45-$55 per kit, PLUS more to go with liquid yeast. So, in the area of $50-$60 a batch for lower OG/ABV brews than I can make for $10-$15 via all grain...
It really is more of a personal preference though. As others have stated, you can make great, or crappy, batches using either set of ingredients/processes. It's more up to the brewer making the batch. But, I have noticed that I'm really liking the flavors I get from all grain batches. I'm not long from being able to drink batches where I crushed the grains the same day (within an hour of mashing) as I brewed them.
Personally, I've always enjoyed cooking from scratch. I see all grain brewing as just an extension of that mind set. I also don't punch a clock when I'm brewing, or figure out how much my time is costing me when brewing. If you start going that, you start taking the fun out of it. It would be like talking with a hunter that goes out and shoots his own game, cleans it, butchers it, and then cooks it. Are you going to try and tell him that he should have just gone to the corner store and picked up a slab of beef instead?? I wouldn't.