I think the reason PM isn't so popular, has more to do with methods available to us these days. Such as BIAB... I did one PM batch, then went all grain. With BIAB, you're really just limited by your pot sizes. Besides, malt extract is pretty expensive compared with grain. You just pay with more of your own time (which when you're making beer, as a hobby, is dirt cheap)...
You can go all grain without spending a lot of money (and not use the BIAB method). Get a large enough cooler (or dig out that big one you haven't used in X years), install a ball valve and some kind of system to separate the grain from the wort when you drain it (anything from cheap braided stainless hoses, ripped apart and attached, to pre-made items running <$25) and you're pretty much in business. Just need a way to heat up your mash and sparge water volumes (half batches can be done if you can't heat up enough for a full 5 gallon batch).
I've gotten my 70 Coleman Xtreme cooler fully converted into a MLT now (leak free too). Hardware cost me about $50-$60 to do it up. But, I can now mash up to ~45# of grain at a time.
That means, I'll be set for my BIG beers, as well as 10 gallon batches when I'm ready.
I have a 32 quart aluminum kettle now (stock pot with a ball valve installed) as well as a 60 quart aluminum kettle, to use. Couple that with a couple of cheap 20 quart pots, and I can brew pretty much any 5 or 10 gallon recipe I wish to.
I see the difference between extract brewing and all grain similar to making a cake. Sure, you can get some solid results with the pre-packaged stuff (just as you can getting it from the bakery already made). But, if you start from scratch, you can make something really great. Plus, you can adjust the recipe to fit YOUR tastes. Not accept what someone else decided to put into the recipe.
Time difference between extract and all grain brewing comes down to mash/sparge times. Even then, chances are it's just the mash time (usually 60-90 minutes) since sparge time is usually similar to what you would steep specialty grains for. Have a way to get 6-7 gallons of wort up to a boil (for 5 gallon batches) and you're done... Boil times are the same between extract, PM and all grain batches.
Personally, I love the freedom all grain gives me, for what I brew. I can look at someone else's recipe, and tweak it to how I want to make it. If I want to make it a bit stronger, I can add more base malt. Adding more extract to a recipe usually costs significantly more. Get your grain in bulk, like with one of the area group grain buys, and your cost for grain goes through the floor. Get your own grain mill/crusher, and you can now keep whole grain on hand, milling it just before you go to mash. I did that for the first time this past weekend... Looking forward to seeing how that batch comes out.
Brew with the method you're comfortable with... But, if you want true freedom to brew anything you like, I see partial mash and all grain being the only real option. With all grain beating out partial mash for real freedom...