I repeat!
We are talking aluminum here?
I agree with you here...
You WANT the aluminum oxide layer ON the pot... It's specifically in the sticky on aluminum pots use... You build up a good layer to PROTECT the pot and to prevent anything from getting into the brew (chances are slim to nil if you condition the pot first, which is a snap)... You want that layer on the pot to prevent low PH from getting to the aluminum of the pot itself... So unless your wort is super acidic (like a PH below 4) you won't have an issue.
I just picked up a 60 quart aluminum stock pot, with steamer basket insert from a restaurant supplier last week. I've installed a ball valve in it, and will be conditioning it as soon as my propane burner arrives (Tuesday).. I've been using a 32 quart aluminum stock pot to brew in for the last 4 batches... Zero oxidation in any of them. Of course, I did build up the aluminum oxide layer on the pot BEFORE I used it the first time. I've also made sure to maintain it (not scrubbing, or scrubbing hard enough to remove any of it).
There are plenty of home brewers that have been using aluminum pots for years without any issue. IF there was an issue, you wouldn't have people saying they work, they'd be saying to avoid them at all costs... Sure, LHBS want you to buy the stainless steel pots they SELL... It's in their better interest to do that. But take one aside and ask what they really think of using aluminum pots, and chances are you'll hear that they're just fine...
BTW, aluminum pots have been used in the restaurant business for many, many years... IF there was ANY risk of off flavors, or harm, they would not be allowed/used by any restaurant at all. How many times have you seen a cooking show, where they're in a restaurant's kitchen, and they're using [basically] all aluminum pots and pans? More times than you can count...