really the only thing to be careful of is scratching the aluminum and using caustic on it, i believe.
there is an aluminum oxide layer that forms and is unreactive, meaning as in it won't leech anything. but if you scratch it, or chemically remove it with a cleaner or sanitizer that wasn't meant for aluminum, you can remove that oxide layer. by so doing, you allow aluminum to leech into the wort if the aluminum isn't given enough time to reform the oxide layer.
that said, I use SS wherever I can. i converted sanke kegs (which is generally pretty cheap for the capacity) and use SS fittings on just about everything. i will soon be getting SS racking canes as well. I just find it easier to work with as far as ease of cleaning and sanitization, as well as flavor issues. i didn't want to deal with pickling brass, and i don't want to have to worry about the aluminum oxide layer if i scratch the pot. steel passivates fairly quickly with exposure to air, so i'm not too worried about that.
this is just me and my desires/priorities with brewing. everyone is different. get what you need/want that is within your budget.