I agree with your general point. I guess my concern would come with using soy as your primary.
I know hops have a significant amount of estrogen, but with the amounts in beer I would think it becomes nearly negligible. Most of us do not handle large quantities at once for extended periods of time.