Yeah, but are the phytoestrogens still biologically active after being boiled for an hour and potentially being chomped on by yeast? Further, what concentration would they end up being at in the beer? How much is in the hops and how efficiently are they extracted during the boil? I've done a lot of searching in the primary literature in the past, and nothing indicates that it is anything to be concerned about.
On the other hand, hops have long been used for medicinal proposes (before modern medicine), for example as an estrogen replacement supplement in post menopausal women, or as an anaphrodesiac in European monks to limit sex drive. It's also given rise to workplace hazards, like brewer's droop (I'm sure you can figure that one out) or early menstruation in girls who were picking hops. But in all of these cases, the people were exposed to way higher concentrations of hops than you would get out of a hoppy IPA. I have a feeling you'd have to blow up your liver, or put on a lot of weight (both of which are far more worrisome) before you got high enough concentrations from beer.
So I'll keep drinking my hoppy beers!