I am not a doctor, nor do I play one on TV, but if it were me I would certainly try a batch without one or the other, or both. That would help test my theory. Campden kills bacteria so I am unsure if that would be something you're more sensitive to or if it is the phosphoric acid. I think that is found in Coke as well and you could be sensitive to that. It's really a crap shoot and this is more or less a question that maybe you should ask your doctor. Is it just one homebrew and you're done for or is it a few? After a few too many homebrews I know that things are little different for me, if you catch my drift. I know that it is common because typically we're not filtering the beer and there is more yeast floating, even after days or even weeks in the fridge.
Simply put, I'd do two things; brew without both of those (use spring bottled water if your tap water is too poor to brew) and see a doctor. If you really want to test theories you can brew a gallon with campden and without the phosphoric and then another gallon without campden but with phosphoric. See how you fair. Again, it could be a sensitivity or you could even be just coming down with something.