Yea I do whirlfloc just have a lot of yeast haze in non wheat beer
Based on the little bit of reading on filters I've done:
Filters are primarily used to "polish" up the clarity; you're generally feeding fairly clear beer to the filter to get brilliant beer out the other side. Too much yeast in solution and you can clog the filter, which can lead to other issues/problems/hassle. Additionally, there is potential for some minor flavor/aroma loss due to the filtering process. Everything I've seen says it's fairly minimal - on par with gelatin fining, from what I've gathered. Last, filters have a short lifetime - so when they're full you'll need to buy another filter.
If you think there may be potential that your process is complicating matters, or your mineral/chemical/pH make up is making things worse then run it by the fine folks here - I'm sure they'd be happy to give some advice. Yeast haze may be the easiest issue to resolve; switch strains and/or give it extra time to clear in keg.
Finally, I would certainly put a little time/effort into researching possible finings that are available in the homebrew market. Most are pretty easy to use and only cost you a small amount of beer and the money for the fining agent.
Boil fining: irish most, whirlfloc, supermosshd
Ferment fining: clarity ferm
Post-ferment fining: gelatin, isinglass, kieselsol, chitosan, super-kleer, biofine clear, biofine powder, sparkolloid