Essentially how both products work (Whirlfloc is a pelletized form of irish moss) is by binding themselves to proteins in the beer and making it easier for those proteins to precipitate out of solution.
Any time you are dropping things out of solution you can affect flavor.
I usually use a whirlfloc tablet in the beer, cold-crash the beer in primary, and depending on style, may also add in gelatin. This ensures I get very clear beer into my kegs.
Where I don't use gelatin is on an IPA, and on hefeweizen/roggenbier. Gelatin can strip out some hop compounds, and on a hefeweizen/roggenbier you want to keep yeast in suspension. So it's very much dependent on exactly what kind of beer I'm making. But for lagers, or standard American ales, or even Belgians and the like, I like to drop the yeast as much as possible for clarity, so I use whirlfloc + cold crash + gelatin.