I also had this same thought once. Like everyone said, whirlfloc is not supposed to affect spices and such.
That being said, if I were using spices as kettle additions, I wouldn't use whirlfloc. If I were using spices in secondary- like when dry hopping- I would.
I know, there is no science or proof for this whatsoever.
My thinking is that if I want spices to come through in my beer, it is more important than acheiving a perfectly brilliant beer. Sort of a ,"just in case it does" kind of thing. No harm at all if I'm completely wrong.
Many spiced beers aren't brilliantly clear anyway....
Pez.
EDIT, I'm not a huge fan of spiced beers, but I do enjoy a lightly spiced winter ale.