I would think that there is a chance of a bit of the spice coming through. I've reused yeast cakes with great results, but I only use them when the yeast is from a fairly neutral beer, or at least similar to the beer that I'm putting on top of it. I wouldn't do it, if I were you - not worth the risk.