I have had a 5 degree swing wake yeast up before, even in secondary, and they chopped on some more sugar. Nothing major, didn't notice anything I would consider an off flavor from it, but I know the yeast got a little more active.
Personally, I don't think there is much of a reason to build...