Never really thought of something getting colder when I added ice as odd...., but... It shouldn't stress the yeast as much as possibly cause it to go temporarily dormant. Any up/down swing isn't a great thing but given the choice, down would be the way I'd want to go. What is the ambient in your house? Have you tried not adding ice, instead adding some fresh cool water or just a couple cubes of ice? If you have your carboy or bucket wrapped as well as sitting in the water, you may not need anymore drop than that will already provide.