Is that room temperature, or the temperature of the liquid inside the fermenter? My experience is that it takes a very long time for the liquid to change temperature, since it's a pretty big mass. I have those cheap stick-on thermometers on my fermenters, and they're pretty decent at giving you an idea of any temperature changes.
When I am experiencing big temperature swings in my house, I often use a water bath as an insulator. It takes even longer for temperature changes to happen, so it seems to really keep the temperatures stable.