finicky analog johnson controller?

so i have an analog johnson temp controller for my chest freezer but it doesn't seem to work very well. most of the time the freezer is at 45-48F when i have the controller set to 40F. i haven't been around all that much since i put the kegerator together a few days ago so i haven't checked the temp all that often, but only once was it at 40F. i have the probe up over the collar, dangling in to a glass of water sitting on the compressor that has a calibrated thermometer in it.