Generally the device should be turned to the MAX COOLING position. The reasoning is that you always want the device ready to turn on whenever the external controller turns it on.
If it's turned down to far it might not kick on even when the external controller switches on.
The Johnson is a bit hard to configure IMO, until you get a good understanding of how it works.
There are 2 sets of jumpers. One switches it from HEAT mode to COOL mode. Obviously you want the COOL mode, so it know which direction the temperature is supposed to go when the setpoint or offset is reached.
The other selects either CUT-IN or CUT-OFF. Cut-IN says that when the setpoint is reached, turn the relay on and start cooling (if you have it in cooling mode..) and keep cooling until the offset is reached. Cut-OFF says that when offset is reached, turn on the relay to start cooling until the Setpoint is reached.
Remember that the cut mode matches the setpoint. So cut-in at the setpoint, or cut-off when the setpoint is reached.
It sounds like it's not in cooling mode to me. Or the relay is defective.