Wood is a poor thermal insulator.
Insulating the inside of the box with 1-2" of rigid insulation, could make all the difference. Make sure every square inch is clad with the foam, no voids.
I've seen fermentation chambers built from just 2" rigid foam, with a refrigerator or a window-type AC...