this is a tricky question. If the box can "maintain" you desired temperature when empty, than it is strong enough to keep anything you put inside it,(provided that this item is at target temperature) be It a huge block of lead or just air at your target temperature. Heat itself is what flows. cold dosent flow. to maintain a temperature lower than than the temperature outside the box, your fridge only has to compensate for the heat that flows through the insulation into the box from the warmer outside air. if what you put in the box is at or close to your taget temp, it will work fine.
Now if you put a large mass of warm or hot matter in the box, your fridge will still need to compenate for its own lacking insulation, and at the same time be burdoned with pumping out the heat of the added mass untill it reaches target temp. on a small fridge, this may cause the compressor to run for much longer than it;s designers ever intended. and the rate at wich it cools will probably be slower than what you intended.
my advice- cool it to as close to the target temperature and see what happens