In my set up, i have a gasket on both sides, right against the cooler wall. I would think that with a washer against the wall, your water/wort could get behind the washer and leak.
Of course, that would mean that the hole through the wall would need to not be too large for a gasket to...