I think you might be going the wrong way in your thinking. You want the cold air to go up into the tower, especially if it's insulated (as mine is). The more of the beer line that is refrigerated, the better for your pour. If you have a 3" tower, then you should drill a 3" hole.
The tower in the link didn't specify whether or not it was insulated, but you could always put some closed cell foam sheet insulation around the outside to prevent any sweating of the tower. The rubber gasket will prevent any cold air leakage outside the fridge as long an you tighten it down enough.
Some discussions on this had people putting computer fans in the tower mouth to blow MORE cold air up there!
I did mine with a 3" hole saw and it works well. Next I'll be upgrading the faucets to forward sealing so the "stick" doesn't ruin my first pour as it does sometimes when I haven't been drinking fast enough.