I usually wait at least 4 weeks. 2 weeks to carb and another 2 weeks to keep conditioning. You can give your beer a try at 2 weeks but you should wait another 2 weeks to try it and I bet you'll notice a difference.
Plus, putting them in the fridge is going to slow down the yeast, so it'll pretty much stop the conditioning process.