With the activators, take the yeast out of the fridge and let it warm to room temp on brewing day. Then pop the nutrient pack right as you're getting started. It usually takes a little time to clean and sanitize all the equipment, brewing, cooling, etc; plus the yeast addition is pretty much the last step (well, I guess you still have cleanup). Fresh packs will expand quite nicely within a few hours. I would recommend popping the nutrient pack at room temp though. I once tried popping it right out of the fridge and it was a little slow to respond (but the beer still turned out great).
Homebrewer_99 comment does make sense though just in case you have an old pack and need to quickly drive to homebrew store to get another yeast pack.