I might be completely wrong but I don't think the swelling of the Wyeast bag is necessary. I think it's simply a way to indicate to the brewer that you broke the nutrient pack open and the yeast is indeed viable. I wouldn't worry about it if it were me. If you are really worried about it you could let the wort sit in a closed fermenter until you feel you have given the yeast enough time. But I think the risk of allowing an infection to take hold by delaying your pitch would out weigh any benefit from watching the smack pack expand. You could always try a no chill fermentation if you had any interest. This would significantly reduce your risk of infection while waiting for your yeast pack to expand.
Like I said, if it were me I would give it as much time as you have and pitch it as soon as the wort is at pitching temp. Below is a link supporting my advice.
Wyeast Laboratories. Activatorâ„¢