You should be fine, especially when you pitch straight from the pouch/vial, some lag time is to be expected. Also, not sure if you're fermenting in a bucket, but sometimes they don't seal up all that well, so you don't see too much activity in the airlock, but it's still chugging along.
Good...