I always make a starter with any fresh yeast, just because I can't stand the slow starts I've always gotten when I pitch right from the pack.
As far as fermentation time, 004 works FAST once it gets going. The last time I made an Irish red with it the beer was at final gravity in about 52...