Most activity I've had, after using a starter, was one that had yeast pitched in about 24 hours before it was used. I used about 1/2 cup DME to ~1 quart water in that one. I didn't even boil it up for long (a few minutes at most) then just let it cool down enough to be safe for the yeast.
Yeast was post-lag in <2 hours and blowing foam through the airlock in about 6 hours. Luckily I had tubing to use for a blow-off...
I did swirl the starter every time I walked past it (for 5-10 seconds)... I'll probably try to replicate the results with a future brewing.
That being said, I've had very strong fermentation going after letting the yeast sit in the starter for ~2 days (sometimes a little longer). Letting it build up a nice size colony. Generally speaking, I plan a brew day for the start of the weekend, making a starter for the yeast so that it should be good for that day. So if I'm planning on brewing on Saturday, then Thursday evening, or early Friday, I put the yeast into the starter. That way, if the yeast needs more time to get fully ready for what it will be pitched into, I can move the brew day to Sunday without issue.
Once I make/get a stir plate, I'll probably have more consistent results with the starters and yeast.