There isn't really enough time to decant at this point. Your starter is going now and you plan to brew tomorrow. I normally chill for 24hours+ if I am decanting it. If pitching directly, I make my starter 18-24 hours prior so it's really rocking when I pitch it. In your case, I would keep swirling it, and pitch directly.