I took the coffee grounds and put them in with 5 minutes left in the boil. Since you're already past that point, honestly, if you haven't been sticking your nose or hands into the grounds, you'll be ok.
One thing I've learned so far is that beer is much more resilient then some people make...