I did this EXACT same thing about 7 weeks ago. I was worried about infection. The wort was probably about the same temp when I "remembered" my late addition, and believe it or not I was brewing an IPA as well!
The only difference is I didn't bring the wort back up to a boil after the super-late addition.
I was worried about infection, but I didn't want to mess with the hop profile by boiling again.
Beer turned out fine!!
I don't think boiling for 5 mins would change much either.