For a while I didn't. Then I started straining my wort as I poured it into the carboy, but I realized there was no noticeable difference with my brew so I stopped and now I just pour it all in again.
If you worry, just use a hop bag. It's something like a tea bag. Once you are done with the hopping, then ust remove the bag and no more hops leftovers to worry about.