I'm going to make a wild guess here in that your using pellet hops.
I used to have the same problem as you and the stirring to get the wort through the funnel still results in some hop material getting into the carboy. I've never had a bad batch turn out from this method. I have never just removed the filter and dumped it straight in, and I suppose it will not cause much issue.
Some recommendations for you to think about. Switch to whole hops, use a very fine mesh hop bag druing your boil, remove the filter pour into your primary. If your very confident with your sanitation processes leave the fresh wort in the primary and let things settle for 24 hours, then rack off the top of the trub to a clean carboy leaving the trub/hop material behind, then pitch your yeast in the fresh carboy. This is a very long process and most will argue against my last recommendation, and they are correct to do so as you increase your odds of an infection.
A final option I would recommend is check you LHBS and pick up an auto siphon. I've been through in 10+ years of brewing and they are a HUGH time saver.
Edit: A boil pot with a drain tube and pickup with a filter on the pickup is another good option.