It will taste fine.
Some problems might be if you harvest the yeast, there will be less yeast.
Also, you might get more hop matter in the bottles. I brewed a Heady Topper clone and didn't do anything to filter out the hops in the boil and put them all in the fermentor and I had so much hop matter at the bottom that it never all settled out and some would get in the glass.
I think the best way to combat hop matter in the fermentor is to use a hop spider or just hop bags. When transferring to the fermentor, you can filter with a paint strainer bag (or not) and then you won't have that much going into the fermentor. And if you dry hop, put the hops in bags.