Separating Trub Before Primary?

[VachonBrewCo]

So here's my situation. I had an IPA with 7oz of hops. When I racked into the secondary, I lost about a gallon to the trub. So I just brewed a Vanilla Cream Ale yesterday, and my idea was to siphon the wort into the primary and leave the (2 oz) of hops and the remaining (undissolved) lactose behind in hopes that it would increase my efficiency. Was this a mistake? I have a feeling it is going to have an impact of the taste.

Thanks for any help!!
Cheers