Well, I mean they stay with the mash except for any of the oils that leach into wort. In my case, using BIAB, the grain and hop material gets pulled out with the bag. In a traditional mash tun, the same material should be vorlaufed out. Even if you were to put it all in a small mesh bag and placed on top of the mash, it wouldn't be removed and placed into the boil kettle after the mash. They are left behind. It's not the same as first wort hopping.
I can't comment on whether it makes a beer more bitter than expected because I created this NEIPA recipe from scratch and didn't base it on previous ones. If the formula in Brewfather is to be trusted, it calculated 10 IBUS for a 2oz mash hop of Cascade as opposed to 40 IBUS for the same amount as a 60 minute boil addition. That suggests that most of the oils don't make it into the boil.
edit: I found more references.
http://scottjanish.com/the-locksmit...hops-and-phantasm-powder-to-thiol-drive-beer/
"If you wonder what the IBU implications might be from mash hopping, Omega found that ~30% utilization in their trials of what you’d expect to get from the same amount of hops used as a 60-minute addition. In other words, you would get ~30% of the expected IBUs from a 60-minute hop addition with a mash-hop addition. Utilizing high bound 3MH hops with low alpha-acid content (like Saaz) is a great way to boost thiol precursors into the wort without worrying too much about IBU increases."