I have also used yeast slurry from an IPA for bread baking. The problem is that almost no matter what you do, you'll wind up with bitter notes that are (to my palate, anyway) very off-putting in bread. The aromatics seem like they should be nice, but in practice you get bitterness too.
I can only imagine this would be magnified greatly by using spent hops. They're never totally devoid of bittering potential, and it'll only take a little to make your bread gross.
That said, I'm an empiricist, and if you try it, I'd be pleased to hear the results--it'd be great to be wrong.