Exactly, pulled a quart out boiled and poured it back in. You could shake, but it will diffuse out fairly quickly, so I think it would be unnecessary. As far as the hops to use, you will saturate the IBUs you can get in that small volume pretty quickly with a small amount of hops. If you have Beersmith or something similar, you can get the theoretic IBUs for the volume you are boiling for a given duration with a given amount of hops/AAs.
If you accept the conventional wisdom that IBUs saturate at 100, I think the following will be true. With a 1 quart boil, you get 25 IBUs/gallon, so will bump up a 5 gallon batch by 5 IBUs. Double to a 2 quart boil and you bump up the batch by 10, which is probably more desirable in your situation.
The only downside I can think of is that you are going to be dropping your alcohol content of your batch, because it will be boiling alcohol out of the beer. I personally wouldn't worry about this, cause you're looking to make it more palatable.
Oh, and don't forget to minimize aeration of your boiled beer when you're straining the hops out. a hop bag would be a good idea.
Good luck with it and if you do this, please report how it changes the character of your beer.