Should be fine, I've done something similar several times with good success, particularly when your looking for a good clean fermentation.
That said keep this in mind:
For some small batches or low gravity beers such as an ordinary bitter, there is a very slight chance you might end up over pitching if you get carried away. High pitching rates can result in a less than ideal fermentation profile (i.e., low or unexpected esters, yeast autolysis flavors, poor head retention) as compared to a properly pitched batch. -MrMalty Starter FAQ
While obviously your talking about a big beer, your also talking about using what is basically a 5 gallon starter for it. Imperial Stouts should have some fruity esters in it which may be minimized or eliminated if you over-pitch. So, you might be better off using one of the online yeast calculators to figure out roughly what the proper pitching rate would be and just re-using that amount of your left over slurry.