There are a couple of tricks to make a 5 gallon mash tun work in a situation like this -- all of them borrowed from historical brewing techniques when they didn't have highly modified malts.
You could:
A) Do a double mash. You essentially make two beers. Split your grain into two batches. You'll probably have to bulk it up to 20lbs to make up for absorption and other losses. Do a normal mash with half the batch and collect as much of the first runnings as possible. Reserve that and sparge normally, collecting the 2nd and 3rd runnings in a separate vessel. Rinse out your mash tun and repeat with the second batch of grain. The high gravity 1st runnings from both mashes get boiled as your big beer, although you'll probably end up with 4 to 4.5 gallons rather than a full 5. The 7 gallons (or so) of 2nd/3rd runnings from both mashes get boiled down to make a nice 1.038 to 1.040 mild or ordinary bitter.
B) Again split your grain bill but mash and sparge normally, keeping all of the wort together. Clear your grains and refill the mash tun with the 2nd batch of grain. Heat the wort from the 1st mash to use as your strike/sparge water.
C) Fill your mash tun with as much grain as it will hold -- about 12lbs for a 5 gallon tun. Mash and sparge normally. When done, clear about 4lbs (actually closer to 5 with absorbed water) of spent grain and "cap" your mash with the remaining grain and repeat the process. This is another old school technique for getting the most out of your grains.
I've been researching this a bit lately. I also have a 5 gallon cooler for a mash tun and want to make a serious 1.080+ Old Ale. I'm going with method A -- doing two mashes and collecting two sets of runnings, one for the big beer and one for a mild.
Method B was used mostly for estate beers, as far as I can determine, to create extremely high gravity beers for long aging. The technique (again, as far as I can determine) was often employed for October/November beers because the barley harvest had just come in and grain was plentiful.
None of these fits any particular model in Beersmith or other standard brewing software, so there will be a fair amount of winging it.
Hope this helps,
Chad