A good rule of thumb I've heard is 1.5 quarts of water per pound of grain. So with the 5 gallon cooler you'd get a theoretical maximum of 13 pounds but there's the volume of the grains to take into account so you're going to have a smaller volume.
Are you going to be doing full boils or are you planning something more of a partial mash schedule? You could also cut the batch size down to 3.5 gallon boil and rack to a 3 gallon better bottle.
My advice: go for the 10 gal round cooler or a larger rectangular. If you ever want to make a big ABV brew or 10 gallon batches, you'll want something bigger. My IIPA filled up my 10 gallon almost all the way to the top.
------------------------------- Fermenter 1: Best bitter (1)
Fermenter 2: Best bitter (2)
Fermenter 3: APA
Fermenter 4: APA
somewhere between 12-14# will fit in a 5g cooler depending on mash thickness/consistency. it'll work for most 5g batches to sg of about 1.060 if that's not enough find a square igloo cube in 48qt or 72qt. or watch garage sales i found my 10g igloo round for 15 bones this spring
primary1 :UTOPIA BABY(still searching for it)
secondary:middling bastard ipa
kegged:simcoe blonde, crystal pale ale, yellow jacket golden ale, lemon shandy blonde DRINKIN DAWG BREWERY LET'S GO RED WINGS