I base it off of how easily you want to transport it and aesthetics. If you want to be able to move it around very easily, and tuck it away, get a smaller tank. 5 pound tanks are very common for this purpose. I actually had a 2.5 pound tank for a while before trading it with another homebrewer.
I personally use 20# tanks for most tasks, and have two 50# tanks for serving at the walk-in cooler. I find the cost of filling the 20# tanks to be a worthwhile compromise for their larger size, and as long as you don't have any leaks, it should last 4x longer than your 5# tank.
Now, with that said, if you have the money, and it comes down to one 20# tank, or two 5# tanks, I would actually recommend two 5# tanks, so that you have a spare that is full at all times. I actually have around five 20# tanks and two 50# tanks, so I don't ever expect to run out of CO2 since I always have one 50# tank filled and at least 1 spare 20# tank completely full while using the others.
I am a big proponent of spares - if you stay around here long enough, you will see threads about people running out of CO2 on usually a Friday, with a party over the weekend, and can't find a place to refill the tank. It always seems to be one of those things that runs out exactly when you need it the most.