I've calculated that about 7 mL of healthy, fresh slurry (or cake, etc), is about the ideal amount to pitch into a 1 liter starter. I generally go with about 10 to be on the safe side, as I harvest from the yeast cake after my batches are done (so I'm probably getting some break and other refuse in there). Have had excellent results using these numbers.
From what I understand, one can overpitch a starter. Essentially what happens is that the starter gets super saturated with yeast, and rather than reproduce (which is what you want), they go directly into fermentation. This leaves you with tired, geriatric yeast cells that will not perform as well when compared to specimens from a properly pitched starter.