Starters are for building up the proper amount of yeast as smack packs and vials aren't nearly enough for most beers. A completely fresh smack pack will have about 100 billion cells. For 4.5 gallons of 1.080 beer, you should have roughly 250 billion cells.
You'll still have beer, and there's even a chance that it'll finish fermenting. It's just that when you underpitch by that much, the yeast get seriously stressed and can add off flavors to the beer.
At this point, try not to worry about it since there's not really anything you can do. Once the airlock stops bubbling, take a gravity reading. If it's higher than you're expecting, you can always throw in some more yeast to help it finish. Taste it first, though. If you like the way it tastes, leave it as is. After all, it doesn't matter how it was supposed to finish as long as you like the way it tastes.
Tap 1: empty Tap 2: ??? Tap 3: Amber Ale Tap 4: Oatmeal Stout Tap 5: 80/-
Fermenting: Märzen, English Dark Mild, Session IPA
Kegged: Barrel-Aged Barleywine, Dry Stout, Barleywine, RIS
Lagering: Doppelbock, Wee Heavy, Helles
Barrel Aging until 6/13: Flanders Brown