From what I understand, even if you have only a few cells of yeast it will ferment eventually. The risks are that wild yeast or bacteria will take hold first and change or ruin the beer. Also if you pitch too little yeast they get stressed while building up to cell counts needed to ferment the wort. This can create off flavors in the finished beer. Also it is widely accepted that a vial or smackpack of yeast do not contain enough yeast cells to optimally ferment a 5 gallon batch in all but the lowest gravities. IMO, always make a starter when using liquid yeast unless making 3 gallons or less.