Depending on the size of the starter you aren't increasing the cell count so much as just waking up the yeast. If you're using liquid yeast, most products are considered ready-to-pitch, so you will be fine. My biggest reason for making a starter is a faster onset for fermentation. I get antsy waiting for the airlock bubbles. As long as you aren't brewing high-gravity (over 1.060+) you can skip the starter with no major consequence.