You and your beer will likely be just fine.
Yes, if it was contaminated the gravity would typically drop substantially. Sometimes a lot more than what the yeast could do on their own.
Contaminating a starter is not something that happens much, at least in my brewery. I think I've only had it happen once in the past eight years that I am certain of and that was due to keeping it around longer than I had planned due to scheduling problems.
The smell test is about all you can do quickly and easily. Don't expect the starter to smell wonderfully. There's usually no hops it them and they are often fermented at relatively high temperatures. What you are trying to detect is any sourness in the wort. If you're still uncertain after the smell test, do a taste test. That should nail it down solidly. Again, it won't taste anything like a finished beer cooled and carbonated, but you should be able to detect a ruined starter without too much trouble.
When seriously in doubt, dump it and pitch a couple of backup dry yeast packets as others have suggested. Dry yeast has recently come into its own. I want to try a lager using dry yeast to see if it will perform as advertised.
Do yourself a favor. Get a two liter erlenmeyer flask and build yourself a stir plate. Making starters should be a slam dunk operation. The procedure becomes very routine and reliable. You have more important things to focus on as brew day approaches such as the recipe and your brewing procedures. You are violating the RDWHHB philosophy. Brewing should not be a stressful experience in any phase.