It depends. Did you have your hands completely soaked in Starsan while you were putting it in the first time, or was there a brief second where your hands were almost dry as you were desperately trying to pry the stopper out?
If at any time your hands were less than completely soaked in Starsan, then your beer is likely infected with some of the most despicable dregs of the bacterial underworld. Once it's done fermenting you'll need to contact some of your local homebrewers for proper disposal procedures.
This is definitely a time for relaxing, not worrying, and drinking beer. The gum rubber from the stopper won't have any effect on your beer, and the odds of introducing anything detrimental are extremely low. In the event that it did happen you'll be among the first to know.
(FYI, if you have to ask "Is this an infection?" it isn't. If you've got a real infection you'll know by the look, smell, and taste and will hopefully be posting a thread entitled "Look at this gnarly growth.")