Swirling the yeast around will not harm them at all. As long as you did not expose them to extreme conditions such as high temperatures they are likely doing just fine.
Many people use stir plates when making starters, this helps to increase the number of yeast cell's created, which swirl the...