Could be a few things. Disclaimer - I am not an electrician, or an engineer, and am in no way qualified to speak on this subject, other tan having built a stir plate, and destroyed a few power sourced and fans in the process.
I too had to spin the fan to get it going a few trys. Is there anything attached to it yet, like washers/magnets?
It could be wired incorrectly (like i said - destroyed a lot of components).
The switch/potentiometer might not be able to pass enough power to turn the fan. I opted for no switch just to get as much amps as possible pushed thru to the fan.
Could you take a picture of your wiring, and include any specs on the switch/ potentiometer?