Ok, thanks for your help, but I figured it out after some extensive googling.
To start I'm a moron. I had the ground from the power and the fan wired to the 3rd pin of the pot. I removed them from the 3rd pin and no more problems. I just assumed you need all the wires connected to the pot. Lesson learned.
Thanks again for responding, I really aprreciate it.