You need a power adapter with a bit more juice; the one you have is just a bit below the rating for the fan.
If you can find another that's 250mA or more, it'll run full speed just fine.
The 4 wire fan motors have internal circuitry that doesn't like a potentiometer on the black wire - it'll work for a while but it's hard on the motor.
You can go nuts and put a pwm circuit on the blue wire just like smooth said, if you want to speed control on it properly, but it needs to be high frequency - 25kHz is the standard. Wikipedia
has lots of good, nerdy reading on the subject.