My drill press slowest speed is around 5-600rpm, too fast for metal.
i would use a step bit over a hole saw. With that said I found someone (through here) that let me borrow a punch
I think I am going to get a pot from High Gravity and have them do that hole. Thanks for all the input everyone.
I ended up getting a neiko step bit from amazon for 13 bucks.