They do make bimetal hole saw bits as well. Just make sure you use cutting oil with it, and go slow.
With any of this, let the bit do the work and you can help minimize any tearout on the wood. For the metal, make sure you drill a pilot hole first that will match the drill bit for the hole saw.
Lastly, if you're really worried about appearances, I'd do this:
Take a piece of scrap wood that is at least an inch thick (2x4 will work np). Drill your shank hole as perpendicular as you can (assuming you're doing this with a hand drill). If you have a drill press, even better for this step. Then clamp that to your collar, and place another piece of scrap on the backside of the collar. Drill your hole. The one you already drilled through the scrap on the front will help keep your bit straight as it goes into the collar, eliminating the need to make sure it's all perpendicular (since you made sure it already was in the piece of scrap). If you can't get a piece of scrap on the backside of the collar, put some masking tape over the area where the hole will be coming out. And again, let the bit do the work. Don't man handle the drill. You should come out with nice clean and straight holes.