If you have a hose end spray noozle, I find that you can can a significant volatilization of chlorine out of the water if you spray into the pot with a fine spray. You should smell the chlorine as you spray. I would still let it sit overnight, but you will have removed some of the chlorine off the bat. Investing in an activated carbon filter is also a good chlorine removal method.
Bumping the calcium up is a good idea. As AJ says, the source of your calcium addition should be based on the flavor goal of the current beer.