You would take it out when the bulk of fermentation has completed as a way to help attenuate the last couple remaining points and clean up some residual off characters (e.g. diacetyl), and 70 to mid70s would be fine. This would basically be when you've reached within a point or two of your expected final gravity.
Big temperature swings are not a good thing during the bulk of fermentation, mostly, because that's when unwanted esters are more likely to be produced. If you pull your fermenter out of it's water bath (~65F) and put it in room temp (~70F), it will take a day or two to reach room temperature therefore the change would be considered fairly gradual.
Simple answer, take it out of your swamp cooler when the bulk of fermentation has stopped and you've nearly reached your final gravity (typically this would be around day 4 or 5, as you've said).