That's really the way to do it. You don't want them to freeze, but dropping the temperature is the standard way to get "wine diamonds" to drop out and that lowers the acidity quite a lot. Even dropping the temperature 10 degrees from what it is now may be enough to do it, but maybe not.
Cold stabilization makes a huge difference in the wine, and is highly recommended. Do you have an old fridge or a place that you can store the wine colder without freezing?