I'm disinclined to try it myself, but I'm having a hard time finding a reason why it'd be harmful, I mean so long as it remains under pressure. I've taken cold bottles all the time out of refrigeration and later cooled them back off with no ill effects. A keg is just a big bottle, right?
I'm going to say that it should be OK. But you try it first.