It will be fine. I have done it quite a few times.
Skunking will only happen if there is light exposure. Not a worry with a keg.
If you pull it out, don't touch it, and then put it back to the same temp in the fridge later (letting it sit for a while), it will even have the same carbonation. No worries.