I don't think it will hurt it to take it out. Spoilage is dependant on micro-organisms. They're either there or not. Cold conditioning is good for beer - if you're looking for fridge room, go ahead and take them out. If you're looking for the best beer possible, leave them in. Either way will be fine as long as you don't go to temperature extremes.
The best way to store malt is to combine it with water, hops and yeast... -Charlie Papazian