Without a big starter, lagers take much longer to get started than ales. What was the pitching temperature? I usually make a bit starter for lagers, then refrigerate it, and decant the spent wort before pitching at 50 degrees.
I guess I'm thinking that if you pitched it at 50, then put it in the fridge at 40, then took it out and let it warm up to 54, that it probably shocked the yeast a little bit.
I think it will still be ok, once the temperature settles into a good range, and the yeast reproduces and starts fermentation. It would be best at 50 degrees, but up to 55 would be ok.