If you have a good active fermentation and Krausen, you are in good shape. With my stouts, I like at least 3 weeks in Primary and usually 4, be patient. My milks usually end up around 1.022 and after I keg, they sit for at least 3 months. Did you add the milk sugar? Good Luck