I am kind of new to this, so wait for confirmation that I am right... I think you want to wait to transfer to the secondary until your fermentation is pretty much done. You need a hydrometer reading that stays the same for 3 days to know.
That being said, I don't think it is going to hurt your beer too much since it is obviously still chugging along, despite the transfer to the secondary. Let it go and do its thing. It takes time. Patience is good (I need more of it too

)