So you racked into the secondary after day 4? If so thats sounds early to me, I guess from my experience and what I have read and seemed to have learned is that the fermentation may take 5-14 days (big window but the only true way to know is to take those gravity measurements). I still normally let mine stay in the primary for at least a week even if the fermentation has finished, the yeast will settle and if you choose to use a secondary (I always do, some chose not to) it will be basically to help clear the beer after you rack it off the inactive yeast...I used to go at least a week here but I did it in just a few days recently for my beer to clear before I bottled. I guess the biggest thing you need to look out for is just to make sure you have the fermentation complete before you bottle, otherwise you might have exploding bottles. Racking it too soon as suggested by another poster might cause your beer not to be quite up to par to where it should be but as long as you let the fermentation complete ( and use proper sanitation so that you dont introduce contaminants) you should have a drinkable beer....if you feel it was a mistake, keep your beer log with good notes and try it all over again and then correct your mistakes and see if it comes out any different...that will ultimately give you your answer.