Some strains of yeast ferment faster than others. My brews have pretty much been done with the first big BLAST of fermentation in about 48 hours. On my very FIRST brew, that got me worried. Now, I know better...or I've learned some patience.
I'd encourage you to leave it in primary for at least seven days. The yeast ARE still active...and will continue to be active until you or someone else drinks the beer!