It sounds like the bulk of the fermentation is over but I would not remove it from the yeast just yet. Even after you stop seeing bubbling in the air lock, fermentation is still going on. Additionally, the yeast are cleaning up some by products of the fermentation process that can cause off flavors.
I assume that you are using a bucket to ferment in. There's nothing wrong with using a bucket but if you were using glass, you would still be able to see movement in the beer where your yeast is hard at work.
Bottom line, leave it at least 7 days. I have an IPA in my fermentation chamber that is on day 13. I'm just about the check to see how close I am to hitting my final gravity.