Well, fermenting the beer is just a part of what the yeast do. If you leave the beer alone, they will go back and clean up the byproducts of fermentation that often lead to off flavors. That's why many brewers skip secondary and leave our beers alone in primary for a month. It leaves plenty of time for the yeast to ferment, clean up after themselves and then fall out, leveing our beers crystal clear, with a tight yeast cake.
So many of us leave our beers in primary for a month, then bottle. The longer you hold off the more time you give to the yeast to eliminate those things that cause off flavors in your beer, and will help make it clearer. I know it's your first batch and you're anxious, but the longer you wait, the better tasting beer you will ultimately have.