You really should leave it in primary for at least 1 week. Search for "Primary v Secondary" and you'll get all sorts of info on this. Basically, the yeast are going to chew through most of the sugars for the first few days and then start to rid your beer of the byproducts they just created for several days after that. You can't ruin your beer by leaving it in primary for 1 to 2 or even *gasp* 3 - 4 weeks. Longer than that it becomes debatable on the good/harm your doing. By racking too soon you are greatly reducing the number of yeast cells that are going to work on your beer. For a brown ale, you can skip the secondary. Everything will fall out of suspension eventually.