I suppose it completely depends on the style of beer brewed. but regardless letting beer sit on fermentation byproduct for too long at room temp
will eventually seep off flavors in to your beer. this wont happen within 3 weeks time, but i wouldnt primary for much longer unless the style specifically calls for it.
The best resource that lays it out without bias I've found is HBT's own wiki.
http://www.homebrewtalk.com/wiki/index.php/Conditioning_the_Beer
Also whether you notice the difference in flavor can depend on other varying factors including but not limited to serving temp, carb level, beer complextiy, avb %, etc.
All I am saying is that chemically beer changes during conditioning and I have to meet someone who says it's for the worse (unless you let it sit too long, which is a
long time). If you can't tell the difference then it doesn't matter one way or the other :-) Find a process you like and stick with it. Repetition and consistency is the only hard guideline to great beer.