It depends on what you're starting with. With city tap water a filter won't do too much of a noticable difference. Filters are more important for well water, or waters high in chlorine and heavy metals as stated above. As far as i know, the simple Brita filters don't remove ions important to constructing water profiles (such as carbonates, sodium, magnesium, etc.)
What make a BIGGER difference, and i know from experience, is bypassing the water softener as missing link stated. Madison, WI has some of the hardest water i've seen, but bypassing the softener has allowed me to start from a point at which i know the levels of the important components - eventhough it's much more work to "create" the water profile i need from city+RO water. After i realized this, i stopped using filtered softened water and by beers have become more crisp and have more "life".
To me, it's just another testimonial that water chemistry isn't an integral aspect to brewing, but it's WELL worth the time and effort to figure out.