If you plan on adding something like brewing salts back to the water, and you are wanting to build your own water profile for each beer, then go for it. The minerals in the water balance the mash to insure proper conversion has taken place, and add flavor components to the beer like excentuated hop or malt flavors, and some people like mimicking the water profile of different places to try to reproduce their beers exactly.
If you don't want to do that, then a carbon filter would suit your needs much better. All you really need to worry about is removing the chloramines in tap water, and a simple carbon filter (Brita, PUR) will get the job done. I have one of those that fits on the faucet and it works fine. Or just use raw sewage, it works pretty well too.