It's more important to control the characteristics of your brewing water when brewing all-grain than when brewing with extract, because the mineral makeup and PH of your water can affect your mash negatively.
Typically when you want to know about your tap water, you should get a water report from your municipal water source if possible. The other way is to send a sample of your water off for analysis. People here seem to recommend
Ward Labs, the test you need is W-6.
Your tap water as it is could be perfectly fine for brewing certain styles of beer, but might benefit from dilution with RO water or addition of minerals for others. You may also need to treat for chlorine and/or chloramine as suggested here by charcoal filtration or using campden tablets.
I haven't gotten this proficient with water treatment yet, and you may not really need to, especially if you're just starting out with all-grain. It's probably better to stick with what you have at first and tweak your water a few batches down the road if needed, maybe just do campden tablets for now, as it's a very cheap and easy way to get rid of chlorine/chloramine. I just brewed my first all grain batch, and just used tap treated with campden, no filtration or mineral salt additions. I felt fairly confident in doing this because I found detailed water reports from my water source, and the mineral levels and PH were largely within range for the style of beer I was brewing. The beer is still in the fermenter so I don't know the finished result yet, but I took a sample yesterday for a gravity reading and it seemed to be just fine.
Palmer's "How to Brew" book has good information on water treatment. Here's an
online link, the printed book is a newer revision and may have more information.