You're not being paranoid, you're being smart. Beer is too much work to dump batches due to something as simple as taking care of your water.
1) Are you using the garden hose for your chiller or are you using it to actually fill your hot liquor tank? Traditional garden hoses, the nice green kind, are not "drinking water" certified, meaning they will leach a lot more nasties than the white "drinking water / RV" hoses. I would strongly suggest not using a hose to fill your brew water and if you do use a hose make sure it's drinking water certified.
2) Your water tastes like chlorine? If I understood your post correctly, nevertheless you are not filtering it because your brewing friends say it's fine? I would ALWAYS filter any municipal water for chlorine. You can get drinking water from the grocery store that is chlorine free (I'd personally avoid R/O water because R/O also removes minerals that are needed in the brewing process - so unless you add your own minerals back in you may be doing yourself a disservice by R/O - only my 2 cents but I comment since the poster above suggested RO). Also, keep in mind just like some people are more sensitive to diacytal than others, some people are more sensitive to phenolics than others.
Are your friends extract brewers? I ask this because chlorine reacts with the phenols, the pre-cursors of which are derived primarily (thought not exclusively) from the mashing process and sparging processes. During fermention yeast interacts with these precursors to form phenols (your clove, pepper, spicy properties when used with the correct yeast and in the correct manner for the style). There are good phenols and bad phenols. One of the types of phenols that is generally bad are polyphenols which are typically associated with the extraction of Tannins from overparging or sparging to hot (+170). I don't want to get overly technical, but the precursor compounds which react with the chlorine are less present in extract because you're skipping the mashing/sparging process so there are less phenols for the chlorine to react with to bond and create chlorophenolics (that nasty bandaid / medicinal aroma). So it would make sense that if you were using extract in your prior brews and didn't notice the influence of the chlorine that now that you've switched to PM and All Grain you'd notice an increase in the reactions. You're also typically bringing your water to boiling when using an extract before adding anything to the water - likely boiling off some, if not all, of the chlorine before you ever add your extract. Not so with PM or all grain because the chlorine is bonding to the phenolic precursors (and any polyphenols or tannins) created during the mashing / steeping process as the water is not hot enough to drive off the chlorine.
3) The taste your describing could also not be the chlorine. Bad phenloics, burnt plastic, medicine, band aid, etc can come from other sources including tannins from oversparging (see above), tannins from over crushing grain, bacterial infection, insufficient rinsing of sanitizing agents, a whole myriad of causes.
If you want to assume it's the chlorine in your water, which I think is a better assumption than the other causes listed in (3) based on the content of your post, you can fix the issue by using bottled drinking water, running your brewing water though a filter, pre-boiling your water, or treating it with cambden tablets. All viable options.
Last but not least, even if you are treating your brewing water, keep in mind that the water you are using for cleaning (including the water used to mix up your sanitizer) also likely has chlorine in it so be sure whatever treatment process you're using on your brewing water you're using on ALL water that comes in contact with your beer.