Well, there's worrying and thenb there's WORRYING about it. Having some chlorine in there won't poison you or make the beer undrinkable, but it may not be as good as it could be. Like I said, I've been using tap water all along and I have no issues with it.
But if you already have an RO system...