Yes.. I use a plastic primary, no bad at all. Have a rinse-after-use sanitiser, of which there are no problems. But I don't scrub my gear, I let it soak in sanitiser for at least 2 hours then rub it.
Not to be hyper-critical or anything, but sanitizer is not the same thing as a cleaner or detergent. Simply soaking your gear, particularly plastic gear, in sanitizer is not really the best way to clean it. Plus, long periods of time in sanitizer can actually lead to damage of plastic parts.
I suspect that if you used something like PBW or Oxyclean to clean your parts, you wouldn't have to rub it all.