I replace when they are abraded from cleaning or have begun to deteriorate (usually the inside edges will chip and split). Discoloration isn't typically a reason for replacement as long as they are still cleanable and capable of being sanitized.
I will say though, I switched from white buna/silicone gaskets to black EPDM gaskets last year and I'm really glad I did. They're just as durable, don't discolor, and are super easy to locate in the bottom of a white bucket full of sanitizer. Pricing is similar if not better, also.