By "plastic primary", I assume you mean a plastic pail, and not a Better Bottle? The Better Bottles are really the best of both worlds- lightweight, easy to move around, and you can see through them.
From all that I've read on here and elsewhere, there are plenty of people happily brewing away with plastic pails as primaries. If it works for you, then don't mess with it. However, someday, the seals on your lids or the scratches in your pails are going to cause an infected brew. It may happen tomorrow, it may happen 50 years from now, but someday, it will happen unless you are being dilligent and swapping them out when they get too beat up. At that time, if you decide brewing is for you, then you may want to go to BB's just for the convenience. Also, the carboys (glass or BB) won't impart off-flavors if used for longterm storage, so in that sense, they are more flexible since they can be used for multiple things (primary or secondary).
Buit with all that said, even if you're just using carboys, you will still want a bucket around for use as a bottling bucket. So, really, buying a new carboy is not going to be a waste- you will still put what you presently have to good use. So, if you really want one, then go for it.