OK now I am confused. I thought that the inaccuracy came from the calibration of the SG side compared to the Brix, but that the Brix side was still accurate for wort. Are we now saying the Brix side has up to 7% error? When I first used my old refractometer I compared it to a hydrometer reading and it matched. I just assumed from then on it was accurate.
Short answer: yes, that's what I'm saying.
Long answer:
Brix is a unit that describes the concentration of sucrose in water. A 20 Brix solution is, by definition, exactly 20% sucrose by weight.
SG, on the other hand, is a measurement of density relative to water. A 1.080 wort will weigh exactly 1.080 times an equivalent volume of pure water, again by definition.
(Plato is an interesting hybrid measurement, but for our purposes you can just consider Brix and Plato to be equivalent).
What a refractometer measures is neither concentration nor density, but rather optical refraction. When you send light through a solution, it bends, and different kinds of solutions bend light to different degrees. What makes a refractometer useful is that the relationship between sucrose concentration and refraction is very predictable. A refractometer does not measure sugar concentration directly, but it measures something else that can be converted to sucrose density in a reliable and predictable way.
Unfortunately, we don't make sucrose solutions. We make wort that includes a wide mix of various sugars and other substances. All these different substances affect the refractive index of wort in different ways. A maltose solution refracts differently than a solution of dextrins, and both of these are different than sucrose. In most worts, the measurements are "pretty close", but there's no way to produce an exact conversion from optical refraction to concentration or density without knowing exactly the composition of the wort. That's not possible without serious lab gear and a test procedure far more complicated than using a hydrometer.
At the end of the day, there is always going to be an inherent uncertainty to converting between the optical properties and density properties of wort. On average, I find that bumping up my Brix reading by 5% or so gets me a value that is close enough to a hydrometer measurement that I don't worry about it. If you are trying to track density more precisely than that, you won't be able to do it with a refractometer. It's just a hard constraint on the nature of the universe.
(As a side note: If 2-3% measurement error is too much, what are you trying to track? Efficiency? ABV? Attenuation? In all these cases, there are other inherent limitations on measurement that introduce as much or more error than a refractometer will. In any system, your data is only as precise as your least precise measurement. For most people, that least precise measurement won't be gravity.)