That's not quite it. That may work, but it doesn't match the actual science that's going on. The refractometer concept functions by looking at how light gets bent/refracted as it passes through your liquid sample. It's calibrated using distilled water that you know to be pure and you turn the set screw until it reads 1.00. From there, when you put on a sample that has "water + other stuff", the "other stuff" (aka wort sugars) causes that light to bend at a different angle, and then you read the gravity by where that reads. The more "other stuff" you have in there, the steeper the angle and in the refractometer you read a higher number.
But since you're comparing the light bend of your wort sample to how it bends through pure water, the comparison gets thrown off when alcohol comes into it. The bending of light as it passes through alcohol is different than when it passes through water. So that 1.023 you read might actually be 1.009, so you need to plug your reading into a conversion calculation (spreadsheet, BeerSmith tool, website calculator) to correct it.
Here's where these corrections are pretty close, but maybe not perfect. For me it's close enough, but for you, you might want to deal with the little glass bobber reading minuscuses (if that's really a word). The refractometer correction formulas ask you to plug in your OG. That's because they use that as an estimate of how much alcohol is in your wort to correct for how much the alcohol is messing around with the bending of light. Of course the OG doesn't tell your actual current ABV so it's just an approximation of how much alcohol is in there. So if you have a highly attenuated beer your actual FG might be lower than if you used a different yeast or mashed at a higher temperature, but the correction factor in the equations is the same. But as I've found, even varying mash temperatures and yeast strains with different attenuations, using my refractometer + these correction forumulas ends me up where the recipes are projected to be, so for me it's worked close enough.
But you need to use one of these formulas/calculators. If you came up with your own "correction factor" by comparing refractometer to hydrometer, that factor would only be correct for that ABV sample of beer. Works fine if you're always running around the same ABV beers, but if you vary ABV your "correction factor" is going to be off.
This link to Northern Brewer's website has calculators on there where you plug in Brix readings (since that's what most refractometers gauge in) and it spits out gravity readings (1.0xx).
http://www.northernbrewer.com/learn/resources/refractometer-calculator/