Personally I am a fan of hydrometers only. I read too many posts like this about refractometers to really feel they are worth their costs when making beer, especially post fermentation.
It is highly possible your samples are stratified and you are getting poor reads. For example, your hydrometer is reading from a vessel with more wort in it and the refractometer is a like a dot of wort. If that dot was on the light side of the sample it's off. Just be sure to calibrate the hydrometer at it's prescribed temperature in distilled water. It should read 1.000. The temperature your hydrometer was calibrated at should be on the small paper at the bottom, most are 60F, some are not.