The hydrometer/refractometer is the brewer's best friend! On brew day it let's you know how your efficiency was (though not as important if you are strictly extract). Most importantly though, it lets you know how fermentation is going. It is important to know that you have a stable final gravity and that it is close to your expected final gravity before you bottle. Otherwise you may end up with bottle bombs, stuck fermentation and generally unpredictable beer. Also, you get a better sense of what different final gravities taste like. There is a huge difference between a beer that finishes at 1.020 vs. 1.004. This understanding of attenuation will be invaluable when selecting yeasts, malts and choosing mash schedules.
What are your methods for taking gravity readings? At what temperatures are you taking your readings? If you give us your procedure I am sure there are some tips that will make it easier.
All that said, I think you can wing it with you taste buds. If the beer tastes right, and has had three or more weeks in the fermenter, then chances are it is done. The yeast know what they are doing; we work for them, they don't work for us. I wouldn't chance it though when it is so easy to take readings.