Try milling your grain twice. That is my standard practice. I've noticed that the first pass cracks open the grains but does always shed the husk. You really want your grains loose from the husks for maximum conversion and milling twice can do this nicely without having to upgrade your mill. Just use the same gap setting on each grind pass.
On a side note, judging efficiency as a stand alone number is really misleading. There are a lot of variables that affect it. Most of all, unless you have the specific extract potential for the grain you bought, your efficiency will vary greatly. For instance, Promash says the extract potential for Marris Otter Pale Malt is 37 points per pound per gallon (ppg). Lets say I get a bag that has this exact extract potential and I calculate the my efficiency for a brew session and it comes out at 80%. Then I get a second bag from another batch and it's actually better than the first bag and gives 38 ppg, but since I don't know this, I just take what Promash tells me (37 ppg) and all of a sudden my efficiency jumps up 2%. I didn't do anything special, I just got a higher yeilding grain and didn't account for it and got a correspondingly higher (but deceptive) efficiency. Actually, the efficiency didn't change at all. The same things works the other way when you get grain that's not so good. Maybe it got a little wet, or it's old, or the moisture content isn't exactly what it should be and all of a sudden your efficiency dips 5% because Promash says you should get 37 ppg but in reality, your grain is only capable of 35 ppg. It only takes a dip of 2 ppg to affect your efficiecy number by 5% or so, when in reality, your efficiency is exactly the same, you just didn't account for the lower yeilding grain.
If you use one kind of base malt all of the time, you can usually get a consistent efficiency which is really what you are after. If you switch malts often, you will likely see varying efficiencies and it is likely due to the varying extact potentials and the deviation from the published nominal values. Just food for thought.