I measured the groove depth with a pair of calipers as 0.03" so as doug outlined, the maximum gap is substantial (more so than he estimated even).
I brewed yesterday. Here is a picture of my grain crush at 0.022", as low as the mill can go. I actually determined it can go down to 0.020" on one side and 0.022" on the other, but kept both sides set to 0.022" so as to keep the shafts perpendicular to the bushings.
https://dl.dropboxusercontent.com/u/16976040/DSC_1410.jpg
I still find some whole grains in the crush, but usually when I pick them out they have been cracked and fall apart in my fingers.
I modified the Munich Helles recipe slightly to fit in a 5 gallon fermenter.
6.96 lbs pilsner malt, 0.96 lbs Munich malt, 0.11 lbs Melanoidin for a total grain bill of 8.03 pounds.
I mashed in 8.25 gallons of water.
15 min: 6.3 brix (1.025 SG)
30 min: 7.1 brix (1.028 SG)
45 min: 7.2 brix (1.029 SG)
-fired up burner briefly to bring mash back up to temp-
60 min: 7.6 brix (1.030 SG)
70 min: 7.7 brix (1.031 SG)
I mashed out to 70 minutes to see what would happen. I didn't go further since I had already overshot my estimated pre-boil SG per Brewer's Friend. My mash pH was pretty far off this time around at 5.72. I added a few more salts than I usually do, and the Brewer's Friend water calc was pretty far off on the pH estimation. I added a tiny amount of table salt, which may have contributed as sodium is basic in solution, but the calculator should account for that, so I'm not sure what happened.
Pre-boil volume: 8 gallons
Post-squeeze: 7.6 brix
According to Brewer's Friend, at 7.7 brix that gives me a conversion efficiency of 92%, or 88% at 7.6 brix. I'm guessing the actual reading was right around 7.65 brix, which is what caused my refractometer to jump back and forth. Brewer's friend calculated an expected pre-boil gravity of 1.029 at 75% brewhouse efficiency, so it seems like they're assuming an 85% conversion efficiency.
My boil-off rate was lower than I estimated, so I still undershot my expected OG by quite a bit. Ended up at 10.5 brix (1.042 SG) instead of the expected 12.1 brix (1.049 SG). Still, that should be an easy fix. I had my boiloff rate set to 1.75 gallons/hour, but I've changed that to 1.5 gallons/hour moving forward. Target was 4.5 gallons into the fermenter. I ended up with 4.8, with almost a gallon of wort + trub left in the kettle.
I think the helical cutters may produce a crush that's weighted heavily toward both ends of the spectrum. I think it produces a lot of flour when the grains pass through the rollers when the non-cutter sections are lined up, but also produces a very coarse crush when the helical cutters line up over the grain. I don't think it produces much in between those two extremes.