I tried this approach, but sadly my mill apparently has a bit too much friction for this poor drill to handle. I had to get the rollers spinning before I started pouring in grain. The drill just didn't have quite enough umph to get things started.
To fix this problem I went back to Harbor Freight and put down another $25 for the larger 1/2" arbor drill. It's still not a DeWalt, but it definitely has enough torque to spin my mill. I think I'm going to turn the smaller drill into a stir motor for my HLT.