Per Radical Brewing, Mosher says nutmeg is "very potent." In one recipe he adds 1/4 teaspoon at the end of the boil along with a bunch of other spices. Also, you could make a "potion." That is, steep the nutmeg (I'm guessing in chunks, not ground) in vodka for a couple weeks, then add at bottling. The advantage of that method is you can test it, adding a few drops of the potion to a commercial beer to get the amount right (then, scale up).