4 oz lactose in a 5 gallon batch (5 * 128 oz = 640 oz = 53 12oz bottles) would yield about 0.075 oz of lactose in a 12 oz bottle.
Googling, I found 9-12 grams of lactose in 1 cup of milk. 12 g = 0.423287 oz.
So, if your lactose intolerant friends can drink (roughly) 1.4 ounces of milk (a bit less than 2 Tablespoons) without a problem, they should be fine drinking a 12 ounce bottle of your beer.