I've made both a cherry and a raspberry porter, and made the mistake you suggest on the cherry. The sugar in the fruit will ferment leaving tartness. Your specialty grains will balance nicely if you leave them. Don't decrease your "porterness". I've never used blueberries, but 3-4 Lbs won't get you what you want, 10 lbs may be a little excessive, but you want blueberry flavor, right? I say go for it.