I believe AJ will agree that mash chemistry is dynamic and variable to the extent that many factors (agitation, mash thickness, quality of malt, quality of minerals/acid/etc) come into play. Adding additional acid or alkalinity to an active mash will need time to complete incorporation into the entire mash, and the affected result may take 15-30 minutes to stabilize. That is a long time when doing a 60 minute mash! Making measurements and adjustment 5 minutes apart will not allow the chemistry to stabilize for a more accurate measurement - hence the chasing your tail comment.
Yes, I do agree with that.
If you see that Bru'n is somehow 0.2 points low - then check your work to make sure something is not in error.
I always say that if something doesn't look right it probably isn't right and experience has taught met that this is usually, but not always the case (see comments above about all the factors). Unfortunately, the only way to gain the instinctive insight of which I speak is lots of experience. There is another thread in Science where I believe the poster also saw an unusually low pH prediction from Bru'n water. In this case it was confirmed by a pH meter reading but it still didn't make sense in terms of what one expects of normal base malts, normal caramel malts etc. So what do you tell this fellow? Exactly what it says above - go back and check everything and repeat the measurements multiple times.
I am disciplined to only make a single adjustment to a mash, and only when it measures outside my required range, say between 5.3 and 5.5 and have precalculated additions to move the mash 0.1 unit in either direction.
This is, of course, more likely to happen to a new brewer than an experienced one and a new brewer is, of course, more likely to mismanage the adjustments and wind up chasing his tail.
I have not had to do this in the last 20 or so brews.
Experience at work!
Bru'n Water estimates have been very close, within +/- 0.05 units, despite AJ's statement. I have not seen Bru'n consistently estimate low, and suspect that there maybe some user error issues at play, OR lack of precision in the brew house. I am quite confident in my methodology and the results.
I must reiterate that my comments about Bru'n Water, Brewer's Friend and EZ Mash are not based on a Consumers Report set of comprehensive tests but more on what I see people report here. Once the notion that Bru'n is low (Martin has mentioned that he gets complaints about this), BF about right and EZ high gets abroad then we have, in addition to concerns about relying on anecdotes, to consider that confirmation bias may be creeping in.
We could look deeper for explanations as to why Matt gets accurate predictions and others don't with any of these tools but that's going to be a tough job as I don't really know how any of them work. I believe that Bru'n is based on collected empirical observations by lots of brewers. I believe BF attempts to work out the acid base chemistry using titration data collected on a bunch of malts. I have no idea how EZ works. In the case of the first two we have to ask how global the emprical data was (did it include mostly ales or a mix of ales, lagers, wheats.....?) and in the case of Bru'n and in the case of BF how global was the ensemble of malts tested and how well does the collected data correlate with color (as malt quality is input as its Lovibond color rating). Depending on these answers it is quite feasible that Matt is brewing beers that are typical of the Bru'n ensemble and that might explain why he gets good agreement. At the same time some of these folks who notice low predictions may be doing unusual things (or at least unusual in terms of the ensemble). This would explain things but I certainly cannot assert that it is indeed the explanation.
IMO the test mash is the most accurate solution, but also requires an extra step that most brewers will not do. I don't personally like taking the time, but I will on a batch intended for competition.
Again experience can be your friend here. If you consistently brew the same beers or the same types of beers you should have a pretty good idea as to what to expect unless someone throws you a curve ball such as your water company changing source or a maltster receiving a crop of barley that is unusually alkaline.
Also, if you have a program that consistently gives you good predicitions then you can rely on that as you have verified that its model is viable for your application.