my insistence upon at least 40 sample size is based on in-depth review of ASTM E1885, "Standard Test Method for Sensory Analysis—Triangle Test".
Thanks for the clarification and sorry for wrongfully putting you in the "it failed to prove X so it proves not-X" camp.
What you seem to be getting at - a 50$ paywall is blocking me off - here is different from the statement from above, which I criticized.
You initially wrote that you considered none of their results statistically significant, due to the low sample size. And here I insist to disagree: the actual statistically significant results that they do produce *are* sound, as I tried to explain above.
The problem is that in a lot of cases, the design of the experiment seems to be orthogonal to the intention or, at least, the general reception and discussion surrounding it:
Typically, the experiment sets out to *prove* a conventional homebrew wisdom, whereas it often seems they actually want to *disprove* them. And when the experiment fails to produce sound proof of "pitch healthy yeast", it is often taken as "MYTH BUSTED: yeast health is irrelevant".
Now, although absence of proof is not proof of absence, it is a natural, healthy and even intelligent reaction to wonder why it had not been possible to prove it: this is the approach most people take to evaluate the question on the existence of unicorns or vampires or the effect of homeopathy (sorry, I couldn't resist).
And here the point that you,
@dmtaylor, make in your last post becomes critical: I don't believe in unicorns because it is quite unlikely there wouldn't be any well-documented reports of them if they existed. If we had a triangle test where only 3'333 of 10'000 people identified the odd beer out, I'd say it's rather unlikely the beers were drastically different.
But an xbmt with 12 out of 26 ... ? I'd be more inclined to chalk this one up to small sample size, for if we had observed the same percentage of 46% in a larger sample size, we would certainly be convinced the beers were different.
That said, it wouldn't be too hard to set up the experiment to actually try and "prove the two beers equal" - there are some arbitrary parameters to be picked, sampling becomes an issue, but it can all be done to a certain degree of "homebrew science".
If you're interested in doing a little collaborative write-up - so people can be less wrong on the internet
- hit me up.