On the quality of Brulosophy exbeeriments:
I don't think they're that bad. Really. I'm a scientist and I don't see a lot of places there that might be improved.
They're doing two batches as identically as they can, changing one variable (usually). Exbeerimenters will do things like split a yeast starter, ensuring each gets the same thing. The temperatures of mash, fermentation, and so on are very close, such that no reasonable, experienced brewer would expect, say, a .4 degree difference in mash temp to produce a noticeable final result.
The essence of experimentation is control of extraneous variables. Unless they're lying to us about what they're doing (and I don't think they are), they're doing a pretty nice job controlling.
There's a flaw in understanding research that sometimes people make--that is, the fact that you might not have perfectly controlled everything doesn't mean that what you didn't perfectly control had a negative influence on the results. It might have, might not have. If the variable not perfectly controlled differs significantly between experimental and control batch, it might be an alternative explanation for the results. Might be. This is why we replicate.
****************
My issue w/ the Brulosophy approach is on the back end--deciding if the beers are different or not.
I don't agree with posters above about the lack of value of tasting as a measure of difference compared to measuring physical attributes of the beers. In the end, we brew beer so we can drink it, and we like the beer we drink to taste good. Thus taste as a measure of beer quality has tremendous face validity.
The problem is what different people perceive. My signature says what it says for a reason. I don't care for Belgian beers. There's something about the flavor that turns me off. I can judge them (just had one from Morrey that was an excellent Belgian--but I'd never order one
). People who like Belgians would, I'm sure, like Morrey's. No off flavors, good mouthfeel, flavor is rich and full. And I don't care for it!
So when we get a panel of tasters in brulosphy exbeeriments, we're seeing if a bunch of people can tell the difference between two beers. We know nothing about what they've been drinking prior to testing, whether they like the style or not, if they're super tasters versus people like me who are not, of what beer drinkers, if any, the panel is representative.
****************
When I teach this I have my students, w/r/t sampling, imagine doing a student survey and asking passersby to take that survey. Would their results be different if they were asking student passersby inside the Engineering building compare to those inside the Art building?
I don't know which building the Exbeeriment sample tasters come from. The IPA building? The Sours building? The BMC building?
****************
On top of that, a typical panel will have more than half the tasters unable to identify the odd-beer-out. Think about that. People are focused on whether the results are significant, which is fine, but there's another very interesting element of the data--most can't tell the beers apart!
****************
Further, and this part gets me, the writeup usually says, if the result is not signficant, something like "tasters were unable to reliably distinguish between the two beers."
Um.....no. Reliability is repeatability or consistency of measurement. If each taster were identical, sure, but they're most assuredly not. I'd be more impressed with the results if tasters came back 3 or 4 days in a row and were able to repeat the same conclusion each time. What we don't know is how many "correct" guesses were in fact that--guesses.
What we have is analogous in some ways to a one-shot case study. No way to know whether the results would be the same tomorrow w/ these same tasters. Or a different set of tasters.
****************
I've read a little, and see a few videos, on how the food industry does taste tests. Much more controlled. Among other things, I'd love to see if tasters could, really, distinguish between these beers on successive days. And before drinking any other beer.
Are tasters in the exbeeriments suffering (if that's the right word) from palate fatigue? If the exbeerimental taste test was their first beer of the day would the results be different than after drinking a couple IPAs? Or a slew of other beers?
These are questions I'd want to have answered before drawing any great and weighty conclusions.
****************
Just to give Marshall a little love, he's trying to do some things. Most of us critical of the methods or the measurements aren't doing a damned thing, including yours truly.
Marshall and his cohorts are "
in the arena."
If and when I can figure out how to brew large batches and split them, I'll try some of this. I'm anxious to do so. And if I do taste tests, I'll try to do them under more controlled circumstances.