Do "professional" brewers consider brulosophy to be a load of bs?

[applescrap]

i really appreciate the scientific rigor you are trying to bring to this debate. I do have a question and a comment.

Question: How many of the brulosphy experiments have you read (more than the headline and results)...actually read the full write up? Actually this question goes for other posters in this thread too...i see a lot of comments regarding the experimental design that don't seem to have really read too the reports.

In reading the vast majority of the experiments it seems to me that marshal and crew are focused on detecting whether process or ingredient changes result in a perceptible difference. Everything else is intended to guide thinking about future experiments.

While i am a firm believer in fermentation temperature control i do find it interesting that their experiments show that some other things i took as largely irrelevant may lead to perceptible changes in the beer that are easier for typical bomebrew drinkers to detect than control of fermentation temperature.

Take for example the tasters were able to distinguish between beer brewed with wlp001 and us05. But tasters were not able to distinguish between beer brewed with galaxy and mosaic hops. Tasters saw a difference between glass carboy and corney keg fermentation. But did not see difference between chocolate malt and carafa special 2 in a schwartzbier. In all of these examples i am much more impressed about whether people could detect a difference than whether the qualified group preferred one over the other. When i design a recipe it is my preference that counts, but i will likely pay more attention to choice of wlp001 vs us05 in the future.

Yes these are all single experiments and i'd also prefer to see them repeated before changing tried and true processes. That said all of them are presented with sufficient detail in the reports than any of us could take on the challenge to try to repeat.

+1

 

[eric19312]

Not a single one! And why not? Because my goal has not, up to this point, been to critique what Brulosphy did or didn't do but rather to point out things like the importance of designing the test according to the information sought and the demographic it is sought from (a triangle test is a test of the panel and the beer) and some of the pit falls such as failure to mask a differentiating parameter that is not being investigated and possible procedural errors (e.g. failure to isolate panelists from one another and failure to randomize the order of presentation of the cups. The only comment I made about Brulosphy in particular was that if any criticism of what they did was justified it was probably in one of those areas and I'd say that about anybody I new was doing a triangle test.

haha ok

To me it seemed that your heavy science/math posts on this thread were intended to argue that the Brulosophy crew was somehow failing in their scientific rigor.

In hindsight this may be exactly the answer to OPs question. Not really all that interested in amature science.

 

[QuadConPana]

A friend recounted a story about his cousin who worked on a Ford assembly line. When asked what he did. He said he screwed these little white things into the engine, but didn't know what they were. Turns out they were spark plugs.

I'm sure there are more than a few people who work in breweries, that know their own jobs, but couldn't describe what's going on inside the brew.

 

[applescrap]

Water is one of several stops (often the last one) on the way to very good beer but other things are just as important or more so. Good water is a sine qua non for good beer but good water is easily obtained by adding 1 gram of calcium chloride to each gallon of RO water used (mash and sparge). Brew with water like that to let you get a handle on grist design and preparation, mashing, hopping and fermentation. Then when those are under control you can come back and tweak the water if you like but don't expect dramatic differences.

These ubiquitous beliefs from this forum, are worthy of debate, imo.

 

[ajdelange]

You should do that and maybe try some of the experiments.

I will. As for doing some of the experiments I have done many experiments usually more in the acid/base properties of malt line than than the sensory line. But I have done sensory based experiments exploring, for example, the question as to whether reducing chloride levels improves beer. It doesn't improve my beer in my opinion and as the demographic I am interested in is me there would be little point in doing a triangle test. Beyond that I don't have access to the facilities necessary for proper conduct of such a test and would therefore have questions as to its validity.

What the hell is "Virginia/Quebec" by the way?

Ogden is a municipality in the Estrie. I live there in the summer. I'll bet, now that you know that, you can figure out the significance of McLean and Virginia.

Quebec is in Canada.

Newfoundland is too.

 

[ajdelange]

interesting stuff no doubt! Why is there a earth?

I fear you are missing the point(s).

 

[applescrap]

whether reducing chloride levels improves beer. It doesn't improve my beer in my opinion and as the demographic I am interested in is me there would be little point in doing a triangle test.

Srsly?

 

[seabrew8]

I fear you are missing the point(s).

Well tell me kind sir. What is the point of our little petty debate.

 

[ajdelange]

That's exactly what Brulosophy and Drew and I at EB expect people to do.... We're presenting the results of our experiments as starting points for further exploration.

I did go and look at one of the experiments - the one that found a warm fermented beer indistinguishable from a cool fermented one. I chose that one as it flies in the face of common sense. In the writeup it is stated that a triangle test was used. There were several identifiable problems with the conduct of this test. First, the picture showed a bunch of guys sitting around a table with beer bottles in front of them. I'm not sure whether that represents the 'triangle test' but if it does then there is a big problem in that these people are able to interact with each other. Even if this photo does not depict the tasting I doubt the 23 panelists were isolated from one another and that is important. Second, the panelists were served 2 samples of A and one of B. Proper procedure requires that the panelists be given AAB, ABA, BAA, BBA, BAB or ABB with the sequence (order is important) randomly chosen from one of those 6, third, two of the panelists were unable to decide. In a triangle test they must guess if unable to decide.

So if we take those 2 out of the panel he found 9 out of 21 panelists were able to correctly identify the odd beer. Assuming that failure to randomize the order of presentation and failure to present 2 A's as frequently as 2 B's in the triplet does not have an effect on on choice under the alternate hypothesis (that the beers are different) we find the probability of 9 out of 21 is 0.24 under the null hypothesis which hardly supports the alternate hypothesis. Since the alternative hypothesis in this case is highly likely to be valid we would suspect the panel (palate fatigue, ordinary fatigue, did this at the end of an evening of camaraderie, just had a squid, garlic and onionpizza...) or the environment (distractions, masking aromas...). Or did the use of dark malts mask some subtle differences in flavor. Or was the particular yeast strain chosen insensitive to temperature relative to others. Some of these things are mentioned in the comments to the web page where the results are posted.

Further investigation is definitely warranted if anyone really thinks that fermentation temperature might not have an effect on beer flavor.

We're not presenting what we feel are scientific conclusions in the way AJ is describing. We're all frustrated that people take them otherwise.

This seems to say "We are doing some experiments and collecting data under questionable conditions but we are putting it through triangle testing. Even so we are frustrated that anyone would try to determine whether our results are meaningful or not". In an earlier post you suggested dropping publication of p values. As p values based on triangle tests aren't meaningful if the tests aren't triangle tests then perhaps that's best. Perhaps its best not to mention triangle tests at all. Just tell the people what you did and report the numbers you got. Let the ones interested (me and the other guy) in significance figure it out for themselves.

 

[ajdelange]

These ubiquitous beliefs from this forum, are worthy of debate, imo.

Many of them, particularly related to water treatment, are debated at length in the Brewing Science forum.

 

[ajdelange]

Srsly?

Absolutely. Too little chloride and beer lacks body. Check the Brewing Science forum (which should really be rename the Water forum).

 

[ajdelange]

To me it seemed that your heavy science/math posts on this thread were intended to argue that the Brulosophy crew was somehow failing in their scientific rigor.

I had, up to this point, suggested that given the myriad details that must be attended to in running a meaningful triangle test, that it would be difficult for amateurs to be successful. I have now checked one Brulosphy experiment and found indeed that there are problems with some of the things they did in that experiment. I suggested that they drop mention of triangle tests at all and just report the tasting panel results.

In hindsight this may be exactly the answer to OPs question. Not really all that interested in amature science.

Many professional brewers aren't interested in brewing science whether the scientist be working in his kitchen or the labs of Inbev. I once questioned a lab guy on what he was doing. He indignantly replied "Look, I don't know why what I do works. I just know that it does work." That's sufficient for many. One of the wise men said something to the effect that success in brewing depends greatly on science but is, ultimately, an art.

 

[betarhoalphadelta]

Imo, the path to better brew is the water.

Water is one of several stops (often the last one) on the way to very good beer but other things are just as important or more so. Good water is a sine qua non for good beer but good water is easily obtained by adding 1 gram of calcium chloride to each gallon of RO water used (mash and sparge). Brew with water like that to let you get a handle on grist design and preparation, mashing, hopping and fermentation. Then when those are under control you can come back and tweak the water if you like but don't expect dramatic differences.

I think there's a middle ground here... The people who discount the value of water--have good water to start with.

I brewed almost 9 years before going to RO+salts. I used campden tablets to neutralize chloramine, but other than that was using what came out of the tap.

During those 9 years, I did what AJ is suggesting. I honed my process. I got all the other aspects of brewing under pretty tight control. I was pitching plenty of healthy yeast. I was controlling fermentation temps. I was chilling rapidly. Etc etc etc. I made a lot of good beer in that time, and when I say that, my beer was consistently good. There weren't "clunkers" that were outliers and bad, so I believe my process to be sound.

After I switched to RO+salts, I *immediately* noticed a difference in the beer. The beers--especially the pale ones--where noticeably brighter, more crisp, and just seemed cleaner. I was actually surprised at the difference. I was NOT expecting the water to make that big of an impact.

At the same time, however, one of my homebrew club buddies (@JonW), lives only about 20 miles away from me in Huntington Beach. His water is sourced differently than Mission Viejo. He makes excellent beer, using only a charcoal filter (IIRC), and sees no reason to move to RO.

Water is a huge portion of beer. I'd venture to say that it's in some ways a question of your starting water, though. If your tap water is "good enough", switching to RO+salts probably will make a minor difference. But if your tap water is NOT "good enough", as mine wasn't, changing the water can be a pretty big step.

 

[ajdelange]

What is the point of our little petty debate.

I wasn't aware that we were engaged in a debate. I was referring to the salient points in the ongoing discussions of triangle and similar tests. Your remark suggests that you have missed those.

 

[applescrap]

Many of them, particularly related to water treatment, are debated at length in the Brewing Science forum.

I hope so, but somehow I feel the prevailing ideas can be found in any common Brewing book. I think you should really rethink your stance on water. And I'm going to offer some unstatistical proof, and there is absolutely no insult intended in that. You are obviously brilliant and i admire your posts. I think a lot of times we all come off different than the inflection that was meant. See I'm not an engineer, I don't need any proof, I don't have any ponies in this race, I barely like Brewing, and I choose what to believe.

Now mino choi was on Denny's show, and he was on basic brewing, I think it was. His first mead, I guess, scored a perfect score from an accredited BJCP judge. From there he's won medals in everything, it's ridiculous. He has golds and best of shows it's crazy. Anyways when someone like that talks I listen. He brings up how the Germans didn't move to Wisconsin for the tap water from the treatment facility. He gets his water from a special well and said that he filled up big 50 gallons or whatever of it. He stated when he stopped using that water he started to lose. Now I know a guy who makes great beer and I've always wondered about it. His well is fed from a glacier behind his house. When Marshall and the crew do a write up, I believe it, almost 100% and it sticks with me, if nothing else as background thought. I know he doesn't want me to take it that way and I know others don't want me to take it that way. I know he's only offering for thought and I know others think differently, but as far as I'm concerned I have no reason to not just go along with what I see.

 

[betarhoalphadelta]

Now mino choi was on Denny's show, and he was on basic brewing, I think it was. His first mead, I guess, scored a perfect score from an accredited BJCP judge. From there he's won medals in everything, it's ridiculous. He has golds and best of shows it's crazy. Anyways when someone like that talks I listen. He brings up how the Germans didn't move to Wisconsin for the tap water from the treatment facility. He gets his water from a special well and said that he filled up big 50 gallons or whatever of it. He stated when he stopped using that water he started to lose. Now I know a guy who makes great beer and I've always wondered about it. His well is fed from a glacier behind his house.

And I'm sure you, AJ, and I are in agreement, basically. Better water = better beer.

But it's when you jump to the "water is the most important" thing in brewing, there I think we diverge. I think it's important, but I don't want to overstate the signficance of water relative to everything else.

It's hard to consistently medal in brewing competitions* without adequate water. And "ideal" water may be one of the differentiating factors in between a beer that merely medals and one that's a BOS candidate. But I'd state this: perfect water + bad process = mediocre beer.

Water is important, but so is everything else.

(* This is of course assuming perfect judging, which we all know is hit or miss. But I think there will be a positive correlation between the better beers and better scores / medals / BOS, even if it's an imperfect correlation.)

 

[applescrap]

@bwarbiany i understand your thinking. Take great water and poor process, etc, and there is no guarantee of quality, medicore you said. The problem is, in real life its just not that way. I mean consider some average skill, one person recirculates at 152 another passively lets it drop from 154 to 151. One ferments at 70, one at 66. These differences arent going to mean anything in the long run, imo. Consider again beer is 90 (90!) percent water. Looks like fuji water would be close to 45 dollars for a batch. There is no way someone is going to out do that, with same recipe because one mashed with tap at 152 and the other at what, 156, 146, 160. I don't believe it, thats where I stand. Im not jumping in, im all in until further notice, GIVEN some basic considerations obviously. This simple looking tea is nothing simple. It was made with spring water from a canyon outside boulder, co filled with a pretty cool hippie community. Anyways you know how good our tap water is here. There is no comparo, imo. This tea is incredible, and its the water.

View attachment 1502653690718.jpg

 

[ajdelange]

I think you should really rethink your stance on water.

Way too late for that.

And I'm going to offer some unstatistical proof,

A contradiction in terms.

You are obviously brilliant

Thanks but let's not get too carried away.

...and I choose what to believe.

Most people eventually come to find out that doing this leads to problems in several life arenas. investment decisions (frequently involving small breweries) is one. Now I do know a guy who made a killing investing in a small brewery (Victory) but I would hardly advise you to invest in one based on his experience because the vast majority of people I know who did this lost at least a substantial part of their investments.

Choosing what to believe is called by various names but "Confirmation Bias" is a common one. We are all, even engineers and scientists, subject to it and because we know of its dangers we take great pains to blind ourselves to it. This is one of the advantages of a triangle test. Neither the panelists nor the people serving the samples to them know which is the test beer.

...as far as I'm concerned I have no reason to not just go along with what I see.

An investor receives a free stock pick newsletter suggesting that a certain stock is going to go up. It does. The first letter is followed up by a second predicting that a second stock will go up. It does. Shortly after than he gets another letter suggesting that if he hold any of a certain issue he should sell it because it is going down. It does. This happens until he has 7 letters correctly predicting stock market moves. An eight letter comes saying that he has gotten free stock advice which clearly demonstrates that the writer can pick winners correctly time after time and that from now on if he wants more of this can't loose advice he'll have to pay $1000 for a year's subscription to his monthly newsletter. Should the guy send in the $1000?

This is an actual scam. Think about why.

 

[applescrap]

An investor receives a free stock pick newsletter suggesting that a certain stock is going to go up. I does. The first letter is followed up by a second predicting that a second stock will go up. It does. Shortly after than he gets another letter suggesting that if he hold any of a certain issue he should sell it because it is going down. It does. This happens until he has 7 letters correctly predicting stock market moves. An eight letter comes saying that he has gotten free stock advice which clearly demonstrates that the writer can pick winners correctly time after time and that from now on if he wants more of this can't loose advice he'll have to pay $1000 for a year's subscription to his monthly newsletter. Should the guy send in the $1000?

This is an actual scam. Think about why.

Was this meant to be funny? Literally, this is great.

 

[ajdelange]

Take great water and poor process, etc, and there is no guarantee of quality, medicore you said. The problem is, in real life its just not that way.

The problem is that in real life it is that way. I have great water. Must be great as I've made some great beers with it. OTOH I've made some stinkers - beer's I'd be ashamed to offer you. And those have been the result of some error in process (e.g. - fermentation temperature dropped too low for an ale).

The only reasonable explanation as to why someone would be so naive as to think that he can make a great beer with 'great' water and despite process errors is placebo effect, a form of confirmation bias.

Was this meant to be funny? Literally, this is great.

No. It was meant to be illustrative. To show that what you see isn't always to be accepted. If something seems too good to be true then it nearly always proves that it isn't true. You might want to consider ESIL water or Kangen water.

You probably ought to take further water questions to the Brew Science forum. But wear your lead underwear.

 

[applescrap]

@ajdelange oh i get what you are saying about something being to good to be true. Stock futures, margin buying, pyramid scams, and get rich quick programs.

 

[Bilsch]

This seems to say "We are doing some experiments and collecting data under questionable conditions but we are putting it through triangle testing. Even so we are frustrated that anyone would try to determine whether our results are meaningful or not".

This, in my mind, is the best description of the science of Brulosophy.

 

[bob1852]

For folks concerned about poor tasters, etc, this write up is fairly interesting.

http://brulosophy.com/2016/01/21/in...t-xbmt-performance-based-on-experience-level/

 

[Denny]

This, in my mind, is the best description of the science of Brulosophy.

That's not exactly an accurate description, but I can live with it.

 

[ajdelange]

For folks concerned about poor tasters, etc, this write up is fairly interesting.

http://brulosophy.com/2016/01/21/in...t-xbmt-performance-based-on-experience-level/

That discussion is about testing hypothesis H1: "Tasters with more experience are better able to discriminate differences in beers than those with less". The null hypothesis then is H0: "The ability to distinguish differences in beers is not related to experience."

In the post he lists correct odd beer detection percentages:
General Beer Drinker: 40%
Craft Beer Enthusiast: 49%
Home Brewer: 43%
BJCP in Training: 46%
BJCP certified: 44%

If we make the assumption that the level of experience ascends as we go up the list then there is a correlation between experience and performance but a weak one. Pearson's r is only 0.235 and the probability that we might see a value of r that high under H0 is 35% which lends very weak support for the notion that performance is related to experience.

Now let's slaughter a sacred cow. Lets assume that the average craft beer enthusiast is actually more experienced (whatever that means) than BJCP judges. After all he may have been critically tasting craft beers for many years while the BJCP judge may have been at this (albeit with great enthusiasm) for but a year or two. Certainly the data suggests that craft enthusiasts have more relevant experience than BJCP judges as they scored better.

Now let's slaughter another sacred cow and assume that the BJCP judge in training is actually a better judge (has more relevant experience) than the average certified judge. The data suggests that this may be the case as they performed better and I know my beer tasting skills were better when in the thick of training (weekly training panels with other judges) than they have ever been. Anyway, this is my analysis and I can make any assumptions I want. This may lend some insight into the engineer's joke I mentioned in an earlier post i.e. that all the statisticians on earth laid end to end wouldn't reach a conclusion.

With my new assumptions the 'levels of experience' are now in the same order as the performances and the conclusion is very different. For this rearranged data set Pearson's r is 0.988 and the probability that we would see r that big or bigger under the null hypothesis is only 0.08%. We are now on solid ground rejecting the idea that experience doesn't make a difference.

Bottom line here is that testing whether experience makes a difference or not depends very much on how we define experience.

The reason for the linked post is that readers, and the experimenters, have been concerned that many of the experiments results in answers that don't carry statistical significance at the levels we like. The author is seeking an explanation and has focused on his panels. This is definitely the right thing to do but overlooks the fact that the power of a panel depends on signal to noise ratio which depends on the beers as well as the panel. The more different the beers the louder the signal. The better the panel's skills the less the noise. Clearly he is dealing with signal to noise ratios that suggest that half or fewer of panelists are going to be able to detect the odd beer in a triplet. Assuming the number is half and that he is running panels of 20 then 10 would be expected to pick the right beer on average. The statistical significance at the 10 out of 20 level is 9.2% - not significant relative to the usual maximum acceptable level of 5%. A more powerful test is needed to attain significance. Improving the panel to the point where the signal to noise is such that 60% choose correctly would imply that 12 out 20 would be successful on average. The significance level associated with 12 out of 20 is 1.2%. Assuming the pool of tasters is what it is improving the panel is going to be tricky. They could be given a tasting test, for example, but that would have to be done with care. Empaneling only tasters who can demonstrate ability to tell the difference between the beers you want to test is clearly folly so you would have to test them on some other beers but how would you choose the other beers? This gets back to the earlier discussions (that so infuriated some) of matching the panel to the demographic you are interested in. A test showing that 85% of people chosen from a pool that has demonstrated that they can taste the difference doesn't tell you much about the man on the street or the average home brewer or in fact about any particular demographic but the one you have sampled - those you already know can distinguish the beers. But improving the panel isn't the only way to increase SNR and thus significance. Just making it larger will do the same thing. If the panel size were doubled to 40 then we'd expect numbers like 20 correct from a panel with 50% probability of choosing correctly when the beers are distinguishable. The significance associated with 20 out of 40 is 2.1%.

From this we conclude that while panels of 40 are tougher to handle than panels of 20 this may be all that they need to do to gain the significance they desire. The fact that they seem to be unaware of this is a bit disturbing. I haven't delved far into the site but it seems that they are attempting to apply statistical methods to their results (which is commendable) without understanding how to do that. They are certainly not alone in this. They should consult someone who knows how these things are done or shell out the $45 for a copy of ASTM E - 1885 Standard Test Method for Sensory Analysis - Triangle Test.

 

[applescrap]

That's not exactly an accurate description, but I can live with it.

Love the show, I would like to be on it!

 

[Denny]

Love the show, I would like to be on it!

Shoot me an email or PM with ideas of what you'd like to talk about.

 

[cmac62]

OMG, this thread was a long read. Now I have to check out Brulolsophy, to see what all this is about LOL. I know I stopped racking to secondary after one of two times because it didn't see to make a difference and I couldn't tell that sitting on the trub was hurting my beer. I also think I may have to get some iodine to see when my mash is done. Well thanks guys for the "light" reading.

 

[Denny]

OMG, this thread was a long read. Now I have to check out Brulolsophy, to see what all this is about LOL. I know I stopped racking to secondary after one of two times because it didn't see to make a difference and I couldn't tell that sitting on the trub was hurting my beer. I also think I may have to get some iodine to see when my mash is done. Well thanks guys for the "light" reading.

The iodine test is pretty worthless IMO. And also check out experimentalbrew.com

 

[ericbw]

The iodine test is pretty worthless IMO. And also check out experimentalbrew.com

Why is that?