Do "professional" brewers consider brulosophy to be a load of bs?

[joshesmusica]

To me Brulosophy shows that there is a lot of latitude in the tolerances of the beer making process, and I'm thankful for that. If the latitude range was not as broad as it is, the world of home brewing would be a much smaller one.

I only wish that they would quit making so many IPA derivatives, and concentrate upon more subtle styles. You can hide a lot behind a massive infusion of hops. They should perhaps settle upon one subtle recipe, to eliminate the recipe itself as a variable. Then later move to another and noticeably different recipe and repeat each experiment.

https://mashmadeeasy.yolasite.com/

I guess maybe I was a bit too wordy and/or my attempt at cleverness failed.

What they show is that there might be some latitude on one single variable, and even then, not all of the time, but sometimes.

This is the problem I have with the way they've worded things, and the problem others have (and now myself as well) with the way their using these statistics. Although there there might be some latitude on one single variable, and even then, not all of the time, but sometimes, what they have ALSO shown is that when you compound all of the latitude on these variables, it does in fact make a difference.

So maybe my brew day goes really well, but I missed on my mash temp by a bit. Ok, now worries as long as it's in the generally accepted (read: scientifically shown) guidelines for when the enzymes are working. Or maybe I don't have enough pale malt, so I go ahead and throw in some pilsner. Or maybe I don't have enough time to boil it for 60 minutes. Or maybe I don't have the time to do a 60 minute hopstand. Maybe I can relax a little and not get so stressed over a hobby because one thing went a bit awry. Or maybe even a couple of those things not going according to plan might not ruin my beer.

But even brulosophy has shown that if you add all of those mistakes together, you're at the least ending up with a perceptibly different beer. When I set out to make a certain beer, that's what I expect to end up in my glass. So for people like me, and likely many others in this thread and on this forum, when people start touting the results as gospel, or even less than that, just saying that there is some latitude in all aspects of brewing, I'll refute that. And in fact the one actual exbeeriment with "triangle testing" and everything also refuted that.

 

[betarhoalphadelta]

This is the problem I have with the way they've worded things, and the problem others have (and now myself as well) with the way their using these statistics. Although there there might be some latitude on one single variable, and even then, not all of the time, but sometimes, what they have ALSO shown is that when you compound all of the latitude on these variables, it does in fact make a difference.

Agreed. Their reliance on p<0.05 really makes the waters murky, particularly with small sample sizes.

If they have 19 tasters on a panel, they need 11 (~58%) to claim "significance". If they get 10 correct responses picking the odd beer, they claim it's not significant. When random chance would be 6.33 guessers.

So they claim it's not significant that tasters couldn't "reliably detect the difference" and then readers say "See! It doesn't matter!!!" No, you're wrong. It probably DOES matter but the sensitivity of the test didn't prove it matters beyond an arbitrary threshold that the world has set.

I think these experiments are very useful as a learning tool. It helps to understand how significant different processes are to making beer. But reliance on p<0.05 gives a veneer of statistical rigor to these experiments that make it WAY too easy for readers to completely misinterpret the findings.

(Edit: I keep using the word "prove" and I *know* it's grating on AJ lol... I should stop saying prove.)

 

[Kee]

So maybe my brew day goes really well, but I missed on my mash temp by a bit.
Maybe I can relax a little and not get so stressed over a hobby because one thing went a bit awry. Or maybe even a couple of those things not going according to plan might not ruin my beer.

We all have our favorite mistakes:

[ame]https://www.youtube.com/watch?v=AmIlUKo4dQc[/ame]

 

[ericbw]

It seems like the thread started with two camps: Brulosophy is awesome! OR Brulosophy is not scientific enough. Now a third camp is emerging: Brulosophy might not be scientific enough, but it still gives us insight into how effective a process might be.

But the third camp is incorrect. If it's not enough of a sample to draw any real conclusion from it, then the conclusion you draw has no basis. And as mentioned, most of them (that I've read) come to the conclusion that what they're testing doesn't matter. And often we know that it does matter, we just don't have any more rigorous experiment to confirm it.

 

[ericbw]

And why haven't the mods shut this down yet? It's off topic, aggressive, and way too much math. And boring besides.

 

[schematix]

And why haven't the mods shut this down yet?.

Because it's turned into a very productive discussion. It's actually quite rare on HBT to have such a long insightful conversation with so many intelligent minds giving their input.

It's a welcome change from "I just made my first beer. It's a 15% RIS. Why did it stop fermenting at 1.060?"

 

[bob1852]

way too much math. And boring besides.

Yet you've read and posted here.

 

[triethylborane]

Because it's turned into a very productive discussion. It's actually quite rare on HBT to have such a long insightful conversation with so many intelligent minds giving their input.

It's a welcome change from "I just made my first beer. It's a 15% RIS. Why did it stop fermenting at 1.060?"

The sarcasm went woosh

 

[HarborTownBrewing]

It seems like the thread started with two camps: Brulosophy is awesome! OR Brulosophy is not scientific enough. Now a third camp is emerging: Brulosophy might not be scientific enough, but it still gives us insight into how effective a process might be.

But the third camp is incorrect. If it's not enough of a sample to draw any real conclusion from it, then the conclusion you draw has no basis. And as mentioned, most of them (that I've read) come to the conclusion that what they're testing doesn't matter. And often we know that it does matter, we just don't have any more rigorous experiment to confirm it.

This is the issue though - you're getting too hung up on scientific process and statistical proof to even allow for the possibility of the third camp existing, thus labeling them as "incorrect".

Regarding your 3rd camp: Am I getting insight into an effective process? Me - absolutely. I'm thinking about things I never considered before. Is it "correct"? I don't know. Is it "provable"? PROBABLY NOT, at least without a tremendous use of resources.

So until people who complain about their testing not being good enough start to donate some serious cash to roll these exbeeriments and report on them in ways which fill their needs, maybe just take it for what it is and appreciate that there is some information there.

 

[Denny]

Tmaybe just take it for what it is and appreciate that there is some information there.

BINGO! Marshall and I have spent a lot of time talking about this. This is what we both intend.

 

[cmac62]

I do Know that I love brewing beer. I have met a bunch of great folks who also love brewing beer. I also know that for me the difference between 148f and 156f mash is not a lot. It may make some differences, but to my burnt out palette, not much. I am glad brewlosophy is out there doing stuff, brewing beers, asking questions, and sharing what they find out. There is so much brewing lore, but how much of it is just passed on from from master to padwan? Again, what I know is I can make some really good beer in my backyard and that is AWESOME!!

Peace Out

 

[dmtaylor]

It's data. It's input. And just like taste preferences, whether the data is good or bad or in between, or is presented well or not well or in between, is totally up to the eye of the beholder. But it's still data no matter what.

 

[ajdelange]

(Edit: I keep using the word "prove" and I *know* it's grating on AJ lol... I should stop saying prove.)

No, not at all but you should be, as you are, aware that statistical analysis does not, in fact, prove anything but rather lets us quantify our uncertainty which can be helpful (and profitable) when it comes to making decisions.

Suppose we have a coin and want to test its fairness (before entering into a game of two-up). We toss it 6 times and get 6 heads. Does this prove the coin is not fair? No it doesn't but as the probability of getting 6 heads in a row from a fair coin is 2^-6 = 0.03 we can say we are confident that the coin is not fair at the 0.03 level. That's below the standard minimum accepted confidence level and so we may be tempted to declare that the coin is not fair even though the probability is pretty good that is is. To the point that we might want to ask the boxer (the guy that runs the game) to substitute a different coin. At a level of 0.03 percent we are not very confident that the coin is fair to the point that we might want to take action. Would we be more confident if we got 10 heads in a row? Numerically, yes as the number of heads supports the thesis that the coin is unfair at the 0.001 level. We would be more likely to demand a new coin. Is 0.001 confident enough? Only you can decide.

:.. Brulosophy might not be scientific enough, but it still gives us insight into how effective a process might be.

But the third camp is incorrect. If it's not enough of a sample to draw any real conclusion from it, then the conclusion you draw has no basis.

Would that it were that simple! What is a real conclusion? Or, in terms of the previous remarks, what level of confidence is required to make a conclusion "real". If real means the conclusion is proven then there are no real conclusions and that is, of course, always the case. So why do we bother? Because we gain information. Before testing we have no idea as to whether a process change effects a perceptible change. In the last Brulosophy experiment the data supports the idea that using 8 tasters correct as a detection threshold that in 46% of panels a difference would be noted if we gave them the same beer. That's not very good support for the notion that the beers are differentiable. But the data tells us that using that same threshold (i.e. what the experiment measured) only 12% of panels would find them the same if we gave them samples of both beers. This is pretty good support for the notion that the beers are not differentiable.

And as mentioned, most of them (that I've read) come to the conclusion that what they're testing doesn't matter.

That's because they don't know how to interpret the data they have obtained nor how to pick a panel size. In many cases it is true - the data is not significant with respect to Type I or Type II errors. In several other cases, however, it is significant with respect to one or the other of those two error type. QED.

 

[ajdelange]

So until people who complain about their testing not being good enough start to donate some serious cash to roll these exbeeriments

Get me an estimate of how much additional cash is required to present AAB, ABA, BAA, BBA, BAB, and ABB in cups of the same color and tell me where to send the check.

...and report on them in ways which fill their needs,

The additional cost of that is the cost of obtaining a copy of the standard which is $0 as it's online free. Tell me where to send that [i.e. the link] too. Now, of course, I am assuming that the additional labor required to properly present the samples, collect and analyze the data is born by interested volunteers. I won't offer to pay salaries. The data crunching can be done by the simple spreadsheet I posted way back in this thread. Incremental cost $0.

I am not volunteering to build a sensory laboratory with the requisite lighting, temperature, air quality control and isolation booths. They would have to come up with some way of better isolating the tasters from one another than the posted pictures show.

.. maybe just take it for what it is and appreciate that there is some information there.

There is. Let's hope that they are willing to learn enough about this kind of testing to be able to find it all.

 

[schematix]

maybe just take it for what it is and appreciate that there is some information there.

BINGO! Marshall and I have spent a lot of time talking about this. This is what we both intend.

This really goes to the heart of this thread... is the information on Brulosophy to be taken seriously at all?

If the author(s) only intend on the information being there essentially for entertainment purposes, then that settles it. It's not meant to be a truly rigorous source and one should not cite a Brulosophy experiment as "proof" of anything. Read it, ponder it, create a meta analysis from it if you are so inclined, perhaps even duplicate an experiment for yourself. It is what it is and nothing more.

 

[ajdelange]

If the author(s) only intend on the information being there essentially for entertainment purposes, then that settles it.

Even so there may be, and evidently is, usable information, contained in the data. If their procedures are deemed consistent enough then one can do meta (and other) analysis,

 

[betarhoalphadelta]

It seems like the thread started with two camps: Brulosophy is awesome! OR Brulosophy is not scientific enough. Now a third camp is emerging: Brulosophy might not be scientific enough, but it still gives us insight into how effective a process might be.

But the third camp is incorrect. If it's not enough of a sample to draw any real conclusion from it, then the conclusion you draw has no basis. And as mentioned, most of them (that I've read) come to the conclusion that what they're testing doesn't matter. And often we know that it does matter, we just don't have any more rigorous experiment to confirm it.

That's the rub. All they do is perform an experiment and offer a [very] limited amount of statistical analysis of the experiment. It's up to the reader to actually draw a conclusion [or not].

p>0.05 doesn't mean the variable being tested "doesn't matter". Nor does p<0.05 mean that the variable being tested "matters". All the p-value does is give us a rough confidence level of if and how much it matters based upon the experiment design and the tasting sensitivity of the panel.

Beyond that, you need to draw the conclusion for yourself.

 

[bob1852]

That's the rub. All they do is perform an experiment and offer a [very] limited amount of statistical analysis of the experiment. It's up to the reader to actually draw a conclusion [or not].

p>0.05 doesn't mean the variable being tested "doesn't matter". Nor does p<0.05 mean that the variable being tested "matters". All the p-value does is give us a rough confidence of if and how much it matters based upon the experiment design and the tasting sensitivity of the panel.

Beyond that, you need to draw the conclusion for yourself.

Unfortunately, I'll make the assumption that the majority of brulosophy readers think that p=</>0.05 is all that matters to be "true" or "false."

 

[HarborTownBrewing]

I wonder how some of you were able to watch the show Mythbusters. Too many variables left untested - how could they POSSIBLY attract a television audience!!!

 

[cmac62]

I wonder how some of you were able to watch the show Mythbusters. Too many variables left untested - how could they POSSIBLY attract a television audience!!!

A great show! Warning Scientific Content!! They made me laugh and think. I still use the 5 second and double dipping results, "there busted". Love it

 

[Denny]

Unfortunately, I'll make the assumption that the majority of brulosophy readers think that p=</>0.05 is all that matters to be "true" or "false."

I think you're sadly correct. I know I hear from people about our experiments that they don't want to test them themselves, they just want us to tel them what to do. That's not where we're at.

 

[brewbama]

The other day I went into my back yard, heated some water, soaked a few pounds of grain, added some hops and yeast and stuff...and tonight I am sitting on the deck drinking the result. That water somehow turned into a tasty beverage. Go figure.



Before that I read a few exbeeriments and my take away was my beer should be OK if I mash at 153 instead of 151.5, or if I ferment at 64 instead of 62.11. I didn't take away this as gospel, or the new brew dogma, or even a path to get my masters. Just some guys in a garage who like to brew and want to play around with "what ifs" for the rest of us schmucks to read or not read, believe or not believe.



My thoughts: Take your p-value argument to the science room so there is some other "science" there except "water"!



Cheers! I am going to finish my home beer! :fro:

I'll drink to this ^^^^^. Cheers!

 

[ericbw]

Yet you've read and posted here.

Heaven forbid! That was before it got soooo boring.

 

[ajdelange]

I wonder how some of you were able to watch the show Mythbusters.

I wasn't (unless promised an explosion).

.. how could they POSSIBLY attract a television audience!!!

Do you really want the answer to that? I won't give the answer in my own words but rather quote some bloke on some website from years ago who signed each of his postings with: "Consider the intelligence of the average American and then ponder that half the population is dumber than that".

Of course he should have said "median" rather than "average".

Now, of course, I recognize that everyone here was originally from Lake Wobegone...

Our education system produces graduates that perform very poorly relative to graduates from other countries (http://www.npr.org/sections/thetwo-...school-students-slide-in-math-reading-science) in STEM subjects. I guess there are those who feel we need to uphold that proud tradition!

 

[Kee]

Now, of course, I recognize that everyone here was originally from Lake Wobegone...

After reading that about three out of four drivers rate themselves as having above average skills, I'm wondering how homebrewers do at self-assessment.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3835346/

 

[slym2none]

Jesus, people, it's making beer at home....mellow out!

Yeah! Even "lazy extract brewers" make beer. No need to shame anyone.

 

[Bilsch]

Yeah! Even "lazy extract brewers" make beer. No need to shame anyone.

Sure it's beer in as much as all beer is essentially fermented barley water with hop flavorings. I kind of have standards for the beer I drink and non lazy brewer is pretty high on that list.

 

[ajdelange]

Yeah! Even "lazy extract brewers" make beer. No need to shame anyone.

When people who, when offered information that is potentially valuable to them, respond with disdain they shame themselves. Those who offer the information do not shame anyone.

 

[ajdelange]

And boring besides.

Is it really? To at least one I guess but is there some way to test that hypothesis? It occurs to me that if people are bored with the posts they aren't going to be hitting the "Like this post" button much. The posts here (449 of them) earned 503 likes (1.12 per post on average) which would suggest a pretty high level of interest to me as that seems much higher than the average for any other thread I have been involved in (but I never actually counted before).

The graph below shows the cumulative distribution over posts. Clearly greatest interest (least boredom) was exhibited over the first 50 posts which were liked, on average, 3.2 times and clearly there were some doldrums, between posts 101 and 174 and between 350 and 367 where likes came in at a rate of 0.08 per post but between post 368 and the post before this one the like rate is consistent and back up to 1.6. The 'boring' comment is about 2/3 of the way through this last segment.

By this measure the thread hasn't become more boring towards the end but less so. At least that's how I interpret it. Others?

 

[Kee]

Maybe 23 pages is not a large enough sample.