Brulosophy expert can't tell a Blonde Ale made with straight RO from RO plus minerals

[stamandster]

I think there's a severity level to "wrong/bad" if you're going to say "it's absolutely wrong/bad". There are some good things that come from the articles and they don't claim to be the end-all/be-all for information (at least from my observation). It's ok to present information and be wrong. We have to remember that they aren't getting paid, neither is the panel, etc, etc. It's not like they are figuring out to cure covid, it's just brewing beer, and the sharing of their experiences in a semi-controlled way. I feel like they should really remove the p values and just present the information as is, with comments from participants based on, "Did you like X sample? why? Did you like Y sample? Why? Did you taste a difference between the two? What did you taste as different in sample X vs Y?". This is basically what the writer does at the end with a semi-blind test.

 

[Qhrumphf]

The question to me is, while they do explicitly state that their findings are never conclusive sole data point yada yada yada, at what point do you find that disclaimer as bad faith. That they know exactly what they're doing and the message they're sending to most of their readership, but the disclaimer allows them to say to those who see through it "but we didn't say it was conclusive". Like the fine print at the bottom of a contract. Salesman bait and switch.

I don't know if that's Brulosophy or not. But I'm leaning in that direction.

 

[VikeMan]

The question to me is, while they do explicitly state that their findings are never conclusive sole data point yada yada yada, at what point do you find that disclaimer as bad faith.

I haven't seen any disclaimers of any kind in the experiment write-ups, let alone an explanation of what the numbers actually mean.

 

[Qhrumphf]

I haven't seen any disclaimers of any kind in the experiment write-ups, let alone an explanation of what the numbers actually mean.

It's never highlighted as a disclaimer. At least not in the few "xbmts" I could be bothered to suffer through reading. But every single one was caveated with language of "this is a single example so take it for what it's worth". The exact phrasing escapes me, but it was almost identical every time. And read like a disclaimer.

 

[VikeMan]

It's never highlighted as a disclaimer. At least not in the few "xbmts" I could be bothered to suffer through reading. But every single one was caveated with language of "this is a single example so take it for what it's worth". The exact phrasing escapes me, but it was almost identical every time. And read like a disclaimer.

I've read tons of them and haven't seen what you're talking about. Example?

 

[IceChisel]

Marshall Schott is a psychologist at a prison.

http://brulosophy.com/contributors/marshall-schott/
He's showing you people the ink blots... what do you see?

He's explained his point of view many times:

http://brulosophy.com/2015/09/17/exbeeriment-results-corrections/http://brulosophy.com/2017/07/20/nothing-matters-reviewing-the-first-150-exbeeriments/
Others have done things with those p-values:



Have any of you emailed this gentlemen and asked him his opinion of the "new" p-value recommendations?

Numbers are slanderous they can be read to say anything you want them to say.

 

[Qhrumphf]

I've read tons of them and haven't seen what you're talking about. Example?

http://brulosophy.com/2015/06/22/fermentation-temperature-pt-3-lager-yeast-exbeeriment-results/
That's the one that came to mind that I saw quoted everywhere at the time.

After all, as we rarely fail to point out, this is but a single point of data involving a beer produced under relatively well controlled conditions using a particular list of ingredients. It’s totally possible a different strain used in a different wort would produce more noticeable differences.

That's the caveat I saw repeated the other times in very similar wording. Though in looking it isn't as common as I thought. Though as others indicate (and the quoted text does as well), it's pointed out often.

 

[VikeMan]

That's the caveat I saw repeated the other times in very similar wording. Though in looking it isn't as common as I thought. Though as others indicate (and the quoted text does as well), it's pointed out often.

Perhaps it was common back in the day, and now, not so much. I just looked at the most recent 15, and they don't have it. It really should be in every write-up, along with about 50 words explaining what the p value actually means (besides "significant" or "non-significant")

 

[Qhrumphf]

Perhaps it was common back in the day, and now, not so much. I just looked at the most recent 15, and they don't have it. It really should be in every write-up, along with about 50 words explaining what the p value actually means (besides "significant" or "non-significant")

If they're not even paying that lip service any more, I'm even more inclined to believe when they do include it that it's in bad faith.

All the ones I actually read the whole thing were from back in the day. If it's gotten even worse since then, I'm all the more glad I disregarded them years ago.

 

[h22lude]

This is my all time favorite Bru experiment (I'm not calling it a exbeeriment). They brewed a German leichtbier and asked their tasters to tell them what style it was. Most said kolsch with a handful saying some kind of pale lager. I'll give them those. To an uneducated blind palate, it can be tough to pick a specific beer when there are a bunch of styles all very similar to each other. One person said vienna lager, which should be pretty easy to tell the difference between but that isn't the interesting part. People said it was a witbier, saison, gose, sour beer, fruit/spiced beer and autumn seasonal beer. Gose and sour beer? And these are the people they use to test their experiments.

http://brulosophy.com/2018/10/11/short-shoddy-german-leichtbier/

 

[Gnomebrewer]

Gose and sour beer? And these are the people they use to test their experiments.

This is why I'm actually a fan of a single taster with multiple tests (providing the taster is blind to the difference between the beers). If one good taster can reliably and consistently tell the difference between two beers, that's good enough for me to say they are perceptibly different. I don't need data that includes people that can't tell the difference between a leichtbier and a gose!

 

[Lupulus]

Every single Brulosophy experiment is invalid.
The way to perform a triangle test in a validated form is in Sensory Evaluation textbooks. Meilgaard is the most known.
If the experimental design is invalid, the conclusions are invalid.
Invalidated designs lead to a very high random error (stats term), and thus lots of experiments with no statistical significance.
Prost!

 

[Silver_Is_Money]

To my knowledge everyone is in the same room, and they can perhaps talk among themselves, watch facial expressions, etc... There are also to my knowledge no flavor purgers utilized. I hope they at least totally randomize cup coloration so no one can presume the smile or frown on their neighbor for the red cup to mean their own red cup is also suspect in any way.

 

[madscientist451]

For those taking potshots at Brulosophy, there's nothing stopping brewers from doing their own experiments with/without water manipulations on different styles and using different brewing methods. Then you can set up your own tasting panels and tabulate the conclusions.

Please post your results, its an interesting subject.

 

[Lupulus]

For those taking potshots at Brulosophy, there's nothing stopping brewers from doing their own experiments with/without water manipulations on different styles and using different brewing methods. Then you can set up your own tasting panels and tabulate the conclusions.

Please post your results, its an interesting subject.

I imagine you sending a research article for peer-review, getting criticized on the merits of your methods, process, and data, and responding "well, if you don't like it, you do it".
Science doesn't work this way. At any level.
Anecdotes are a dime a dozen.
Data collected this way are just anecdotes.

 

[Jayjay1976]

So, is it fair to say it's okay to mislead people, because they should know better?

Entire political careers have been built on this logic.

 

[madscientist451]

I imagine you sending a research article for peer-review, getting criticized on the merits of your methods, process, and data, and responding "well, if you don't like it, you do it".
Science doesn't work this way. At any level.

Imagining what other people may or may not do doesn't actually accomplish anything.
Your assertion that "science doesn't work that way" is merely an anecdotal opinion, which you have stated is a "dime a dozen".
Looking for differences between Brulosophy's blog posts and peer reviewed scientific research articles is an unfair comparison. The results are simply what they are for the particular beer that was being sampled. The methods used may or may not be the same if you try to use them.
I stand behind my statements: if you can do it better yourself, and you want to put the time, effort and money into the research, please post the results.

 

[Lupulus]

Imagining what other people may or may not do doesn't actually accomplish anything.
Your assertion that "science doesn't work that way" is merely an anecdotal opinion, which you have stated is a "dime a dozen".
Looking for differences between Brulosophy's blog posts and peer reviewed scientific research articles is an unfair comparison. The results are simply what they are for the particular beer that was being sampled. The methods used may or may not be the same if you try to use them.
I stand behind my statements: if you can do it better yourself, and you want to put the time, effort and money into the research, please post the results.

Actually, it's not an opinion. The process to do research and publish science is well established. Anyone that does research and publishes knows it.
Nice try on your "statement". I actually do it better. Every day. Just not in this area.
And read lots of papers of people who does real research in brewing.
Feel free to have the last word. I said what I needed to say.
Prost.

 

[Pappers_]

You are starting to talk past each other, when in fact you both agree that Brulosphy's experimentations and peer-reviewed science are not the same. Give the other members a little room, please, no need to 'win'.

 

[Beermeister32]

I think we need to appreciate that Brulosophy is there and the amount of time and expense Marshall Schott has spent building a creative and interesting website for us to enjoy. His experiments are full of all sorts of relavent information and challenging experiments in this hobby and bloggers here should quit with ripping into nuances of testing values and procedures. Just be glad the guy is there, he is doing a lot of groundwork and putting it out there in an interesting way that challenges the mind. Great job, and thanks!

 

[VikeMan]

His experiments are full of all sorts of relavent information and challenging experiments in this hobby and bloggers here should quit with ripping into nuances of testing values and procedures.

It's precisely because of the nuances that Brulosophy should change the way the data is presented. The way the write-ups are worded is misleading to casual readers (i.e. non-statisticians/non-scientists). Whether that's intentional or not, I can't say.

For many readers, the benefits of all the good stuff you mentioned is lost or potentially damaging without clear presentation. Frankly, I'm baffled by the amount of resistance to that fact. (Is it a fan-boy phenomenon?) People are literally changing (or not changing) their brewing processes based on words that do not accurately describe what the numbers mean. Would you agree in principle that the words ought to accurately describe what the numbers mean?

Note: I'm referring to "exBEERiments" here. I know there's lots of other stuff on the site.

 

[Kent88]

This thread is temporarily closed.

 

[Kent88]

This thread is being reopened.

Keep it civil. Keep it on topic.

Knock it off with the personal attacks.

 

[eric19312]

OK, I agree. Most don't understand significance. If something is "significant," it must be important, right?

I taught college-level statistics for....well, 30+ years--retired in May--and one of the best examples of this came from the Brulosophy site.

There was an early Brulosophy experiment that compared Maris Otter with 2-Row. MO is my favorite malt, so naturally I was interested.

The results of the testing showed a significant result. But if you dug down into the preferences, guess what? Exactly the same number of testers preferred MO as did 2-Row.

Even though the result was "statistically significant," there was no actionable intelligence. If I had a brewery for profit, I'd want to know what most people preferred, so I could brew it. But nothing that came out of the results would have been helpful in that way. I suppose, if it doesn't make a difference, the "actionable intelligence" is to choose the cheaper of the two.

I suppose what it indicates, if you can take it at face value, is that some people like a maltier/fuller/richer flavor (MO), and some like a lighter/crisper flavor (2-Row). Who knew that people vary in what they like?

If everybody liked the same thing in equal amounts there would be no need for more than one brand of beer. We could just all have a (bud?) and call it a day. The macro breweries could run perfectly controlled triangle tests to arrive at the ultimate beer that was seen as most preferable to all tasters and that is what everyone would drink.

Hmm doesn't work that that way right? I like a super dry IPA with some bitterness in it and zero caramel notes. My wife likes one with even more bitterness and a substantially stronger malt and caramel background. Another friend doesn't like IPAs at all. I'm pretty sure any of the three of us could pass a triangle test with our preferred beer and one of the other's preferred beer. They are noticeably different beers in aroma, mouth feel and flavor. But preference wise are my wife and I going to come to our senses and realize we don't actually prefer IPA at all and think of all the time we have been wasting on craft IPA when a nice Miller Lite was really the most preferred beer after all?

I think it is cool that Marris Otter was reliably detectable in a triangle test in opaque cups. Do you think Mecca Grade vs Rahr would pass a similar test?

 

[Unicorn_Platypus]

The worst thing IMHO is their trying to draw general, broad conclusion from a single data-point...

They usually mention the experiments are only a single data point and further testing is needed. I don't think they ever advocate to take the results from one experiment and broadly apply it

 

[Qhrumphf]

They usually mention the experiments are only a single data point and further testing is needed. I don't think they ever advocate to take the results from one experiment and broadly apply it

As we were discussing above, at what point do you concede that they know exactly how their results would be interpreted (as immutable fact "busting brewing myths" despite caveating against that, which at this point any homebrewer with an internet connection is aware of), and that rather than changing their methodology to actually fix the issue, continuing to toss in a caveat is an act of bad faith. Basically they know they're being misleading and refuse to stop. Presumably because mouse clicks and ad revenue.

 

[VikeMan]

They usually mention the experiments are only a single data point and further testing is needed.

They haven't mentioned that in the write-ups in a very long time.

 

[doug293cz]

So, what are the more acceptable alternatives to Brulosophy? That is: someone doing experiments with homebrew level equipment, not just referencing your favorite commercial brewing texts, or web sites that repeat conventional wisdom.

Brew on

 

[Kent88]

We all have access to the internet, and most of us have a lot of homebrewing equipment, along with the means to productively dispose of homemade beer.

We could all start blogs to add more points of data to the Brulosophy folks. They document their process pretty well. Their experiments can be duplicated.

 

[Big Monk]

So, what are the more acceptable alternatives to Brulosophy? That is: someone doing experiments with homebrew level equipment, not just referencing your favorite commercial brewing texts, or web sites that repeat conventional wisdom.

Brew on

An acceptable alternative would be to determine whether an alternative is even required. Do we really need someone running these types of experiments to tell us best practice in brewing, especially given the amount of great information out there for free?

My point is not to ditch a discerning eye toward making sure we don’t bias ourselves WRT brewing, but rather to be more cautious who we place our trust in, includIng myself and others.

 

[doug293cz]

An acceptable alternative would be to determine whether an alternative is even required. Do we really need someone running these types of experiments to tell us best practice in brewing, especially given the amount of great information out there for free?

My point is not to ditch a discerning eye toward making sure we don’t bias ourselves WRT brewing, but rather to be more cautious who we place our trust in, includIng myself and others.

This post is not responsive to my question. It's just another "appeal to authority" and a way of rephrasing "read the commercial textbooks," which may or may not be directly applicable in a homebrew setting.

Brew on

 

[Big Monk]

This post is not responsive to my question. It's just another "appeal to authority" and a way of rephrasing "read the commercial textbooks," which may or may not be directly applicable in a homebrew setting.

Brew on

Scrap my previous answer. I’ll separate myself from my site for a minute and say this: I’ve had good luck interfacing with people directly and using my knowledge of brewing calculations, practical experience and theoretical knowledge to help them achieve what they feel are worthwhile and essential improvements in their brewing. That’s all I (or We) can do honestly.

 

[goodolarchie]

In my club there is a resounding "Well, the brulosophy guys have proven that..." ...and they don't seem to grasp isolating and testing for one variable, vs compounding 3-4 "shoddy" methods you're precipitously, observably hurting that beer. These are folks who can definitely brew great beer, but they are given license to laziness, and the "science" gets repeated as debunked-fact. Nobody in the club is decocting anymore, but I can sure notice a difference in commercial and my own beers that were decocted.

I like that they are doing these experiments. My favorite thing they do is the Hop Chronicles, if I could donate to them doing those more often, I would. So my major contention is that the Bru crew is a sort of headwind against making increasingly good beer, honing your technique. They are like a think tank that says global warming isn't that bad because an individual commuter contributes a insignificant (p=0.07!!) amount of carbon emission, go ahead and roll coal in that 12mpg truck. Rather than taking the findings in stride (for that group of 20 it was insignificant... but we know nothing about them), they start repeating it as "debunked" gospel.

 

[Sammy86]

I've been following this thread since it started and I've stayed quiet until now...y'all are really letting a couple of guys who are doing "experiments" bother you when in reality it doesn't affect you or your brewing in anyway, shape or form.

Marshall and Crew have been more than clear from the website and podcast that their experiments should not be taken as gospel and on more than a few occasions Marshall and his Crew have said they don't normally do things the way they do in these experiments.

Another thing, for those of you talking about statistical significance you all realize you're saying that the accepted quantitative statistical significance of P(.05) that researchers use should be changed is comical and frankly ridiculous.

Maybe instead of letting these guys bother you so much maybe just maybe you should take a step back and ask yourself why you're letting it bother you so much? Do you think you can do a better job? Do you think the experiment is poorly created and executed? Do you not believe the results? If so, then do it yourself, publish your results and let your data speak. Otherwise, why let it affect you so much?

 

[enkamania]

Maybe instead of letting these guys bother you so much maybe just maybe you should take a step back and ask yourself why you're letting it bother you so much? Do you think you can do a better job? Do you think the experiment is poorly created and executed? Do you not believe the results? If so, then do it yourself, publish your results and let your data speak. Otherwise, why let it affect you so much?

Well stated. I think some folks just like to be angry.

 

[VikeMan]

Another thing, for those of you talking about statistical significance you all realize you're saying that the accepted quantitative statistical significance of P(.05) that researchers use should be changed is comical and frankly ridiculous.

Are you under the impression that p < 0.05 means there was a difference detected and that p >= 0.05 means there was no difference detected? That's not the case. Not in a Brulosophy experiment or in any experiment.

Brulosophy could choose to be very clear about what any given p value means, by stating what the odds were of achieving it if there were no difference. They choose not to do that.

When p = 0.10, for example, that means that, assuming no difference, there was only a 10% chance of getting as many correct choices (or more) as they got. Anyone who understands that would realize that it is not likely there is no difference. But what words do they use? This: "... indicating participants in this xBmt were unable to reliably distinguish ..."

@Sammy86, do you think it would be better if Brulosophy actually explained what the numbers mean in each experiment? All I'm asking for is the truth.

 

[Silver_Is_Money]

Another thing, for those of you talking about statistical significance you all realize you're saying that the accepted quantitative statistical significance of P(.05) that researchers use should be changed is comical and frankly ridiculous.

Not frankly ridiculous per the US Supreme Court in a 2011 decision. A company which claimed zinc shortens the duration of colds was about to potentially lose a bunch of money due to lawsuits piling up from those claiming as to having lost their sense of smell instead of or in addition to their cold. The companies defense in requesting that the lawsuits should be tossed out as frivolous was that numerous studies showed with P<0.05 or better statistical accuracy as proof that their zinc product did not statistically cause such loss of smell. The SC said that the statistical studies were completely irrelevant and useless and those claiming loss of smell could proceed to sue accordingly. This SC decision actually played a big part in the ASA's diligently researching and then also subsequently concluding in 2016 that P<0.05 isn't at all what it had been previously purported to be.

Some of the ASA members involved developed a math model that shows P<0.05 to fail ~29% of the time. And P<0.01 to fail ~11% of the time. This was in good agreement with claims that at least 25% of medical studies, when repeated, fail to arrive at the same conclusion as for the initial statistical studies conclusions.

Which of the following is true?
P > 0.05 is the probability that the null hypothesis is true.
1 minus the P value is the probability that the alternative hypothesis is true.
A statistically significant test result (P ≤ 0.05) means that the test hypothesis is false or should be rejected.
A P value greater than 0.05 means that no effect was observed.

If you answered “none of the above,” you may understand this slippery concept better than many researchers.

“P < 0.05” Might Not Mean What You Think: American Statistical Association Clarifies P Values

https://www.tandfonline.com/doi/full/10.1080/00031305.2016.1154108

 

[VikeMan]

Maybe instead of letting these guys bother you so much maybe just maybe you should take a step back and ask yourself why you're letting it bother you so much?

It bothers me because it misleads people, and leads them to take unwise shortcuts. One of the things I enjoy about this hobby is helping new brewers learn to make better beer. So naturally it grinds my gears when I see them being misled.

 

[Sammy86]

Are you under the impression that p < 0.05 means there was a difference detected and that p >= 0.05 means there was no difference detected? That's not the case. Not in a Brulosophy experiment or in any

Brulosophy could choose to be very clear about what any given p value means, by stating what the odds were of achieving it if there were no difference. They choose not to do that.

When p = 0.10, for example, that means that, assuming no difference, there was only a 10% chance of getting as many correct choices (or more) as they got. Anyone who understands that would realize that it is not likely there is no difference. But what words do they use? This: "... indicating participants in this xBmt were unable to reliably distinguish ..."

@Sammy86, do you think it would be better if Brulosophy actually explained what the numbers mean in each experiment? All I'm asking for is the truth.

Truth is you're proving my point...it doesn't matter what I think, they are using the current accepted research practice of P=.05 and they are clear with how many participants are required to show statistical significance based on the P value. In every single article they state the number needed to show statistical significance based on the P value of .05.

 

[Sammy86]

It bothers me because it misleads people, and leads them to take unwise shortcuts. One of the things I enjoy about this hobby is helping new brewers learn to make better beer. So naturally it grinds my gears when I see them being misled.

Again, proving my point. It doesn't affect your brewing in anyway, shape or form. And if people read the article using the same critical thinking skills they were taught in let's say middle school, then it shouldn't affect them either.