Value of brulosophy exbeeriments, others experience, myths and beliefs

[betarhoalphadelta]

Some audiophiles become so involved in the hobby that they can even hear a difference between two different sets of speaker cables. I have experienced this in my own system so I know it's true, but many people scoff at the idea.

I do accept that two different sets of cables can make a difference, but generally I think that most audiophiles go well beyond this statement into lunacy.

I think basically speaker wire falls into two categories:

1. Good enough for the power it carries and the distance of the run.
2. Not good enough for the power/distance it runs.

As an electrical engineer, I think there's really no evidence that the properties of copper wire change appreciably between two different brands given that it's of sufficient gauge to not be "lossy" over the length of the run. These things just aren't that complex.

 

[Minky]

I do accept that two different sets of cables can make a difference, but generally I think that most audiophiles go well beyond this statement into lunacy.

I think basically speaker wire falls into two categories:

1. Good enough for the power it carries and the distance of the run.
2. Not good enough for the power/distance it runs.

As an electrical engineer, I think there's really no evidence that the properties of copper wire change appreciably between two different brands given that it's of sufficient gauge to not be "lossy" over the length of the run. These things just aren't that complex.

Haha! I knew I'd get myself in trouble with that analogy, and yes, I do agree that some audio claims do qualify as lunacy. There are endless arguments about this on the audio websites, so I don't want to go there, but I too was a skeptic and I'm a believer now.

One of the arguments I've heard, in regard to speaker wire, is the size of the strands and how they're wrapped can cause different interactions between adjacent strands, much like arcing, that can cause distortions in the audio signal at different frequencies.

An example of this effect may be exhibited in the different sound of the sought-after old guitar pickups from the 50's and 60's. Current day manufacturers would attempt to duplicate them on machines with the same type and gauge wire, same magnets, same number of turns, but they just didn't have the same sound. When they hand-wound them in the imperfect, uneven way the originals were wound, they got that sweet, coveted distortion guitar players were looking for. Electrically, I'm sure they measure the same, but they sound different.

At any rate, some people will never hear this sort of subtle difference just as some of the beer tasters may not pick up on subtle differences in taste.

 

[triletter]

I've always thought the Brulosophy guys have done a pretty decent job of caveating all of their exbeeriments in almost all of their posts. The latest, covering the 150+ that they have done, went out of its way to point out these issues. These guys aren't professional scientist, but are just beer geeks asking questions and coming up with one off answers (that they repeatedly state isn't necessarily THE answer). Seems like some people just want to criticize, poke holes, burn strawmen, etc. If it's so bothersome, go ahead and establish your own methodology to be more scientifically "stringent", brew away and publish the results. From my aspect, it provides some interesting results that I may or may not incorporate. They are having fun, brewing beer they like and having some good discussions based on results that are not scientific from an academic perspective, but probably more robust that Joe Smoe saying I did this so it is the way. I look forward to their articles and mesh that with all of the other input I get from my brewing, reading and research.

Sorry if it sounds like I'm bustin' anyone's chops, but sometimes we (myself included) need to lighten up and remember, no one is going to die because of a failed chi squared, p-test or other statistical measure. It's about making better beer that WE (hobbyists) like, maybe learning something and having a bit of fun. Where does the science end and the art begin? Or vice-versa.

 

[mongoose33]

Well, it *is* a thread to discuss the issues of methodology and results. I don't know how to do that without carefully examining what they do and where the holes in the research are, if any.

The funny thing is that I think the processes of doing the exbeeriments are pretty decent. Pains are taken to make the two batches as equivalent as they can be, except for the variable at issue.

The problem is what the results mean. In the end, everybody can draw their own conclusions about them.

 

[bendog15]

I give mad props to Marshall and his team. Brulosophy is my fav brewing website. I love it and look forward to all their postings.
I started brewing beer in an area with almost zero Homebrewers. I had to learn literally everything I know about brewing from the web. Almost all my brewing skills come from brulosophy, and I wouldn’t be making killer beer today, stoking my friends and neighbors, and winning competitions without that website.
I’ve written Marshall personally and asked him questions. He always takes the time to write me back. My current process of brewing is simple, easy and straightforward. I’ve cut my brew day down to 4 hours from start to finish.
I feel that in this hobby, you get out what you put in. Brulosophy has shown me that a lot of firmly held beliefs about brewing don’t always translate to the homebrew scene, and for that I’m always gonna be grateful.

 

[jack13]

For sure. I've made these kinds of mistakes more times than I care to admit. Without going down the rabbit hole of statistical schools of thought, I will say one of the best criticisms of the use of p-values, in my opinion, is that they are often exactly the opposite of what people really want: the probability that the hypothesis is true, given the data. As noted, the p-values value assumes the hypothesis and gives the probability of the the data.

My favorite is this:

If you decide that there is an effect, then there's a 5% chance (assuming you used alpha 0.05) that you've made the incorrect decision. So, of course there must then be a 95% chance that you've made the correct decision. Since you had to have done one or the other, those probabilities have to add to 100%. How could it possibly be otherwise? But no.

 

[Nubiwan]

I honestly don't find most of the experiments to be that helpful in brewing. Absolutely nothing can replace firsthand experience. unless you use the same equipment, process, and ingredients as a particular experiment, you really cant pull any generalized rules from them.

I do see a TON of generalization online, citing the brulosophy blog as a source or proof for arguments, and giving advice for which the experiment that may be related but not necessarily applicable. Even the authors are guilty of it. my own personal bias about blogs aside, I think a lot of brewers use the blog as a substitute for personal experience, which is a mistake.

There is a TON of generelization on both sides of the "brulosophy divide" its fair to say. Whether people quote brulosophy or 100 year old brewing tradition. Neither might be accurate advice today.

By design, laziness, or accident, my process has become rather simplified. Some, thanks to stuff i have either read, or later "justified", with brulosophy experiments. Some on advice provided by the people on this forum, and reading elsewhere, which i rather wanted to believe, and or try.

In the end, most of us will only get to taste OUR beer results ourselves, and perhaps share with a few friends or family. Ive tasted other peoples home brew that was simply terrible. It was actually offered to me, so they must have thought it Ok. My beer i swapped for theirs, they could not get over. It was clearly superior. I am intrigued to see just how good other peoples home brew might taste. I have a sneaking suspicion id be criticsl.

Ive also bought craft beer in bars etc., thats just not tasted great, but they've clearly taken pains to make a presentable product. Hit all their numbers, just not a recipe i like. A commercial or craft brewer is often at the mercy of the distribution system, or bar owner, in terms of quality getting to the taste they brewed to the consumer imho, and that often fails where i live.

I think you can cut some corners in the home brew process, that are simply being overstated as important process at times. Often because the method simply mirrors that of a commercial brewer seeking consistency, more than better taste, or just because thats how everyone says its been done. Its just another anecdotal response, but ive done many of the things youre not supposed to do, based on brewers lore, and got away with still making a beer that is more than adequate. Much of it, Brulosphy experiments validate.

In its defence, everything brulosphy does is there for all to see and reference. They are completely neutral with regards the interpretation of results. Whether its pitch temps, frementation temps, mash control, their list of experiments is impressive. Beyond taking a response from the forum, which may be repeated several times as gospel, its just another source of info.

Whether right or wrong, scientific or not, i find their results quite compelling. Until someone else takes time to do the sheer volume of experiments that do, then itll continue to be so.

 

[odie]

brewing is an art, not a science...unless you desire to be the next Anheuser-Busch.

I see lots of snide comments on other threads slamming this guy. Well at least he's doing something...step up to the plate and make a better blog yourself...

I find some of his things interesting. Some so-so. Some rather irrelevant. But if nothing else, it does makes you think about...BEER...

 

[Silver_Is_Money]

Is there a means of reaching out to Brulosophy whereby to offer suggestions as to potential future exbeeriments?

 

[Holden Caulfield]

Yes, you have to pay to play, see Patreon section...

https://brulosophy.com/support/

 

[eric19312]

Is there a means of reaching out to Brulosophy whereby to offer suggestions as to potential future exbeeriments?

Probably their Patreon page would get you the most consideration. Just kidding...sort of. Am sure they are happy enough to consider suggestions but believe they get way more suggestions than they can possibly run. Haha @Holden Caulfield beat me to it.

The bottleneck isn't the cash flow though. Takes time to make the batches and effort to find a tasting panel. There are plenty of skeptics that will tell you why their experiments are wrong/garbage/useless but I've never seen the skeptics offer a better designed experiment that they ran themselves. I've read a few posts of skeptics who gave it try and then realized it is a lot harder to do than it sounds. Hope the Brulosphers are able to get back to tasting panels soon but when they do hope they keep the brewer's 10 trials as part of the evaluation.

 

[VikeMan]

They are completely neutral with regards the interpretation of results.

They are absolutely not completely neutral with regards to the interpretation of results. In experiments where they find a P-value of, say, 0.06, they use these standard words: "... indicating participants in this xBmt were unable to reliably distinguish..."

A p-value of 0.06 means that if there were actually no detectable difference, there was only a 6% chance that as many (or more) tasters would have chosen the odd beer as actually did. Does that sound to you like there's very likely not a difference?

They absolutely could be completely neutral if they instead said "Results indicate that if there were no detectable difference between the beers, there was a 6% chance of "X" <fill in the blank> or more tasters identifying the beer that was different." That would be neutral and accurate.

 

[Nubiwan]

brewing is an art, not a science...unless you desire to be the next Anheuser-Busch.

I see lots of snide comments on other threads slamming this guy. Well at least he's doing something...step up to the plate and make a better blog yourself...

I find some of his things interesting. Some so-so. Some rather irrelevant. But if nothing else, it does makes you think about...BEER...

Yep, no shortage of brulosophy nea sayers on many a brew forum.

...

The bottleneck isn't the cash flow though. Takes time to make the batches and effort to find a tasting panel. There are plenty of skeptics that will tell you why their experiments are wrong/garbage/useless but I've never seen the skeptics offer a better designed experiment that they ran themselves. I've read a few posts of skeptics who gave it try and then realized it is a lot harder to do than it sounds. Hope the Brulosphers are able to get back to tasting panels soon but when they do hope they keep the brewer's 10 trials as part of the evaluation.

And this...... indeed..... show me the money.

 

[Nubiwan]

They are absolutely not completely neutral with regards to the interpretation of results. In experiments where they find a P-value of, say, 0.06, they use these standard words: "... indicating participants in this xBmt were unable to reliably distinguish..."

A p-value of 0.06 means that if there were actually no detectable difference, there was only a 6% chance that as many (or more) tasters would have chosen the odd beer as actually did. Does that sound to you like there's very likely not a difference?

They absolutely could be completely neutral if they instead said "Results indicate that if there were no detectable difference between the beers, there was a 6% chance of "X" <fill in the blank> or more tasters identifying the beer that was different." That would be neutral and accurate...

i am not treating their reports as scientific proof. Merely validating - along with other anecdotal evidence of my own, and other posters - that a process i use can be less stringent - less critically controlled - less important - than people might otherwise want me to believe. I am the ultmate arbiter, as explained, and thats my experience.

I invite everyone to test the limits of brewing outside some of the guidelines. Im not here to argue it.

 

[VikeMan]

i am not treating their reports as scientific proof. Merely validating - along with other anecdotal evidence of my own, and other posters - that a process i use can be less stringent - less critically controlled - less important - than people might otherwise want me to believe. I am the ultmate arbiter, as explained, and thats my experience.

I invite everyone to test the limits of brewing outside some of the guidelines. Im not here to argue it.

That's cool. But that in no way supports your claim that "They are completely neutral with regards the interpretation of results," which is all that I addressed.

 

[Nubiwan]

That's cool. But that in no way supports your claim that "They are completely neutral with regards the interpretation of results," which is all that I addressed.

EDIT: Neutral in that they have nothing to gain by how they intepret the result. If their mathematical process is incorrect, then i cant comment.

They make people taste the same beer made two different ways. They have a panel of 10-30 people taste it. They try it themselves, knowing / hoping there should be a difference. Sometimes there is, sometimes there isnt. They appear to be sincere in what they attempt to do. Certainly have no cross to bare, and indeed often state they are surprised by their findings.

If the number who can tell a difference is nearly half, or even a third, i might think its significant. More significant than the Brulosophy folks do at times. Its up to me what i decide, and how i interpret their findings. How I decide to proceed.

Few other websites offer the same levels or array of home brew testing experiments. Few on the HBT forum can offer more compelling evidence than their own experiences. I see no difference in using either as a source of information.

 

[VikeMan]

They make people taste the same beer made two different ways. They have a panel of 10-30 people taste it. They try it themselves, knowing / hoping there probably should be a difference. Sometimes there is, sometimes there isnt. They appear to be sincere in what they attempt to do. Certainly have no cross to bare, and indeed often state they are surprised by their findings.

If the number who can tell a difference is nearly half, or even a third, i might think its significant. More significant than the Brulosophy folks do at times. Its up to me what i decide, and how i interpret their findings. How I decide to proceed.

Few other websites offer the same levels or array of home brew testing experiments. Few on the HBT forum can offer more compelling evidence than their own experiences. I see no difference in using either as a source of information.

Ok, if you're not going to address what I actually said, or answer the easy yes/no question that I put to you, there's really no reason to discuss any further. Peace out.

 

[Nubiwan]

Ok, if you're not going to address what I actually said, or answer the easy yes/no question that I put to you, there's really no reason to discuss any further. Peace out.

I did edit my comment above......

 

[VikeMan]

Ok, if you're not going to address what I actually said, or answer the easy yes/no question that I put to you, there's really no reason to discuss any further. Peace out.

I did edit my comment above......

EDIT: Neutral in that they have nothing to gain by how they intepret the result.

Good, now we're getting somewhere. Do you think it's possible that by not making sure that readers know how to correctly interpret the P-values (which most do not), the result is that many of those readers take the words "... indicating participants in this xBmt were unable to reliably distinguish..." as an indication that there's likely no difference? If not, why not?

 

[dmtaylor]

Brulosophy is cool. They come up with some good experiments on occasion. I respect their effort and their raw data. However I do not agree with their choice of wording in their interpretation of their results. Which I am free to do. We can all interpret the results however we see fit. It might however be nice if they would present their results in a more factual manner without any slant to the "unable to reliably distinguish" bit, which is a misrepresentation.

 

[Nubiwan]

Good, now we're getting somewhere. Do you think it's possible that by not making sure that readers know how to correctly interpret the P-values (which most do not), the result is that many of those readers take the words "... indicating participants in this xBmt were unable to reliably distinguish..." as an indication that there's likely no difference? If not, why not?

They often validate tests themselves and, in general, agree with the panels findings. Id trust those perceptions, knowing there should be a difference, more than someone on a forum suggesting you'll make "paint thinner" if you dont ferment lager at 35 degrees. Its happened. .

Each experiment is different, so you have to weigh them independently. How the results are used is up to the individual. I can only go back to my own brewing results, and weigh it with other peoples comments. And personal results.

At the very least, they do the experiments and publish results in an open manner. Nobody else does it.

 

[VikeMan]

@Nubiwan, do you think it's possible that by not making sure that readers know how to correctly interpret the P-values (which most do not), the result is that many of those readers take the words "... indicating participants in this xBmt were unable to reliably distinguish..." as an indication that there's likely no difference? If not, why not?

Yes or No would be good for starters.

 

[Nubiwan]

@VikeMan do you think the folks over at brulosophy are intentionally trying to fleece people wit p-values, or help them in their brewing process? I believe its the latter, and its interesting.

 

[VikeMan]

@VikeMan do you think the folks over at brulosophy are intentionally trying to fleece people wit p-values, or help them in their brewing process?

No, I think the P-values are probably honest. I think they are misleading people (intentionally or unintentionally) by not explaining, in the experimental writeup, what the P-value means (and doesn't mean). Instead, they use these words " ... indicating participants in this xBmt were unable to reliably distinguish..." even when the P-values (if they were to be understood by the reader) clearly show that it was more likely than not a difference was detected.

Consider, theoretically, 100 experiments where the P-Value = 0.10. Every one of those would get the words " ... indicating participants in this xBmt were unable to reliably distinguish..."
But statistically, there is an overwhelming probability that approximately 90 out of those 100 experimental results were the result of a difference being detected. Let that sink in.

Yet a lot of readers are left with the impression that the experiments showed that there was no difference. We know that they do, because of the many forum threads where folks cite the results as proof of something not making any difference. And it's perfectly reasonable for them to think that, given the language used, and the language not used.

Do I think Brulosophy is trying to help people in their brewing process? Possibly, but they could be a lot more helpful by making the meaning of the results clear.

Given all of the above, do you think it's possible that readers tend to keep coming back for "validation" that lots of stuff makes no apparent difference?

 

[Vale71]

There are plenty of skeptics that will tell you why their experiments are wrong/garbage/useless but I've never seen the skeptics offer a better designed experiment that they ran themselves.

Maybe those skeptics are aware that you can't really devise meaningful, effective experiments using simple homebrew equipment but unlike this guy they don't have a vested (possibly commercial) interest in pretending that you actually can...

 

[eric19312]

@VikeMan did you wake up on wrong side of Monday?

I'm sure Marshal and company have discussed this language with numerous experts (and a lot of armchair experts AKA homebrewers) though there are likely some rare cases of overlap in those two communities) in the field and are satisfied the "reliably distinguish" is reasonably descriptive. Have listened to the discussion on more than one podcast. But I still found you underlined thought experiment intersting.

So I went to the current front page of the exbeeriments the 50 most recent exbeeriments are listed. Of these 30 had p values greater than 0.05. Of these 30 exbeeriments that I assume have the "not able to reliably distinguish" only 3 had P values that calculated between 0.10 and 0.05. Now the two of these I checked (one was done by a club member not a regular contributor) I saw the brewer did the test 10 times and need to guess right 7 times out of 10 to hit 0.05 p value. In these two cases he was right 6/10 times for a p value of 0.08. Reading into the further comments neither time did the author say "well I thought I had them figured out but just missed on that last sample"... Nope in both cases the author admitted the beers were virtually indistinguishable.

What I get out of brulosophy is that taste and smell are blunt instruments. Lots of times the beers are visually different (today's exbeeriment eg) or measurably different in final gravity or attenuation, but still not easily distinguished in opaque cups. It's probably a good thing that our brains work this way or else we might all starve to death rejecting food that doesn't exactly match our expectations. Makes sense that restaurants put so much effort into plating the food....

 

[Silver_Is_Money]

My perspective is that they are clearly beginning to struggle and grope for relevant topics, and they are hitting upon topics with less and less relevance and more an more outright desperation based irrelevance as time marches forward. As a for profit organization, that is no way to run a business.

 

[odie]

My perspective is that they are clearly beginning to struggle and grope for relevant topics, and they are hitting upon topics with less and less relevance and more an more outright desperation based irrelevance as time marches forward.

well, probably because brewing is a fairly simple process that we are all hell bent on making complicated.

Just like burgers...there are only so many different topping combos...but in the end it's just a freaking burger

 

[Nubiwan]

No, I think the P-values are probably honest. I think they are misleading people (intentionally or unintentionally) by not explaining, in the experimental writeup, what the P-value means (and doesn't mean). Instead, they use these words " ... indicating participants in this xBmt were unable to reliably distinguish..." even when the P-values (if they were to be understood by the reader) clearly show that it was more likely than not a difference was detected.

Consider, theoretically, 100 experiments where the P-Value = 0.10. Every one of those would get the words " ... indicating participants in this xBmt were unable to reliably distinguish..."
But statistically, there is an overwhelming probability that approximately 90 out of those 100 experimental results were the result of a difference being detected. Let that sink in.

Yet a lot of readers are left with the impression that the experiments showed that there was no difference. We know that they do, because of the many forum threads where folks cite the results as proof of something not making any difference. And it's perfectly reasonable for them to think that, given the language used, and the language not used.

Do I think Brulosophy is trying to help people in their brewing process? Possibly, but they could be a lot more helpful by making the meaning of the results clear.

Given all of the above, do you think it's possible that readers tend to keep coming back for "validation" that lots of stuff makes no apparent difference?

Well, when i read the experiment reviews, i pay no mind to p-values, its foreign gibberish to me. i simply look at the number that are testing, and how many can tell a difference. Then i look at of those who can tell a differene, which beer they prefer. Ultimately, the number narrows down to tell me that, even when there is a detectable difference, some still prefer the non traditional approach, which starts me wondering, whats the big fuss? When the brulosphy guys have a difficult time telling the difference, then i am sorry, but i think thats pretty telling.

In the end, who is making the big fuss over Brulosphy here?

 

[RichBenn]

I was a black belt in engineering management and a stint as a six-sigma blackbelt(think statistical modeling) before I retired. That may give me an ad

I use Brulosophy alot. Like Homebrewtalk, it's helped me improve my beers. Yes, it isn't perfect, but who else is actually doing experiments with some controls and statistical significance. He needs our support for that.

I have may have some practical advantage over some at looking at the data due to understanding P values, lurking variables, process control, how to set up experiments and the like, but as long as one looks at them as testing over a narrow range of conditions, one can learn alot. Having brewed since 1981, I have been surprised by how many things in books and literature we have taken as fact and reposted that have been debunked over the years. The folks over at Brulosophy have contributed to some of those.