applescrap
Be the ball!
Excellent, agree very much with you and I think you shed some light on some of the differences between pros and homebrewers.
Do "professional" brewers consider brulosophy to be a load of bs?
- Thread starter BrewVerymore
- Start date
Help Support Homebrew Talk - Beer, Wine, Mead, & Cider Brewing Discussion Forum:
- Status
Excellent, agree very much with you and I think you shed some light on some of the differences between pros and homebrewers.
Our sensory instructor pointed out that the earlier your beer is tasted in a trial, the better your score is likely to be....order matters.
A perhaps more intuitive explanation as to why a triangle (or quadrangle) test is better than a binary test:
Suppose you have a panel of 60 guys 30 of whom are BJCP masters and 30 of whom can't tell the difference between Bud Light and Pilsner Urquell ("All yellow fizzy beers to me" as a colleague used to say). In the binary test every one of those gets to vote as to whether one beer is better than the other so the panel is has 60 members of whom half (50%) are unqualified. In a triangle test 2/3 of the unqualified (40) will be eliminated and the panel now consists of 40 people 10 of whom are unqualified with 30 qualified and the percentage of qualified voters is now 75%.
Suppose you have a panel of 60 guys 30 of whom are BJCP masters and 30 of whom can't tell the difference between Bud Light and Pilsner Urquell ("All yellow fizzy beers to me" as a colleague used to say). In the binary test every one of those gets to vote as to whether one beer is better than the other so the panel is has 60 members of whom half (50%) are unqualified. In a triangle test 2/3 of the unqualified (40) will be eliminated and the panel now consists of 40 people 10 of whom are unqualified with 30 qualified and the percentage of qualified voters is now 75%.
- Joined
- Jun 2, 2008
- Messages
- 64,951
- Reaction score
- 16,516
It's been awhile since I reviewed their methodology, but does Brulosophy keep the order of the triangle tests the same for each taster, or do they randomly assign order?
That would be interesting to look at in repeated tests....can those who correctly identified (or guessed right) the odd beer out repeat that result if the order of samples is different?
applescrap
Be the ball!
Good point.
applescrap
Be the ball!
Aj and mongoose, awesome work, I hope to understand your discussions better some day.
I've had to be cautious dry hopping. Most of the time 3-4 days. However some seem to have an issue with renewed fermentation after dry hopping (and research is shedding light on it, Cascade seemingly a hop to do it among others) that if we crash out prematurely may cause VDK to pop up down the line. Not as big an issue in the tap room where it can be monitored but we can't have kegs go out to distro like that. And obviously I'd rather not have it at all.
So yes. Paying attention to research.
- Joined
- Jun 2, 2008
- Messages
- 64,951
- Reaction score
- 16,516
Makes me sad that I only took one stats class in college. Who knew it would be so useful!
Fascinating discussion of the statistics aspect by ajdelange and mongoose - I appreciate all this. But my own “testing” consists of making a specific change from what I did on a similar brew a couple of months ago and noting which one I liked better. Regardless of the statistical validity, the Brulosophy method is many orders of magnitude better than my “tests”, so I pay close attention.
Makes me sad that I only took one stats class in college. Who knew it would be so useful!
And fun.
I'll comment:
That is really up to you to decide. It is incumbent on the investigator to report what he measured and what those data imply in terms of the null hypothesis but is up to you to decide if you wish to reject it based on what was measured. People who do this for a living usually don't accept p > 5% but if you are convinced at p > .15 or .20 then that's OK. It's just that most people won't agree with you.
It is also up to you to decide whether the null hypothesis tested against is valid. Remember that it is "This panel can't tell the difference between these beers". If you think the panel is biased or unqualified or that difference has been telegraphed to it or... through improper procedure or panel selection then you don't really care what the confidence level is because, in your opinion, the test is invalid.
Take out "to match the scienciness of these experiments" and you are spot on. That is exactly what a close to significance level tells you. That something might be going on but you didn't measure it to statistical significance because your panel wasn't big enough. Before one can draw even that conclusion, however, he must be convinced that the methodology was sound. Marginally high confidence in the data from a flawed experiment is a useless as high confidence in the data from a flawed experiment.
AJ, thanks for the thoughtful response.
My opinions and further insights:
Personally I am pretty confident that Marshall & Co. are good brewers -- maybe even very good. While I've never tasted their beers, I also have no reason to doubt their experience and mastery of their processes, and they sure seem to keep better control of things than I would, plus I know they've won awards in competitions and are generally well respected by all who know them. Plus, I have interacted with Marshall several times and he seems to me to be a very level-headed and like-minded individual.
What I question personally is whether we should place much validity in their results when they expect John Q. Randomguy -- who might know little or nothing about beer -- to be able to detect differences between two beers at a 95% confidence level. But in my view, if we take a more loose approach and only expect John Q. as well as all the other various experienced tasters to detect a difference an average of maybe about 80% of the time, with the ultimate goal being, "MAYBE, JUST MAYBE, there is something going on here", rather than expecting a result of "yea, verily, this experiment has 95% confidence that there seems to be a difference", then with an 80% bar instead of 95%, this lower bar is easier to meet or to "qualify" for further experimentation, rather than rehashing the same old "nope, we didn't achieve 'statistical significance' yet again". Statistically, if they only expect to be right about 80% of the time instead of 95%, the results reported should prove more interesting, at least in my own chumpy eyes.
Shameless admission: I actually review closely the p value achieved for each and every Brulosophy xbmt with a goal of about 0.15-0.20 as explained above so that I can find gold dust beneath the lack of gold nuggets from the conclusions that they often present to us. The data is there; I just have to interpret it myself.
I believe you and I are singing somewhat the same tune, just that you're more like Luciano Pavoratti and I'm more like the chump humming along to it and bopping his head with the tempo.
Cheers again all.
My opinions and further insights:
Personally I am pretty confident that Marshall & Co. are good brewers -- maybe even very good. While I've never tasted their beers, I also have no reason to doubt their experience and mastery of their processes, and they sure seem to keep better control of things than I would, plus I know they've won awards in competitions and are generally well respected by all who know them. Plus, I have interacted with Marshall several times and he seems to me to be a very level-headed and like-minded individual.
What I question personally is whether we should place much validity in their results when they expect John Q. Randomguy -- who might know little or nothing about beer -- to be able to detect differences between two beers at a 95% confidence level. But in my view, if we take a more loose approach and only expect John Q. as well as all the other various experienced tasters to detect a difference an average of maybe about 80% of the time, with the ultimate goal being, "MAYBE, JUST MAYBE, there is something going on here", rather than expecting a result of "yea, verily, this experiment has 95% confidence that there seems to be a difference", then with an 80% bar instead of 95%, this lower bar is easier to meet or to "qualify" for further experimentation, rather than rehashing the same old "nope, we didn't achieve 'statistical significance' yet again". Statistically, if they only expect to be right about 80% of the time instead of 95%, the results reported should prove more interesting, at least in my own chumpy eyes.
Shameless admission: I actually review closely the p value achieved for each and every Brulosophy xbmt with a goal of about 0.15-0.20 as explained above so that I can find gold dust beneath the lack of gold nuggets from the conclusions that they often present to us. The data is there; I just have to interpret it myself.
I believe you and I are singing somewhat the same tune, just that you're more like Luciano Pavoratti and I'm more like the chump humming along to it and bopping his head with the tempo.
Cheers again all.
ericbw
Well-Known Member
- Joined
- Dec 11, 2012
- Messages
- 3,592
- Reaction score
- 1,225
ericbw
Well-Known Member
- Joined
- Dec 11, 2012
- Messages
- 3,592
- Reaction score
- 1,225
2 days of dry hopping has no effect. But it was the best IPA he's made. Therefore, less hop contribution is better.
Evidence that IPAs just aren't good?
Makes me sad that I only took one stats class in college. Who knew it would be so useful!
never got into stats, more of a probability man myself...
'draw five cards from a shuffled deck, what is the probability that two are red and one is a seven?'
factorials, ftw!
They go together.
About some of it, no. But about qualifying for panels, absolutely.
I've never had a problem including guessers in the triangle test statistics. That's what the test evaluates, whether the numbers are greater than what one would expect by pure guessing.
Here's why I do think that, and it's not a statistical reason, it's a measurement reason. It flies in the face of common sense that one would use, in a test of preference, people who demonstrably cannot make a preference decision.
They're guessing!
You want people who demonstrably can make that distinction doing such preference evaluation. There are other ways, better ways, to find such people, including seeing if people who choose the odd one out can do so repeatedly before moving them on to the preference test.
It's simply a reliability issue, nothing more.
Before one of the mods terminates this thread, I want to commend the OP for constructing one of the best loaded titles I've seen on HBT 
Seriously, that's art right there
Seriously, that's art right there
Not only is that not clear from the way they do it, you point out one of the elemental difficulties with having a one-shot guess "qualify" tasters for the preference test.
Show me you can pick the odd-one-out three or more times in a row, and I'll believe you can detect a difference....and you are qualified to go to the next level.
Guessers cannot tell the difference; why would anyone want them judging preference, and guessing on that too?
The 95% confidence limit requires tasters to be right far far less than 80% of the time. If you have 30 tasters for example, you only need 15 to pick the right sample to be 95% confident they didn't pick right by accident. If the simply guessed you would expect to get 10 right answers and 20 wron answers. If 80% chose correctly and you only had 8 tasters you would be 99% certain this was not lucky guessing. If you had 14/20 tasters get right answer...that's not 80%... you have p value of .001
- Joined
- Jun 2, 2008
- Messages
- 64,951
- Reaction score
- 16,516
Before one of the mods terminates this thread, I want to commend the OP for constructing one of the best loaded titles I've seen on HBT
Seriously, that's art right there
Why would they terminate this thread?
I understand your point. My point is that given the unknown variables about how qualified the tasters are, as well as perhaps how well the beers were brewed, if they'd just compensate for these additional unknowns by expecting only to kinda sorta find out if maybe the variable is having an effect by setting confidence to just 80% (p<=0.2), then for the same 30 tasters, instead of needing 15 people to select the right beer to reach 'statistical significance', you'd only need like 13 of them to be 80% confident that the result might be showing us something worth further experimentation. To me, 80% confidence is a pretty decent goal for us silly homebrewers. We are not scientists working in laboratories; this is just beer brewed at home. If the experiment is screwed up, then drink it and run it over again, no big whoop. We don't need to set a 95% confidence goal all the damn time. That's my point. Sure, if you could get 100 tasters for every experiment, or better yet, 200, then 95% confidence would be achievable. But to expect 95% confidence with a little sample size of only 20-30 people? Not so realistic.
Realism. That's what it boils down to for me. Goals need to be realistic. Otherwise we usually get the same result: "did not achieve statistical significance with a goal of p<=0.05". If the final result is p=0.06, or 0.1, or even 0.15, could there be something going on? This is really never talked about by M&Co, they just say "not statistically significant", which really misses the point IMO.
Makes me sad that I only took one stats class in college. Who knew it would be so useful!
I liked my stat class in college so much, I took it twice!
(Who the F*** schedules statistics for 7:30 AM anyway???)
This is what bothers me *NOT* about Brulosophy, but how some people interpret the results.
It sounds like some people say "Oh, this experiment failed to achieve significance? I guess I'll remove that step from my process! Woo time saved!"
But that's not a viable conclusion. As I tried to show in the meta-analysis of fermentation temp, few of the fermentation temp experiments achieved significance. However, notable was that there was NEVER an experiment where the triangle test was <33% at picking out the odd beer. A number of the tests were very close to significance. And if you did a meta-analysis (which is dangerous because it's different experiments), it suggested that the significance was stronger than individual panels could provide.
So yes, if it is close to p<0.05 significance, ONE possible takeaway is that there is an effect and that the panel wasn't large enough. ANOTHER possible takeaway is that guessers might have screwed it up. That's the problem with going to p<0.15 as your significance test. But if 7 experiments all occur and they're almost ALL "close" to significance, that's harder to wave away as the variable having no effect due to not achieving significance in a single experiment.
The good news for @dmtaylor is that he can figure this out on his own. Brulosophy publishes the number of testers in each panel and the number who correctly choose the odd beer out. If he prefers a lower significance value than p<0.05, he can actually quite easily calculate what the significance is of each experiment. (Truthfully, I don't think he even has to... I think Brulosophy provides the p-value of every experiment...)
seabrew8
Well-Known Member
I have had excellent results using saflager at ale temps. I'm in no way a perfectionist when it comes to beer but its what i've been looking for since i started homebrewing. I've burned though probably over 10, 55lb sacks of grains in a year and a half! I buy nothing but CMC pilsner sacks now for my most of my bases.
I love munich helles type beers. They are the best beer i ever made, my friends say the same.
The only ales i make now are wheat and ambers. All my pale beers with 34/70. I haven't tried any dark ales. I'm a session beer guy mostly.
The biggest "mistake" i made when first starting out was making all kinds of smash beers to "figure out" the ingredients. I think to avoid a steep learning curve people should avoid smash beer unless its in the style guidelines like some bohemian pils.
Made some decent beer but when i started using jamils book and recipes online my beer took a big leap.
I love munich helles type beers. They are the best beer i ever made, my friends say the same.
The only ales i make now are wheat and ambers. All my pale beers with 34/70. I haven't tried any dark ales. I'm a session beer guy mostly.
The biggest "mistake" i made when first starting out was making all kinds of smash beers to "figure out" the ingredients. I think to avoid a steep learning curve people should avoid smash beer unless its in the style guidelines like some bohemian pils.
Made some decent beer but when i started using jamils book and recipes online my beer took a big leap.
I suspect you'll find little disagreement from anyone here. They approach things with as much scientific and statistical rigueur as they can and seem to be very competent brewers. The suggestions raised in this thread I think they would wholeheartedly agree with. But are probably for the most part out of their scope. The issue isn't Brulosophy, it's the giving more credit than is due to the results.
Give each of 20 guys 3 pingpong balls, one of which has a black mark on it, concealed in black bags and ask them to pick two of the three bags randomly and then randomly pick one of the two. What's the probability that the chosen ball will be the one with the black mark? The answer to that question is the confidence, p, we have been speaking of here repeatedly.
factorials, ftw!
p = ∑[ j= N to M](1/3)^j*(2/3)^(N-j)*M!/(j!*(M-j)!
is the probability that N or more out of M choose the black ball in the first step i,e. the confidence level. IOW this part of statistics (Hypothesis testing) is concerned with computing the probability of one of the hypotheses given the data observed. And given that observations are often integers (or at least limited to a non continuous set of possible numbers) combinatorics and thus factorials come into play.
ericbw
Well-Known Member
- Joined
- Dec 11, 2012
- Messages
- 3,592
- Reaction score
- 1,225
Before one of the mods terminates this thread, I want to commend the OP for constructing one of the best loaded titles I've seen on HBT
Seriously, that's art right there
True dat!
I understand your point @dmtaylor but I'd argue that in the 13 out of 30 example where you would choose to say study suggest a trend ... and maybe it does .... am am troubled that a clear majority of the tasters failed to pick the right sample.
ericbw
Well-Known Member
- Joined
- Dec 11, 2012
- Messages
- 3,592
- Reaction score
- 1,225
Why would they terminate this thread?
I said that because very once in a while, a mod steps in when people get too excited/passionate OR call names like fool or chump.
This isn't the internet, it's a private forum, so they can enforce the rules they want to maintain a civil environment.
ericbw
Well-Known Member
- Joined
- Dec 11, 2012
- Messages
- 3,592
- Reaction score
- 1,225
I haven't read their fermentation temp experiment, but that seems like a critical part of the process from my experience - depending on the yeast. S-04 works fine in the mid-60s, but is gross if it goes over 70. I also think you can coax different characteristics out of some yeasts by changing the temperature. I have a Belgian yeast that seems to get a lot more barfy (Belgian) at higher temps and is more mellow fruity at lower. I've never done a side by side or even the same beer twice to compare, though.
Do you control temp at all to keep it in ale range, or just let it do what it does?
13/30 versus 10/30 is still MAYBE (~80% confidence) better than random guessing. The majority thing doesn't bother me. Sometimes maybe "maybe" is "good enough". To each our own.
applescrap
Be the ball!
Really, you are still holding on to your brew dogma for dear life. Trying your hardest to mathematically make your position have strength when none is there.
Let me ask you this, how many more times do they need to test fermentation temp for you? Now they have done it eight times by three people in three different states. The last one compared 48 degrees and 72. 24 degrees different. Only eight of 20 people could tell the difference if they even could. The person who made them could only get a triangle test right two of four times and anecdotally felt certain they tasted the exact same. But somehow that's not good enough for you or the results from the other 7 tests aren't good enough for you to make an opinion. And I'm just some sort of "fool" for going along with these results.
So how many more times do they need to test this? Really gives us all a number.
That last test was wyeast 2124, they have used wlp800 as well. So they've used at least three different yeasts now. Furthermore the meaningless preference data in this case was once again prefer the warm ferment. In the 82-degree experiment the preference was warm ferment.
So really bwarbiany, explain why it would be so foolish, to come to the conclusion that fermentation temperature isn't as big of a deal as you make it? What's the reasoning, because you read it in a book somewhere, because someone you really trust told you it mattered, because it's just something you think, because you think that's what taught in College. Once again where's your data other than somebody said it to be true. They made the beers and they tasted them blind in a triangle that's how it's done. There really isn't another way to do it. Oh I guess you know we could measure it with a spectrometer or something. Needing a million-dollar machine to discern a difference doesn't make a difference to me.
You need to boil 90 minutes right? Well how come in the two tests he did there was no DMs. And in that test he did send it to a lab and there was no DMs. How come the boil with the lid on experiment didn't come back significant? How come the weak verse strong boil didn't come back significant? How come people couldn't reliably tell the difference between Whirlpool Hops and flame out additions? Is it all just a bunch of BS and you have the real answers, or is there a chance that an overemphasis on process considerations has skewed your perception into tasting and believing things that don't exist. Is there a chance that maybe some of the stuff just doesn't matter and the real answer to better making beer lies elsewhere,
The excitement and suspense keeps us all coming back.
I am not a statistician, but I know a guy who knows *some* stuff... and he thinks a sample size of roughly 100-150 is appropriate for 95% confidence. Below 50 samples, he expects only 80% confidence in any result.
So the answer to scrap's question, given my slightly-more-than-zero knowledge of the topic, is that we should keep testing until we get around 100 tasters involved, IF you want 95% confidence in the result. Less tasters than that, and the confidence should not be as high.
That's my 2 cents. Cheers.
I am not a statistician, but I know a guy who knows *some* stuff... and he thinks a sample size of roughly 100-150 is appropriate for 95% confidence. Below 50 samples, he expects only 80% confidence in any result.
So the answer to scrap's question, given my slightly-more-than-zero knowledge of the topic, is that we should keep testing until we get around 100 tasters involved, IF you want 95% confidence in the result. Less tasters than that, and the confidence should not be as high.
That's my 2 cents. Cheers.
Because I did a similar experiment. I brewed 15 gallons of an IPA. I kept 10 gallons for myself, fermented in a temperature-controlled chamber. I gave 5 gallons to a fellow homebrew club member, which he fermented without any temperature control. We then presented the two beers to our homebrew club at a meeting, and had them evaluate them to BJCP guidelines.
The temp-controlled beer had an average score 11 points higher than the non-controlled beer.
applescrap
Be the ball!
To me the most mind-blowing experiment is the triple decoction. It is incredible to me that people were unable to reliably tell the difference between a single infusion Mash and one where one-third of the mash was taken out and boiled three times. Basically even boiling the mash three times didn't create a difference, that and the first Mash temperature experiment of 146 and 161 being non-significant makes a serious argument about the importance of mash temp.
The reason why I'm bringing this up is I noticed in the discussion that he said people felt that if people were told what the variable is they would be better. That has been brought up on this thread. So they went ahead and told people the variable in a separate group of 22 tasters and only 8, p.46, got it right. But man oh man the merits of triple decoction are splattered across this website and the people who make them will tell you how amazing they are comparatively. I have been accused, not directly (thanks), of being foolish or whatever for blindly trusting these experiments. While I don't blindly trust them a hundred percent, they paint a very interesting picture. While I won't go on record and say that triple decoction doesn't matter, based on all this information, I just won't make one figuring the difference isn't that big. If the difference was so massive there would have been significance in at least one of the two times they tested this on people, especially having given people the variable. So while it may be very well true that there is a difference, based on this I will adjust my Brewing practices a little. I don't need to see more than one test. I have no problem trusting this information and as of now it Still Remains the only data that's been offered. As homebrewers we all owe a thank you to brulosophy, whether you believe them or like them or not. For free at least they're doing something and testing and challenging thought.
The reason why I'm bringing this up is I noticed in the discussion that he said people felt that if people were told what the variable is they would be better. That has been brought up on this thread. So they went ahead and told people the variable in a separate group of 22 tasters and only 8, p.46, got it right. But man oh man the merits of triple decoction are splattered across this website and the people who make them will tell you how amazing they are comparatively. I have been accused, not directly (thanks), of being foolish or whatever for blindly trusting these experiments. While I don't blindly trust them a hundred percent, they paint a very interesting picture. While I won't go on record and say that triple decoction doesn't matter, based on all this information, I just won't make one figuring the difference isn't that big. If the difference was so massive there would have been significance in at least one of the two times they tested this on people, especially having given people the variable. So while it may be very well true that there is a difference, based on this I will adjust my Brewing practices a little. I don't need to see more than one test. I have no problem trusting this information and as of now it Still Remains the only data that's been offered. As homebrewers we all owe a thank you to brulosophy, whether you believe them or like them or not. For free at least they're doing something and testing and challenging thought.
applescrap
Be the ball!
applescrap
Be the ball!
Were the judges blind to the variable? Did they know which was which? How careful were you to make sure that none of them knew which was which? I think it would be cool if you did that again and did a blind triangle test. I suspect ale yeast might be more reactive to temperature than lager, just a hunch.
- Status
Similar threads
