Designing a Taste Test for BMC

[nickharbour]

Apologies if this has come up before, my search abilities are failing me.

A majority of my coworkers are under the impression that they can discern the nuanced flavor differences between Bud Lite, Miller Lite, and Coors Lite in a blind taste test. Lets just say they are not BJCP judges but they do at least appreciate good beer when they have it. They take challenges very seriously so there will probably be money riding on this test.

I would like to design a test to identify if a person can identify at least one product from the three with some statistical significance. How would you define statistical significance in this case? I am not math illiterate but I took more discrete math classes in college than statistics classes.

Any pointers on designing the test at all would be helpful!

 

[TyTanium]

First off, you'll need a decent sample size.

Also helpful to distinguish between false positive & false negative.
False positive - Think it's Bud, but it's not
False negatives - Don't think it's Bud, but it is

So you'll have 3 possible answers
Wrong - FP
Wrong - FN
Correct

Then you have to decide how you'll present beers to them. Groups of 3, one of each? Groups of 3, random (each one could be any of the 3)? One at a time or sequential? Since you're trying to see if they can tell one from the other, I think I'd do Groups of 3, random. That way they can't use process of elimination to deduce the 3rd if they know 2. With 3 random, they have to recognize the taste of all 3 samples.

For added complexity, you may throw in random alternatives (PBR, Keystone, etc) to see if they really know their BMC's, but that doesn't sound like what your stated objective is.

For statistical significance, this is where you need a larger sample size. Maybe have 10 people take the same test, and calculate an average "correct" response rate for the 10 people, and the standard deviation of those "correct" response rates. Then you can see if any one person's "correct" response rate is more than 2 standard deviations above the mean. You could do it separately for each beer and in aggregate.
E.g. Bud has the highest "correct" rate, and Jim was significantly better at guessing Bud, though Jack was significantly better at guess all beers. Miller had a significant number of false negatives.

 

[LandoLincoln]

Ask them which lite beer is their favorite, and then have them try to figure out which of the three beers is their beer of choice.

 

[JJL]

It's been a while since I did any research or statistical analysis on data sets, but IIRC you are dealing with discrete variables, which limits the numbers of statistical analyses you can perform. Meaning each person is either one thing or another. There are exactly 3 beers to choose from. There are exactly 3 possible answers that can result from tasting each beer. Ty listed them above - FP, FN or correct. IIRC, the analysis you are looking to do to determine significance would be a Chi Squared. This will tell you if there is a significant difference between the number of FP's, FN's, and corrects for each of the beers. Also, you are going to want to keep track of some very simple descriptive statistics, like what percentage of the people correctly identified each of the different beers vs. FP's vs. FN's.

 

[JJL]

As an aside, they did basically this same type of test in the movie Beer Wars, and basically no one could correctly identify which beer they were drinking with any level of consistency.

 

[weirdboy]

Ask them which lite beer is their favorite, and then have them try to figure out which of the three beers is their beer of choice.

Well I dunno about the "light" versions per se, but I have always found Coors pretty easy to distinguish because of the distinctive "corny" flavor.

 

[Bobby_M]

Since you can't have more people for bigger sample size, I'd just do it for fun. I agree that you'd want to ask some questions first. Which is your favorite beer between Bud, Millerlite, and Coors Light. Also ask if they think they can tell which is which out of all three. Pour four numbered cups and have one of them be doubled. That would test if they can even identify which two are the exact same beer. Probably not.

 

[Fritobandito]

I did a blind taste test between Bud Light, Coors Light, Miller Light, and Busch Light with a friend of mine and his girlfriend. We all agreed Coors Light was by far the worst and easiest to pick out.

 

[nickharbour]

First off, you'll need a decent sample size.

Also helpful to distinguish between false positive & false negative.
False positive - Think it's Bud, but it's not
False negatives - Don't think it's Bud, but it is

So you'll have 3 possible answers
Wrong - FP
Wrong - FN
Correct

Then you have to decide how you'll present beers to them. Groups of 3, one of each? Groups of 3, random (each one could be any of the 3)? One at a time or sequential? Since you're trying to see if they can tell one from the other, I think I'd do Groups of 3, random. That way they can't use process of elimination to deduce the 3rd if they know 2. With 3 random, they have to recognize the taste of all 3 samples.

For added complexity, you may throw in random alternatives (PBR, Keystone, etc) to see if they really know their BMC's, but that doesn't sound like what your stated objective is.

For statistical significance, this is where you need a larger sample size. Maybe have 10 people take the same test, and calculate an average "correct" response rate for the 10 people, and the standard deviation of those "correct" response rates. Then you can see if any one person's "correct" response rate is more than 2 standard deviations above the mean. You could do it separately for each beer and in aggregate.
E.g. Bud has the highest "correct" rate, and Jim was significantly better at guessing Bud, though Jack was significantly better at guess all beers. Miller had a significant number of false negatives.

Intersting. There is a bit of a variance in the office over what people think they can do. Some say they can always pick out Miller out of the line-up etc. I'd like to see their ability to cluster even if the cluster is consistently wrong. I would almost concede the point if they were able group the beers accurately even if labeling them incorrectly.

Using your method and getting down to one variable essentially, what is a correct % that would be considered statistically significant? one standard deviation perhaps?

As an aside, they did basically this same type of test in the movie Beer Wars, and basically no one could correctly identify which beer they were drinking with any level of consistency.

I saw that and it seemed a little informal, but definitely got me thinking! I'll definitely need more than a single round of tasting and some actual statistical rigor for this test, we are very serious about challenges here.

Well I dunno about the "light" versions per se, but I have always found Coors pretty easy to distinguish because of the distinctive "corny" flavor.

Even to a novice? These guys have no idea that corn is used in the production of that beer. If you ask a person off the street to describe the flavor they get from coors, would they say corn?

 

[wildwest450]

I did a blind taste test between Bud Light, Coors Light, Miller Light, and Busch Light with a friend of mine and his girlfriend. We all agreed Coors Light was by far the worst and easiest to pick out.

If you thought that the Silver Bullet was worse than Busch light, I weep for you.

 

[nickharbour]

Since you can't have more people for bigger sample size, I'd just do it for fun. I agree that you'd want to ask some questions first. Which is your favorite beer between Bud, Millerlite, and Coors Light. Also ask if they think they can tell which is which out of all three. Pour four numbered cups and have one of them be doubled. That would test if they can even identify which two are the exact same beer. Probably not.

I'll have about 15 people, I'm currently thinking about doing 5-10 flights each and perhaps enforcing a mandatory spit policy so they can't claim the alcohol affected their judgment.

I like the double cup idea, that's pretty elegant

 

[Fritobandito]

If you thought that the Silver Bullet was worse than Busch light, I weep for you.

Busch Light ended up being second worst. I'm not sure about the Coor's, but it was really sour. Like grossly sour.

 

[weirdboy]

Even to a novice? These guys have no idea that corn is used in the production of that beer. If you ask a person off the street to describe the flavor they get from coors, would they say corn?

Well maybe not be able to identify it as corn, but it definitely has a distinctive flavor that stands out quite a bit from the other members of the BMC pack. I mean I can remember being able to pick out a Coors even when I was in high school because of that flavor.

 

[nickharbour]

Well maybe not be able to identify it as corn, but it definitely has a distinctive flavor that stands out quite a bit from the other members of the BMC pack. I mean I can remember being able to pick out a Coors even when I was in high school because of that flavor.

Have you done a blind taste test? its harder than you'd imagine. I had to do them in culinary school and I pretty much have lost faith in humanities ability to discern flavor without intense study

 

[mattd2]

I'm a bit confused with where this thread headed. The original challange seamed to be "who is the best at telling BMC apart", and not what this has turned into with talk about sample size, etc. "can the average person tell BMC apart".
I would just do the 4 (small or normal) cups as mentioned to each person 3 times, whoever gets the most right wins. Don't worry about making them spit it out because if they claim alcohol affected their judgement just tell them they can hold their liqour and are therefore disqualified for whining

 

[android]

i've done this with my family and nobody got any of them right (which says a lot, because they all tend to prefer one BMC product over another) and the only two pieces of advice i can give are, 1) don't worry about computing statistics for this and 2) use cups that are opaque, some of those BMC products have different colors than others and that can make them easy to tell apart.

 

[ReverseApacheMaster]

I'm positive I could distinguish Bud or Bud Lite from the other two. I could probably figure out which is Coors and which is Miller. Miller tastes more like corn than Coors (to me) and Coors tastes the most like nothing. However if you added any other light lagers in the mix I probably couldn't distinguish any/many with more than a guess.

 

[weirdboy]

Have you done a blind taste test? its harder than you'd imagine. I had to do them in culinary school and I pretty much have lost faith in humanities ability to discern flavor without intense study

Well I have had someone hand me a glass of beer, and recognized it as Coors when I took a drink. I haven't done a formal blind taste test specifically designed to test that. Although I have done blind taste test recently where I had to identify the style and commercial example used for various BJCP category #1 and category #2 styles, along with some other unknown beers that were included to throw me off.

 

[Teromous]

Your efforts are in vain. Once somebody hooks up after drinking cup #3, everyone will know it's Bud Light.

 

[daksin]

A local alehouse does the Big Bland Blind Taste Test- they give you MGD, Coors light, Tecate, Bud Light, Miller High Life, PBR, and Keystone Light and if you can identify them you get a $100 gift card. It's so difficult they allow you to do it in groups and deliberate. So far nobody's done it as far as I know.

 

[nickharbour]

I'm a bit confused with where this thread headed. The original challange seamed to be "who is the best at telling BMC apart", and not what this has turned into with talk about sample size, etc. "can the average person tell BMC apart".
I would just do the 4 (small or normal) cups as mentioned to each person 3 times, whoever gets the most right wins. Don't worry about making them spit it out because if they claim alcohol affected their judgement just tell them they can hold their liqour and are therefore disqualified for whining

my original post is to design a test to test normal people. I would expect a portion (maybe large portion) of people posting on this site to have the expertise to pass the test.

 

[nickharbour]

Your efforts are in vain. Once somebody hooks up after drinking cup #3, everyone will know it's Bud Light.

my office is all male and straight, would have to be someone really strong beers for there to be a hookup after the third cup never say never though, i guess.

 

[Teromous]

my office is all male and straight, would have to be someone really strong beers for there to be a hookup after the third cup never say never though, i guess.

Understandable. One other vital piece of advice that I will offer you, is that Miller Lite should not be served first. Being triple-hopped, it will clearly overpower the palate and result in compromised data.

 

[wildwest450]

Your efforts are in vain. Once somebody hooks up after drinking cup #3, everyone will know that's the guy with no chance of getting laid.

.

 

[dallasdb]

Why not just test them over several days?

Give them 3 tastes (one of each) or I did like the idea of 4 tastes one of which is a double.

 

[nickharbour]

I've finished designing the test and we are going ahead with it tomorrow. A few days ago some of the guys decided to "practice" and bought a bunch of each and were taste testing. It was enlightening because we all noticed right away that the color of each was identifyable far more than the flavor. Miller Lite was noticably darker and coors and bud were farily close but with coors having a slightly different hue. I've accounted for this a few places in the test so that they are only evaluating based on flavor, which what the initial point of the test. If you don't want to read it all basically I made a double blind version based of Bobby_M's idea of doubling up. Three flights of four, each beer is doubled once in the test.

I was working out the probabilities but due to all the rules I have the combinatorics became a little heavy so I just whipped up a quick program to simulate 300 Million random test takers and I've included their results at the bottom for an idea of how people do against random guessing.

Rules:

- Overall Test Methodology:
* Bud Light, Milller Lite and Coors Lite beers will be used from both can and bottle sources
* The test will consist of three flights of four samples
* Each flight will contain at least one of each brand and one will be doubled
* Each brand will be doubled in only one flight in the test
* Sample Selection Regulations:
* Each flight will contain two beers from Can sources and two beers from Bottle sources
* The brand in a flight which is represented by two samples must contain one sample from a bottle and one from a can
* For each brand, the can sources must be purchased from a different location than the bottle source
* The samples must be served near their recommended serving temperature
* The order of samples must be random in each flight

- Regulations for Identifying Samples
* In each flight the test subject must identify at least one beer from each brand
* The test subject may only identify a single beer brand a maximum of five times during the entire test
* An opaque cup must be used for the samples
* A cup of water and a spit cup will be provided to the test subject for the duration of their test
* The test subject cannot begin evaluating a new flight immediately after they finish a flight. they must leave the judging table and wait at least two minutes.
* No beer from external sources to the test samples may be consumed in between flights by the test subjects
* No beer from external sources to the test samples may be consumed for 15 minutes before the test by the test subjects
* No beer from external sources to the test samples may be consumed during the evaluation process itself by the test subjects
* The test subject has access to sample all the beers in the flight during the evaluation process.
* Once final identifications are made by the test subject and logged by the proctor the subject may not change their result
* Samples must be evaluated based on aroma, flavor and mouthfeel only. A test subject attempting to evaluate a sample’s appearance or color will be blind-folded and the flight will be restarted.
* The test subject has a maximum of five minutes to evaluate each flight

- Proctoring Regulations
* The test must be double-blind. The test proctor who is giving the subject their samples, logging their results and ensuring the subject stays within test regulations must not know which brand is in which cup.
* The test proctor who is responsible for pouring the beverages must do so out of sight of the other proctors and test subjects
* The test proctor who is responsible for pouring the beverages must not look at the test subjects while they are evaluating

- Test Environment
* The test must be conducted in a reasonably dark environment to prevent visual evaluation of the sample’s color

Scoring and Right to Claims:
* The test scoring consists of the following weighted categories:
* Overall Number Correct (Weight x2)
* Bud Light Identification (Weight x1)
* Bud Light Clustering (Weight x1)
* Miller Lite Identification (Weight x1)
* Miller Lite Clustering (Weight x1)
* Coors Light Identification (Weight x1)
* Coors Light Clustering (Weight x1)

* For each category a score of A is 4 points, B is 3 points, C is 2 points, D is 1 point and F is 0 points.
* A test subject who gets 10 or more correct choices will be awarded an A for the overall category and can make the claim that they can differentiate all the brands like an expert
* A test subject who gets 9 correct choices will be awarded a B for the overall category and can make the claim that they can differentiate all the brands better than the average person
* A test subject who gets 8 correct choices will be awarded a C for the overall category and should make no claim as to their skill in differentiating all the brands
* A test subject who gets 7 correct choices will be awarded a D for the overall category and should make no claim as to their skill in differentiating all the brands
* A test subject who gets 6 or less choices correct will be awarded an F for the overall category and must claim that their statistically insignificant skill is no better than random chance at differentiating all the brands.
* A test subject who accurately identifies a single beer brand 4 times gets an A in the identification category for that brand and can make the claim that they can identify that brand
* A test subject who accurately identifies a single beer brand 3 times gets a D in the identification category for that brand and can make no claim as to their ability to identify that brand
* A test subject who does not accurately identify a single beer brand at least 3 times gets an F in the identification category for that brand and should claim that they have no ability to identify that brand
* A test subject who accurately clusters a single beer brand 4 times, meaning they labeled that beer consistently even if it was consistently wrong 4 times, gets an A in the clustering category for that beer
* A test subject who accurately clusters a single beer brand 3 times, meaning they labeled that beer consistently even if it was consistently wrong 3 times, gets an D in the clustering category for that beer
* A test subject who does not accurately cluster a beer brand at least 3 times gets an F in the clustering category for that beer



Random Guess Probabilities (out of 300 Million people randomly guessing):
at least 1 correct: 296686675 people (%98.895558)
at least 2 correct: 279325498 people (%93.108499)
at least 3 correct: 237977363 people (%79.325788)
at least 4 correct: 176676709 people (%58.892236)
at least 5 correct: 111524319 people (%37.174773)
at least 6 correct: 59099357 people (%19.699786)
at least 7 correct: 26072105 people (%8.690702)
at least 8 correct: 9231269 people (%3.077090)
at least 9 correct: 2705025 people (%0.901675)
at least 10 correct: 549309 people (%0.183103)
at least 11 correct: 94051 people (%0.031350)
at least 12 correct: 7223 people (%0.002408)
clustered Bud at best 3 times: 75776362 people (%25.258787)
clustered Bud at best 4 times: 4692424 people (%1.564141)
clustered Miller at best 3 times: 75770863 people (%25.256954)
clustered Miller at best 4 times: 4688770 people (%1.562923)
clustered Coors at best 3 times: 75795610 people (%25.265203)
clustered Coors at best 4 times: 4690082 people (%1.563361)