So you're saying beer judging is subjective

[jd1984]

Decided to enter my first two competitions with a recipe I made. Both were mailed on the same day and the competitions were a week apart. Both required 3 bottles and I took 6 consecutive bottles from the middle of bottling the same batch.

Competition 1 - 2 weeks after bottling. Average score 42.2 and made it to the second round.

Competition 2 - 3 weeks after bottling. Average score 28.

Interested to get the score sheets to see the judges comments. My guess was that it would come in low to mid 30s, so averaging the two outs it about where I thought it should be. I plan on brewing another batch with some small changes and sending that in to another couple comps. It's just funny to see such drastically different scores in something so similar.

 

[TheMadKing]

Yep absolutely agree, I've had similar scores on the same beer. That's why it's important to brew beer YOU like first ave foremost

 

[LBussy]

Edit: Never mind.

 

[schematix]

There are a number of external factors to consider:
1. Judging quality (not always experienced)
2. Judging preferences (some may prefer various ends of the style spectrum more than others)
3. Competitor quality (how many times have you seen a newb get a medal - it happens and it's not because they area great brewer)

Bottom line, competitions are a waste of beer and money. Make the beer you like to drink.

 

[Minky]

Ideally, once the judges determine that the beer falls within the parameters of the category entered, they should judge the beer on its own merits. This isn't always the case, as you have found out.

I once brewed a hoppy pale ale that was on the borderline between an American Pale Ale and an American IPA, so I entered it in both categories. On one of the APA score sheets, the judge stated that the beer was not hoppy enough and "should have been dry-hopped". On one of the IPA score sheets, the judge stated that the beer was "a good representation of the style". Huh?

As long as humans are judging, personal preferences will occasionally creep in to skew the results.

 

[Qhrumphf]

Consistent performance is the key. A single comp with or without a medal doesn't mean much. Had beers go through 8 competitions, medal in 7, and the 8th give it a 23 the same day another comp gives it a 42 and a Gold.

Judging experience is a big factor. Judge bias is closely tied to it.

How the comp stores your beers is also a factor. I have NEVER skunked anything that I have brewed. I had one comp return a few "skunked" comments with enough pattern to say the organizers were damaging beers during storage.

How things are served is another. Bad temps, stinky cups, poor judging locales (imagine judging in a room stinking of a bad sour mash or next to a barn) or even the beer being poured for you (more applicable to commercial comps where labeled bottles are submitted so judges cannot see the bottle). These can be out of the judges to control to large extents, but can severely impact judging.

I don't agree that comps are a waste of time. But they need to be viewed as sole data points and not extrapolated from until there's multiple comps to view from.

 

[C-Rider]

Consistent performance is the key. A single comp with or without a medal doesn't mean much. Had beers go through 8 competitions, medal in 7, and the 8th give it a 23 the same day another comp gives it a 42 and a Gold.

Judging experience is a big factor. Judge bias is closely tied to it.

How the comp stores your beers is also a factor. I have NEVER skunked anything that I have brewed. I had one comp return a few "skunked" comments with enough pattern to say the organizers were damaging beers during storage.

How things are served is another. Bad temps, stinky cups, poor judging locales (imagine judging in a room stinking of a bad sour mash or next to a barn) or even the beer being poured for you (more applicable to commercial comps where labeled bottles are submitted so judges cannot see the bottle). These can be out of the judges to control to large extents, but can severely impact judging.

I don't agree that comps are a waste of time. But they need to be viewed as sole data points and not extrapolated from until there's multiple comps to view from.

I've had the comment "gusher" a number of times, but funny at home there were NO gushers.

 

[gbx]

Decided to enter my first two competitions with a recipe I made. Both were mailed on the same day and the competitions were a week apart. Both required 3 bottles and I took 6 consecutive bottles from the middle of bottling the same batch.

Competition 1 - 2 weeks after bottling. Average score 42.2 and made it to the second round.

Competition 2 - 3 weeks after bottling. Average score 28.

Interested to get the score sheets to see the judges comments. My guess was that it would come in low to mid 30s, so averaging the two outs it about where I thought it should be. I plan on brewing another batch with some small changes and sending that in to another couple comps. It's just funny to see such drastically different scores in something so similar.

Here are a few other scenarios that haven't been mentioned:


You could have had one bad bottle and that was unfortunately the one the second comp scored.

or

You had one really good bottle that got scored by the first comp and the second bottle was mediocre which is why it didn't do well in BOS judging.

or

The handling was bad and the stewards gave the judges a yeasty pour that sat around getting warm and degassing before the judges got the sample.

 

[jd1984]

Got score cards back from the lower score competition and I am very pleased with them actually. Both judges seemed to like the beer but I was knocked down as they felt it was not completely in style. That was something I was slightly worried about and will probably move to 10A instead of 21A in a future competition. Still good to see a judge repeatedly say it is really good even though they scored it down.

 

[pdxal]

I've had the comment "gusher" a number of times, but funny at home there were NO gushers.

I've had that too. Anything is possible- 1,2, or 3 bottles out of entire batch only ones infected, etc., but how they store and steward your beer could certainly cause a gusher. Warm, jostled/shaken could be at work.

 

[Minky]

Still good to see a judge repeatedly say it is really good even though they scored it down.

That is why I still enter competitions. It always helps to get an opinion of your beer from someone who doesn't know you.

 

[gbx]

Got score cards back from the lower score competition and I am very pleased with them actually. Both judges seemed to like the beer but I was knocked down as they felt it was not completely in style. That was something I was slightly worried about and will probably move to 10A instead of 21A in a future competition. Still good to see a judge repeatedly say it is really good even though they scored it down.

Wow, they knocked you down from a 40 to 28 for the style differences between US Pale and US IPA? Thats brutal.

 

[Qhrumphf]

Wow, they knocked you down from a 40 to 28 for the style differences between US Pale and US IPA? Thats brutal.

Sounds to me like the OP understands he got an inflated score on the 40 and for a "good but balance is a little off" about halfway in between is probably about right as he said originally.

 

[gbx]

Sounds to me like the OP understands he got an inflated score on the 40 and for a "good but balance is a little off" about halfway in between is probably about right as he said originally.

Thats probably a fair interpretation. IPA is a horrible category to judge as there are so many bad beers and its a complete palatte wrecker. OP probably would do better entering a US pale if its in between styles. If entered in IPA, it would be the least intense beer in the flight. It might score well but will never medal in side by side tastings in the medal round.

 

[jd1984]

Thats probably a fair interpretation. IPA is a horrible category to judge as there are so many bad beers and its a complete palatte wrecker. OP probably would do better entering a US pale if its in between styles. If entered in IPA, it would be the least intense beer in the flight. It might score well but will never medal in side by side tastings in the medal round.

Sounds like yours and mine are probably similar. By the numbers, my beer is too high of an OG to be a true pale but barely squeaks in to 21a at ~45 IBUs.

Glad I entered as I am not connected to a lot of other brewers and wanted feedback from relatively knowledgeable people.

 

[Turkeyshot]

Funny how scores can change so quick. I just received my score sheet back from a competition a couple of weeks ago. First round my RIS scored a 41, second round a 28. Both rounds were in the same day, but by different judges.

 

[mongoose33]

I've entered a few local comps, and I'm amazed at the variance in the beers, AND which are considered good and which not.

What's unfortunate (IMO) in all this is that beers are being judged not on how good they taste, but how they conform to some particular style standard. I get that there needs to be something reasonably objective about this (and style standards would presume to be), but within the styles there's so much variance.

Early on, I felt my ability as a brewer would be determined by how well I could hit a style. Well. The focus became hitting a style, NOT on brewing beer I wanted to drink. I moved away from that quickly to brewing beers I wanted to drink, not that nailed a style right on the head.

Had a throwdown at the LHBC last night. My rye beer made the final three, but it doesn't really fit any style guidelines. To hit the style guidelines for Rye beer the color is supposed to be "Usually pale yellow to gold." Mine's not, as you can see below. I add 6 oz of chocolate wheat to add that color plus help a bit with head retention.

In the end, I "lost" to a brown ale that did hit the style guidelines fairly well; nobody really knew what to do with mine, I suspect. Further, I saw how much they poured from the bottle; not enough for even a good mouthful. That's how mine tastes best to my taste, a good full mouthful. Not a sip.

I had one guy who was quite complimentary of the beers I included (a pils and a cal common too).

***********

Rant over. My biggest concern about style guidelines is that they are quite broad within them; could have hoppy presence but ok if not; could have some of this flavor, but it's ok if it doesn't. The flavors could be this or this.... sure seems to me like there's still a lot of room for subjectivity within the style guidelines.

 

[steveoatley]

There are a number of external factors to consider:

Bottom line, competitions are a waste of beer and money. Make the beer you like to drink.

Humans are the Judge
Humans are flawed
No 2 Humans are the same

The main reason why I do NOT enter competitions.

just my 2 cents
S

 

[Morrey]

Separate but related subject regarding judge's subjectivity:

I am the previous Executive Director of the SC BBQ ASSN in which our Assn is contracted to supply trained/certified judges to score competitive BBQ. In some cases the prize money is upward of $20K so this is serious business.

Too many times as I trained and monitored aspiring judges I heard "I scored it low since I personally didn't like that style of BBQ". I immediately noted this comment and followed that judge during their certification. I'd rather hear "I didn't like this style personally but scored it very high since it was done to standard so carefully". Unfortunately, as humans scoring anything, we have our preferences and biases that inevitably surface.

Mongoose33 makes a good point as he was brewing to satisfy a "style" but not necessarily a beer he enjoyed drinking.
"Early on, I felt my ability as a brewer would be determined by how well I could hit a style. Well. The focus became hitting a style, NOT on brewing beer I wanted to drink. I moved away from that quickly to brewing beers I wanted to drink, not that nailed a style right on the head".


Due to disparity in scoring and subjectivity being discussed, I have little interest in becoming a beer judge nor having my beer judged. But I completely respect those who make the choice to become a beer judge or have their beers judged. I have specific ideas of what I like in my own beer, and honestly, I make beer for me and what I like.

 

[tmendick]

Funny how scores can change so quick. I just received my score sheet back from a competition a couple of weeks ago. First round my RIS scored a 41, second round a 28. Both rounds were in the same day, but by different judges.

I've given the same entry a 40 for the first round and would have given it a 30-32 on the second round. The difference being storage, we judged it as one of the first ones and it was a crisp, clean, with very low DMS Czech Pilsner. Once it warmed and was ready to go onto the second round, it had been sitting out for a few hours and the DMS was over the top.

Inconsistent: Bottle, Judge, Storage each can make a huge difference between two scores of the same beer

 

[gbx]

Due to disparity in scoring and subjectivity being discussed, I have little interest in becoming a beer judge nor having my beer judged. But I completely respect those who make the choice to become a beer judge or have their beers judged. I have specific ideas of what I like in my own beer, and honestly, I make beer for me and what I like.

I was(still am) a bjcp judge but I feel the same way, hence not judging or entering any comps in the last 2 years. Judging endless flights of IPA stopped being fun and I seemed to have some different ideas than most judges: In a local comp with a ton of less than stellar entries, the style guide should be secondary to an assessment of the quality of brewing (I think a 40 pt german pilsner entered as an American IPA should score at least 30 pts).

I also saw a lot of lazy/sloppy judging practices from all levels of judges - smoking and eating spicy foods between flights, judging long past palate fatigue has set in to level up their rank, ignoring style guidelines during mini-BOS judging (this is the place the guidelines should be rigorously applied) and ranking the beer by intensity/gravity (this might be better with the new guidelines), not even referring to guidelines during judging when they clearly don't have it memorized, judging based off a single style example (if someone had entered Ayinger Celebrator, they would have filled the scoresheet with an extensive list of its differences from Paulaner Salvator and given it 30pts), etc, etc.

 

[Morrey]

I was(still am) a bjcp judge but I feel the same way, hence not judging or entering any comps in the last 2 years. Judging endless flights of IPA stopped being fun and I seemed to have some different ideas than most judges: In a local comp with a ton of less than stellar entries, the style guide should be secondary to an assessment of the quality of brewing (I think a 40 pt german pilsner entered as an American IPA should score at least 30 pts).

I also saw a lot of lazy/sloppy judging practices from all levels of judges - smoking and eating spicy foods between flights, judging long past palate fatigue has set in to level up their rank, ignoring style guidelines during mini-BOS judging (this is the place the guidelines should be rigorously applied) and ranking the beer by intensity/gravity (this might be better with the new guidelines), not even referring to guidelines during judging when they clearly don't have it memorized, judging based off a single style example (if someone had entered Ayinger Celebrator, they would have filled the scoresheet with an extensive list of its differences from Paulaner Salvator and given it 30pts), etc, etc.

I totally see (and know) where you are coming from, and motivation to judge beer and/or BBQ may be more connected than may appear.

I saw in my role of BBQ administrator many potential judges looking for a way to scoff down what would potentially be better BBQ than they can get most places. It became a "hoggish" situation instead of a way to actually become a professional judge. A Gourmand of sorts, plus there is the cool factor to brag about.

So, with this said, are there the similarities between the two that I am assuming? I am sure most aspiring Cicerones do it for the love of beer and for the knowledge that comes with it. But as can be gleaned...is this potentially a way to try beers and don't really care about the end result with sloppy judging?

 

[Hoppy2bmerry]

Well, the judging is based on the category it was entered. So possibly the best option if one is not sure what style it is, enter as experimental and note in the comments what the judges should consider.

 

[Hoppy2bmerry]

I totally see (and know) where you are coming from, and motivation to judge beer and/or BBQ may be more connected than may appear.

I saw in my role of BBQ administrator many potential judges looking for a way to scoff down what would potentially be better BBQ than they can get most places. It became a "hoggish" situation instead of a way to actually become a professional judge. A Gourmand of sorts, plus there is the cool factor to brag about.

So, with this said, are there the similarities between the two that I am assuming? I am sure most aspiring Cicerones do it for the love of beer and for the knowledge that comes with it. But as can be gleaned...is this potentially a way to try beers and don't really care about the end result with sloppy judging?

No. Imagine if you can spending 10 minutes sipping and sniffing a poorly made beer so that you can judge it and write notes about it. I am not a judge, nor do I think I have tha palet for it, but I have been a steward and closely observed certified judges judging beer. When the beer is not that good they still sniff, taste and evaluate. That is what an entrant deserves and the judges I saw did just that. It is still subjective but the guidelines for the style are what they have to adhere to, and that is the part that is objective.
.

 

[greywolf]

BBQ contests are the same. I've tried some of higher placing entries and found them to be basically inedible. I no longer do comps, just q and brew what I like.

 

[gbx]

No. Imagine if you can spending 10 minutes sipping and sniffing a poorly made beer so that you can judge it and write notes about it. I am not a judge, nor do I think I have tha palet for it, but I have been a steward and closely observed certified judges judging beer. When the beer is not that good they still sniff, taste and evaluate. That is what an entrant deserves and the judges I saw did just that. It is still subjective but the guidelines for the style are what they have to adhere to, and that is the part that is objective.
.

I've seen good judging, but I also saw enough bad judging to turn me off it. The other thing that kinda bummed me out is the number of hardcore competition brewers who submit 15+ entries. Thats fine if they want to do that they don't need the full written scoresheet and someone else can judge it. In some comps, a large percentage of the entries are from spammers who have submitted the same beer to multiple comps and the checkbox style scoresheet would save a lot of time and effort. The time isn't being spent helping brewers and the spammers generally don't have brewing issues and are really just looking for medals.

 

[Qhrumphf]

I've seen good judging, but I also saw enough bad judging to turn me off it. The other thing that kinda bummed me out is the number of hardcore competition brewers who submit 15+ entries. Thats fine if they want to do that they don't need the full written scoresheet and someone else can judge it. In some comps, a large percentage of the entries are from spammers who have submitted the same beer to multiple comps and the checkbox style scoresheet would save a lot of time and effort. The time isn't being spent helping brewers and the spammers generally don't have brewing issues and are really just looking for medals.

Eh. Before I stopped entering homebrew comps I was one of those glory hounds.

I didn't care about the feedback. Hell half the time I didn't care about the prizes. I just wanted glory. Then again I'd submit 4-6 beers at a time (or the max if lower than that), not 15+.

I think in certain circumstances (NHC, commercial comps) where it's 100% glory, I'm all for checkbox sheets. However you can't differentiate in smaller local comps while maintaining blind judging and as an entrant the feedback from a checkbox scoresheet is mostly useless (and harder to immediately tell when a judge is full of sh!t).

I think the biggest problem with bad judging besides pure laziness is judge arrogance. Lower ranked judges think they're suddenly master fault finders and look for problems that aren't there and create them in their heads. Higher ranked judges tend to be more accurate but then you run into the style bias a lot (see it with a shameful number of National judges who don't read the guidelines while judging, particularly ones who've been at it a long time).

And then of course there's differences in sensitivity. One ultra-sensitive diacetyl taster can then bias other judges who may not have picked it up, and tank a score as a result.

 

[Oginme]

Yes, beer judging is subjective. Let's start with the fact that not everyone has the same skills at sensory perception. People have different sensitivities to flavors and aromas. That is why guidelines are put in place.

Next, though I have not been through judges training, it seems like a majority of the focus is on (1) knowing the guidelines and (2) evaluation of faults. Number 2 is important, because as a judge if you are being trained to look for faults, you will find them -- regardless of their presence or not.

I come from another world of rigorous training to be licensed to judge: Dairy goats. Here too, there is a focus on the conformation scorecard and... faults. Needless to say, inexperienced judges have a hard time expressing only positive feedback. Luckily, the judges are also taught how to give reasons (if you think writing down reasons for scoring are difficult, try doing it on a microphone in front of a crowd of people) in a positive manner.

Now (getting back to beer judging, though not really) through in style biases, individual ego, and the judges actual ability to discern differences in taste, aroma, color, etc., and palette fatigue and you can understand why there can be a vast difference in scores between judging competitions.

When I enter a competition, it is because I DO want the feedback. This is also why I enter the same beers in several competitions. Each time I get a set of scores back, I sit down, pop a bottle of that brew, pout it into a tasting glass and review the sheets. Sometimes I learn something that I missed and sometimes I can write off that judges opinions and move on.

 

[Morrey]

Yes, beer judging is subjective. Let's start with the fact that not everyone has the same skills at sensory perception. People have different sensitivities to flavors and aromas. That is why guidelines are put in place.

Next, though I have not been through judges training, it seems like a majority of the focus is on (1) knowing the guidelines and (2) evaluation of faults. Number 2 is important, because as a judge if you are being trained to look for faults, you will find them -- regardless of their presence or not.

I come from another world of rigorous training to be licensed to judge: Dairy goats. Here too, there is a focus on the conformation scorecard and... faults. Needless to say, inexperienced judges have a hard time expressing only positive feedback. Luckily, the judges are also taught how to give reasons (if you think writing down reasons for scoring are difficult, try doing it on a microphone in front of a crowd of people) in a positive manner.

Now (getting back to beer judging, though not really) through in style biases, individual ego, and the judges actual ability to discern differences in taste, aroma, color, etc., and palette fatigue and you can understand why there can be a vast difference in scores between judging competitions.

When I enter a competition, it is because I DO want the feedback. This is also why I enter the same beers in several competitions. Each time I get a set of scores back, I sit down, pop a bottle of that brew, pout it into a tasting glass and review the sheets. Sometimes I learn something that I missed and sometimes I can write off that judges opinions and move on.

Oginme, this ^^^^ was a darn good post. I will concur that no two people perceive a taste the same way. I used this example in my BBQ Judge training classes:

I made a New Orleans styled gumbo. I purposely made it mild knowing the potential reaction I would be getting. My wife sampled hers to say it was a little on the spicy side but just right. Her mother starting fanning her tongue saying it was the hottest thing she ever ate. I was shaking Texas Pete on mine to add heat. All three of us perceived the heat level at different thresholds. Same with salt (and virtually all tastes)....everyone perceives saltiness differently.

This perception is bound to play into beer judging. I may say boy this beer is hoppy through the roof...while another may say did you forget to add the hops? Perception. Palate Sensitivity.

Perhaps we take exception to a judge and fault their rulings/scores as subjective. Maybe..maybe not. Maybe they simply perceive various aspects of a beer differently than you or I. Agreed, getting feedback (like the feedback or not) can aide us in improving our beers. That's the positive side and I totally agree.

 

[Elaelap]

Beer comps are a crap shoot. It doesn't surprise me reading the comments about different scores, gushing bottles, etc. I've done plenty of competitions when I first started but was quickly turned off by the subjective nature of the judging. I realize that everyone's pallet is different, but mainly I was looking for constructive feedback. Their are a lot of great BJCP judges out there, and when you get one that actually takes the time to write, and educate you feel a sense of accomplishment because it was constructive. Most score sheets are 1 to 2 words, and little thought put into it.

Not to say I do this all the time, but it's quite fun brewing and entering a total ****ing disaster of a beer, gusher, sour, infected and enter it into the comps, just to see what they say. To my surprise, one of the worst beers I made actually came back with a decent score, which made me laugh. My inspiration came from some dude that actually entered a chili beer, complete with fermented hot dogs and all.

Brew what you like and have fun, but take comps for what they are, subjective. Once in awhile you get a great BJCP that will actually teach you something.

Cheers!