I think I am done with contests

[BoomerHarley]

I just got the results back from my latest contest. I didn't expect to win. But I did expect some sort of useful comments. Something I could use to improve my beer. Instead, what i got was one judge telling my the beer was too hoppy while the next says not enough hops. One saying nice and crisp, the other saying not crisp enough. Too bitter, not bitter enough. It's like they were tasting different beers (they weren't).

I am losing faith in the BJCP certification system. I literally have no idea whether my pilsner was overhopped or underhopped. Was my helles too sweet or not sweet enough? Too bitter? Not bitter enough?

It's ridiculous. Hell, I might as well just throw a contest and invite random street people to judge it.

 

[Tiber_Brew]

I had a poor experience with my IPA submission last year. I got comments about a high amount of diacetyl. I know what diacetyl feels like in low quantities, and tastes like in higher quantities. My IPA had none. Either they mixed up my submission with someone else's, or they mistook some other characteristic for diacetyl (can't imagine what). I also brought a bottle to Yooper from this forum (she is a BJCP judge whom I trust) and she agreed with me.

Looking for useful comments and leaving disappointed with incorrect and/or contradictory comments is a bummer.

 

[unionrdr]

After reading so many of these sort of posts, I'm thinking it's coming down to memory retention of the individual judges. Experience is the best teacher, but memory cannot be discounted. It must become second nature in order to give the best review of the beer in question. Maybe that aspect of the program needs to be addressed?

 

[SudsyPaul]

I'm still waiting for my results from my most current competition, but I've had some where they just say "hops" and "ok" in the descriptions for aroma and taste... then they score me 35/50.

Well, I need a better reason for knocking off 15pts. 35pts sounds about right, but perhaps outline to me where I went awry so that I can improve!

 

[Minky]

There are some judges who are definitely judging beers on their own personal preferences rather than judging the beer on its own merits. I often enter the American Pale Ale category because I prefer to hop my pale ales in less than mega-doses. I'm talking around 35-40 IBU (Tinseth), which really isn't all that wimpy. In the comments section under "overall impression", one of the judges apologized for lowering my score, saying that I was a victim of "style creep". He said that the judges now expect hop levels in the APA category which used to be considered in the IPA range.

HUH??!! I was certainly a victim of some kind of "creep"!

 

[podz]

Population wise, most people exercise subjective judgement. The percentage of people who exercise objective judgement is something small.

I am objective to a fault when it comes to most things. But I simply don't like beer made with sugar and if I were a judge then you'd lose by putting sugar beer in front of me. In other words, my beer judgement is highly subjective. And I believe that most BJCP judges are the same.

 

[ram5ey]

Yeah it just a really subjective thing to judge. I recently had a scoresheet that was totally off the map. Dinged a pale ale for some bogus flaws that were not there and gave it a 30 somehow moved to a minibos where it beat out another beer of mine in a different subcategory that scored a 42(different set of judges)...as someone training to take the tasting exam, I know that beer wasn't a 30 (it's won a couple times) but I think the judge was just worn out and searching for things...I think the important thing is to get lots of feedback from different judges and then you can see a pattern for a beer. Now of course sometimes I just enter something to try and win a medal but most my entries are purely to see what judges think. I also trust my pallette so I'm fine calling BS on a bogus score sheet...

 

[Minky]

I'm still waiting for my results from my most current competition, but I've had some where they just say "hops" and "ok" in the descriptions for aroma and taste... then they score me 35/50.

Well, I need a better reason for knocking off 15pts. 35pts sounds about right, but perhaps outline to me where I went awry so that I can improve!

I entered a Kolsch once which won 2nd in Light Hybrid, so I shouldn't really complain, but I will. Under "Aroma" the judge said "Classic Kolsch aroma" and then gave me a 8/12 in that category! Whoa pal, I want to know why I get dinged 4 points if it's "classic"

 

[podz]

Why do people who've never tasted a 100 point wine give the best wine they've ever tasted 89 points?

 

[TheZymurgist]

Population wise, most people exercise subjective judgement. The percentage of people who exercise objective judgement is something small.

I am objective to a fault when it comes to most things. But I simply don't like beer made with sugar and if I were a judge then you'd lose by putting sugar beer in front of me. In other words, my beer judgement is highly subjective. And I believe that most BJCP judges are the same.

Yeah it just a really subjective thing to judge. I recently had a scoresheet that was totally off the map. Dinged a pale ale for some bogus flaws that were not there and gave it a 30 somehow moved to a minibos where it beat out another beer of mine in a different subcategory that scored a 42(different set of judges)...as someone training to take the tasting exam, I know that beer wasn't a 30 (it's won a couple times) but I think the judge was just worn out and searching for things...I think the important thing is to get lots of feedback from different judges and then you can see a pattern for a beer. Now of course sometimes I just enter something to try and win a medal but most my entries are purely to see what judges think. I also trust my pallette so I'm fine calling BS on a bogus score sheet...

The entire purpose of any judging standards, and the BJCP, is to limit subjectivity as much as possible! I realize it's impossible to eliminate subjectivity altogether, but it is possible to mitigate it. I don't personally like overly bitter IPAs, but I know perfectly well that my IPA will get marked down for bitterness, and I know when I taste an IPA that's more true to style that will score higher than mine. As a general rule, I don't like Pilsners or other light lagers, but I can tell a good one from a bad one. Heck, I think sweet tea is nasty, but I know good sweet tea from bad sweet tea.

If judges are approaching the judging table without their personal biases in check, then there is no point to having any standards. Every competition becomes simply a best-in-show, and whichever beer gets the highest points from the most judges wins. They are doing a disservice both to the BJCP and to brewers.

 

[Erythro73]

Judging should the less subjective possible. When I grade my student's work, I try to be as objective as possible and leave some comments so they can understand what mistake they made. I make my correction sheet as detailed as possible to leave no room for interpretation afterwards. It wouldn't fly with my students if I went "Great solution, but -4 just because!". Or leave a comment on the exam copy going like "Forces, torques ---> 35/50". Fairness in grading is all about objectivity.

Now, I understand beer judging is bound to be less objective than physics, but if it's as bad as discussed in this thread, what's the point of the whole BJCP? So that a bunch of judge can get drunk on other people's brews? And what good information can you get out of it if it's so subjective? It seems to me like a waste of time.

 

[Minky]

If judges are approaching the judging table without their personal biases in check, then there is no point to having any standards. Every competition becomes simply a best-in-show, and whichever beer gets the highest points from the most judges wins. They are doing a disservice both to the BJCP and to brewers.

You make a very good point, especially in light of the fact that the judges can enter categories that they aren't judging. I hate to make accusations and I would hope it's not common, but I have wondered if there may be judges who purposely score lower to give themselves a better chance at "best-of-show".

 

[Darwin18]

There is a generally a better big difference in the scoresheet quality as you move up in the BJCP ranks. I would put much more weight in the higher ranked judge's scoresheet than the lower ranked judge's scoresheet. Sometimes it can be the opposite, but generally the non-BJCP, apprentice, or recognized ranks don't have the experience judging and evaluating entries to the BJCP standards. This is a generality of course.

You may want to try to target future competitions that advertise quality feedback and evaluation as their primary objective.

 

[myndflyte]

I don't know the hierarchy of BJCP but maybe they need to have someone above the judges. My example comes from submitting scientific papers to peer-reviewed journals. The paper gets red by a few or several reviewers and they are required to make comments which then get set to the editor who puts it all together and send it back to the writer. If they had something like that, the "editor" could at least say you need to be more descriptive as to why you deducted or something. But I have no idea if something like that is even feasible.

 

[TheZymurgist]

Judging should the less subjective possible. When I grade my student's work, I try to be as objective as possible and leave some comments so they can understand what mistake they made. I make my correction sheet as detailed as possible to leave no room for interpretation afterwards. It wouldn't fly with my students if I went "Great solution, but -4 just because!". Or leave a comment on the exam copy going like "Forces, torques ---> 35/50". Fairness in grading is all about objectivity.

Now, I understand beer judging is bound to be less objective than physics, but if it's as bad as discussed in this thread, what's the point of the whole BJCP? So that a bunch of judge can get drunk on other people's brews? And what good information can you get out of it if it's so subjective? It seems to me like a waste of time.

I really don't think this is an epidemic within the BJCP, but it definitely does happen. I've only had one entry in one competition that had widely different scores and reviews, and I think that's the experience of most. But to the few judges that can't get past their personal preferences, they definitely don't need to be judging.

You make a very good point, especially in light of the fact that the judges can enter categories that they aren't judging. I hate to make accusations and I would hope it's not common, but I have wondered if there may be judges who purposely score lower to give themselves a better chance at "best-of-show".

I'm not sure that's even an issue, although I haven't thought about it. Any good competition would disqualify judges from entering a category they're judging.

 

[Simps]

Wall of text, bear with me...

As a beer judge reading this I can say I have felt the same as you at times but have changed my opinion through experience. I originally thought the whole thing was a scam but now I realize there are some very important factors in the equation to consider before discrediting the whole system.

But to address some comments I have read before I explain why I think it is still a useful system:

-Judging normally isn't from memory. There are copies of the style guidelines (or at least there should be per competition rules) at every table. I have seen judges "wing it" from memory but every table I am at I pull out the guidelines and read back the description if they say something out of line.

-They can usually choose the category they judge, meaning they shouldn't be judging beer styles they hate. If they have to because of numbers (or lack of) then they judge the beer by definition not by preference. In a perfect world this happens, in the real world you will get people complaining they didn't want to drink *said random style* all day. Even though this happens, 99% of the time I have been on a style they are still impartial. The remaining 1% we get into an argument at some point, everyone steps back and re-calibrates and we get back to definition.

-You don't get drunk. I normally work one of the largest competitions in the US (Indiana State Fair) in addition to the small ones and have judged over 50 beers in a day, other judges have done more I'm sure. You have a water pitcher, neutral snacks like crackers and as many breaks as you want. It's hard to get drunk writing essays for 6 hours, try it some time! Plus many entries have problems, you would be a masichist if you wanted to get drunk like that.

-Scores don't start at 50 and get deductions. Look at the score sheet, most beers belong around 30(very good). Competition is steep. If you get a 35 you are above average. Generally anything above 45 is reserved for beers that are so rediculously good they break the mold. Beers that take what a style is and find a new way to exemplify it beyond what has been done before but make you think "This is what it should have been!".

-It matters what competition you enter. The smaller competitions will have fewer, less experienced judges who are overworked. The bigger ones will have Grand masters and experienced professionals divided between tables evenly to take advantage of their expertise. If you don't like your result I highly recommend you submit to a larger even out-of-state competition.

Finally I will say not giving feedback is a major issue in BJCP and people know it. Before every competition the organizers normally give a mini-speech about how you paid good money to get answers and if we want better beer we need to educate people. Despite this some people still ignore it. In my opinion they shouldn't be judges. This brings me to my next point...

-You will at some point have a bad experience in a competition because of an individual. Beers are judged by people, some of them are arrogant or lack enough knowledge to be there. The vast majority care about beer and the people that make it. I am sorry you had a bad experience (I have been there too), don't take it too seriously. Try a different competition and you will be surprised.

Now the reason why I still do them. The one truth about the system that makes it worth while is that simply the best beers really do win. Your score is just a number. It could be 26 in one place and 36 in another but at the end of the day the best beers always win. Don't focus on feedback like "hops" or "sweet", if you get a moderate score with little advice that really means you should look at your recipe. Get a odor-free plastic cup, a couple of friends and really tear it apart. Try to blend the flavors and tweak the components so they work together. Buy a commercial beer and taste it side-by-side. If this sounds like too much work then maybe competitions really aren't for you, but you don't have to stop making great beer at home for yourself.

If you did something bad you will get a very low score and they will definitely say something about it. If you did great ignore the praise and focus on the negatives to make it even better. Finally, even if your beer was great in the keg and great in the bottle you have in front of you the one they got might have gone bad. It happens, mixups are EXTREMELY rare. You can either enter it again or tweak your recipe, bottle it over a month in advance and taste it before you send it.

 

[Darwin18]

Well said - Simps.

 

[Calichusetts]

I see this with wine, beer and even everyday people who rate things. Often, good/better than average is a 30 out of 50, or 3 out of 5. As a teacher, I have to laugh at this, THATS A D-!!!!!!

What would you say or do if a teacher gave you a D- and told you your work was better than most?

Average is, in my world, a C. That would be a 37 in the beer world. In my opinion, you work from there.

 

[TheZymurgist]

Simps really hit the nail on the head.

 

[mdbrewer1]

Now the reason why I still do them. The one truth about the system that makes it worth while is that simply the best beers really do win. Your score is just a number. It could be 26 in one place and 36 in another but at the end of the day the best beers always win. Don't focus on feedback like "hops" or "sweet", if you get a moderate score with little advice that really means you should look at your recipe. Get a odor-free plastic cup, a couple of friends and really tear it apart. Try to blend the flavors and tweak the components so they work together. Buy a commercial beer and taste it side-by-side. If this sounds like too much work then maybe competitions really aren't for you, but you don't have to stop making great beer at home for yourself.

You make a lot of good points throughout, but the scientist in me struggles with the bolded. I get what you are saying about how the best beers usually win, but why even have a numbering scheme if it doesn't lead to consistency? It sounds like you could just distill it down to bad, good, better, best if the entire point is just to come up with a winner. If you send it to 3 competitions, I would imagine scores should similar within each competition and between the different competitions as well. If they're not, I don't see the value of having a 50 point scale at all.

 

[jbaysurfer]

I see this with wine, beer and even everyday people who rate things. Often, good/better than average is a 30 out of 50, or 3 out of 5. As a teacher, I have to laugh at this, THATS A D-!!!!!!

What would you say or do if a teacher gave you a D- and told you your work was better than most?

Average is, in my world, a C. That would be a 37 in the beer world. In my opinion, you work from there.

If Gordon Strong were to substitute teach one day should he hand out scores of 50 to students who's work deserves and A+? That's his world. Judges job is to judge based on the criteria established by the ruling entity (the BJCP in most beer contests), not to judge based on the A+ through F scale employed by most educational institutions.

 

[Darwin18]

You make a lot of good points throughout, but the scientist in me struggles with the bolded. I get what you are saying about how the best beers usually win, but why even have a numbering scheme if it doesn't lead to consistency? It sounds like you could just distill it down to bad, good, better, best if the entire point is just to come up with a winner. If you send it to 3 competitions, I would imagine scores should similar within each competition and between the different competitions as well. If they're not, I don't see the value of having a 50 point scale at all.

People are not machines that can be precisely calibrated. That's why sometimes a beer will score a 26 by a person in a competition and a 36 in another competition by a different person. A judges opinion on the beer in front of them is also influenced by the beers in the flight that preceded it.

Additionally, there are many different factors completely out of the entrants control such as how their beer was stored, temperature it was served to the judges, etc. that vary from competition to competition. Beer judging is subjective - if that bothers you don't enter competitions. Volunteer a competition sometime and you'll have a much better understanding of the judging process.

There are a lot of people who complain on the Internet about how unfair or flawed the judging process is but who don't get involved in the judging process or competition in anyway. Judging is not easy. Judges do not get paid. Every judge I've met or judged with has done their very best to determine the best beers in their flight. For better or worse this is the best process in place to judge homebrew - if there's a better one out there then enter their competitions.

 

[barnaclebob]

I see this with wine, beer and even everyday people who rate things. Often, good/better than average is a 30 out of 50, or 3 out of 5. As a teacher, I have to laugh at this, THATS A D-!!!!!!

What would you say or do if a teacher gave you a D- and told you your work was better than most?

Average is, in my world, a C. That would be a 37 in the beer world. In my opinion, you work from there.

You need to go back and look at the definitions for the scores. The scores don't have to conform to what number makes you feel good.

Outstanding (45 - 50): World-class example of style.
Excellent (38 - 44): Exemplifies style well, requires minor fine-tuning.
Very Good (30 - 37): Generally within style parameters, some minor flaws.
Good (21 - 29): Misses the mark on style and/or minor flaws.
Fair (14 - 20): Off flavors/aromas or major style deficiencies. Unpleasant.
Problematic (00 - 13): Major off flavors and aromas dominate. Hard to drink.

It doesn't sound like you grade on a curve...

 

[jbaysurfer]

You make a lot of good points throughout, but the scientist in me struggles with the bolded. I get what you are saying about how the best beers usually win, but why even have a numbering scheme if it doesn't lead to consistency? It sounds like you could just distill it down to bad, good, better, best if the entire point is just to come up with a winner. If you send it to 3 competitions, I would imagine scores should similar within each competition and between the different competitions as well. If they're not, I don't see the value of having a 50 point scale at all.

The point of a numbering scheme is not to promote consistency so much as it is to quantify the judges thoughts on a particular beer. A 26 and a 36 score gap would trouble me as well, but for different reasons. A 26 is defined as "Misses the mark on style and/or minor flaws." A 36 does not "miss the mark" and should generally conform to: "Generally within style parameters, some minor flaws."

Consistency is achieved by everyone agreeing to judge a beer by the same set of standards, not an implicit agreement that everyone will give beers the same numerical scores.

I think I get where you're coming from here, but it's a subtle distinction, and it explains why the same beer might get those different scores (on different tables/different competitions), but still have the same ordinal ranking within it's category.

 

[jbaysurfer]

Some people aren't cut out for the scrutiny of having strangers subjectively JUDGE their craftsmanship. I have had some outstanding contest results and miserable ones with the same beer.

As brewers we all implicitly agree to abide by the results whether we like them or not. It's when beers do poorly and our egos get involved that we have trouble abiding.

 

[mdbrewer1]

People are not machines that can be precisely calibrated. That's why sometimes a beer will score a 26 by a person in a competition and a 36 in another competition by a different person. A judges opinion on the beer in front of them is also influenced by the beers in the flight that preceded it.

Additionally, there are many different factors completely out of the entrants control such as how their beer was stored, temperature it was served to the judges, etc. that vary from competition to competition. Beer judging is subjective - if that bothers you don't enter competitions. Volunteer a competition sometime and you'll have a much better understanding of the judging process.

There are a lot of people who complain on the Internet about how unfair or flawed the judging process is but who don't get involved in the judging process or competition in anyway. Judging is not easy. Judges do not get paid. Every judge I've met or judged with has done their very best to determine the best beers in their flight. For better or worse this is the best process in place to judge homebrew - if there's a better one out there then enter their competitions.

I just said I was struggling with the concept. I was not attempting to discredit judges and their ability nor their desire to do their best which your post implies I was doing.

Perhaps the existing process is the best there is. I've never entered a competition, but I've gotten to the point where I'd like to get some feedback from experienced palettes. Neither me nor my friends are there at this point, so your earlier suggestion wouldn't work for me.

Isn't the point of the guidelines to attempt to add some level of "calibration" so that there is, using your terms, something approaching a machine-like objectivity? I realize there a number of factors outside of an entrant's or judge's control, but given the same factors, such as those within a certain competition, I would expect the guidelines to steer the judges to a similar score. If they don't, then I would expect a bad, good, better, best system to be just as good and less confusing to those entering a competition.

Maybe I'm looking at this the wrong way, but if I got a score of 26 on a particular beer, I would take away a very different message than I would on one that received a 36.

 

[mdbrewer1]

The point of a numbering scheme is not to promote consistency so much as it is to quantify the judges thoughts on a particular beer. A 26 and a 36 score gap would trouble me as well, but for different reasons. A 26 is defined as "Misses the mark on style and/or minor flaws." A 36 does not "miss the mark" and should generally conform to: "Generally within style parameters, some minor flaws."

Consistency is achieved by everyone agreeing to judge a beer by the same set of standards, not an implicit agreement that everyone will give beers the same numerical scores.

I think I get where you're coming from here, but it's a subtle distinction, and it explains why the same beer might get those different scores (on different tables/different competitions), but still have the same ordinal ranking within it's category.

Thank you for the response. If the point is to have an ordinal ranking, then wouldn't a plain old 1st, 2nd, 3rd, etc. approach achieve the same results without the implicit quality determination of a 50 point scale?

 

[Simps]

Thanks to everyone jumping in and answering the additional questions about my post. I appreciate the support.

To address the scoring issue - the description of the number ranges is there for the judges, not the entrants. Removing them from the score sheet has even been discussed. A good, better, best system would be more true to the process but when someone sees a number they want more info and if someone sees good they want to know "how good".

I understand both sides of the argument but the number system today should be treated as a continuum and not a cutoff. I support the number but I think the text description goes too far. Unfortunately they get around to updating this stuff once every...7 years I think it has been. Who knows if or how they will ever change this...?

 

[jbaysurfer]

Thank you for the response. If the point is to have an ordinal ranking, then wouldn't a plain old 1st, 2nd, 3rd, etc. approach achieve the same results without the implicit quality determination of a 50 point scale?

I'm not sure what you mean by "the implicit quality determination of a 50 point scale?"

But I think the points system is designed more for common application of the style criteria then it is for consistency. The adherence to the style criteria itself is what should drive consistency.

 

[jbaysurfer]

Thanks to everyone jumping in and answering the additional questions about my post. I appreciate the support.

To address the scoring issue - the description of the number ranges is there for the judges, not the entrants. Removing them from the score sheet has even been discussed. A good, better, best system would be more true to the process but when someone sees a number they want more info and if someone sees good they want to know "how good".

I understand both sides of the argument but the number system today should be treated as a continuum and not a cutoff. I support the number but I think the text description goes too far. Unfortunately they get around to updating this stuff once every...7 years I think it has been. Who knows if or how they will ever change this...?

They're two years overdue as we speak. I do think that goes to credibility for the BJCP. The current guidelines are 6 years old. A lot has happened in the beer world since then.