Trying To Understand BJCP Judging Process...Long Read

[beernutz]

Perhaps I am taking this too seriously. I come from a science background and try to be as objective/critical/empirical as possible. I am merely curious and perplexed how my perceptions can be so different amongst professionals and even their opinions can differ so greatly.

I don't envy the role of judging in the least. I find it incredibly difficult. I am somewhat paralyzed with the notion of even going through the rigorous training to attempt it. I received a significant amount of helpful criticism in the beers I presented. But also mixed messages. English malt taste with American hops is what style? How did mine fit? I think I just wanted the judges to understand what I was attempting and BJCP competitions are not the place to do it. I recognize that now. I wanted a depth of criticism that requires interactive discussion of the process a la clubs, informal competitions, etc. and now I know what I should do.

I once read a quote on the forums here that said something to the effect of "brewing beer YOU like and brewing a beer that will WIN a competition are two different things". Is there any truth to that?

I often think of beer as similar to music in that styles/categories are better perceived by certain critics/historians/judges than others. You may be able to recognize basics but unless you are well steeped in the history, tradition, and style you may not be a good judge. Some are well versed in all styles. Both are sense related and have guidelines to their respective categories. Is that a reasonable analogy?

I don't want anyone to think I'm discounting or trashing the competitions, the judging process, or the judges themselves. I am truly grateful for the experience and look forward to many more.

Of course there is truth in that statement. I have a good friend who owns a BBQ restaurant and has competed on the circuit of BBQ competitions. I don't believe the food he prepares for competition is very close to what he serves in his restaurant.

That is not because he takes more time when in competition versus in the store, it is because competition BBQ has a different set of expectations from the judges than what customers prefer in the restaurant. Competition BBQ has to "stand out" from the rest of the competitor's entries, often in ways like "richness" achieved through the application of margarine when cooking which may be a problem if served continually in a restaurant setting.

 

[mattd2]

But he does have a reasonable basis: his experience. He should be able to say "That's not a $3M job. Let me see your labor estimates" and then find, for example, that you used junior engineers where you could have used senior techs or something like that. If he isn't right when he speaks, based on his experience, or gut if you prefer, then either he won't be in the position of signing off for very long or the company won't be in business very long...

Not generally in the cases I have witnessed time and time again - mostly it is what was set in the budget 5 years prior or a +/-50% cost estimate that is being refined to a better accuracy. The typical answer is "you said it was $5m (+/-50%) 2 years ago and now you are saying it is $6.5M (+/-10%), sorry it has to cost $5M or less"... but that is getting off the point. Trying to spin that back to the BJCP I would equate it to a judge taking only one (or two) of the classic examples of a style and saying that is the 40-50 definition and then saying a competition beer does not deserve to be in that range because it is different from his example (although it is within style, no flaws and a great beer - just different to his experience)... sort of

I imagine this does map over into beer judging pretty closely. An experienced judge knows that a particular beer is a 39 whether he does bottom up or top down. If he does bottom up and gets a score other than 39 then he knows his individual scores are wrong and goes back to find out where the error is...

I would agree that a judge should know roughly the beer will land... but if expert judges can give a total score without having to actually evaluate the components why bother with all that hassle - they could get through more beers, quicker if it was a simple take a sip "that's a XX beer, it has YY going for it / flaws, do this / that" - next beer.

 

[ajdelange]

Renting tables and chairs, renting venue, renting reefer truck, cups, table cloths, Avery stickers, mailing costs for score sheets, reproduction costs for score sheets, palate cleansers, bottled water, lunch, a keg of beer to go with the lunch, pencils, reefer truck, coffee and doughnuts,...

Depending on where you do the competition (do you know, for example, a local brewery or brewpub that will give you space, including in their cold room, free?) some of these expenses will not apply. There are doubtless other expenses that pop up in other situations.

 

[mattd2]

Renting tables and chairs, renting venue, renting reefer truck, cups, table cloths, Avery stickers, mailing costs for score sheets, reproduction costs for score sheets, palate cleansers, bottled water, lunch, a keg of beer to go with the lunch, pencils, reefer truck, coffee and doughnuts,...

Depending on where you do the competition (do you know, for example, a local brewery or brewpub that will give you space, including in their cold room, free?) some of these expenses will not apply. There are doubtless other expenses that pop up in other situations.

I agree, no matter what you have to pay for the entry fees will get gobbled up fairly quickly. Leaving nothing for monetary compensation for the judges.

 

[opiate82]

I'll let you all in on a little secret. Some judges work from the bottom up (is the color right? is the carbonation right? etc) and add up the scores. Many, if not most, work from the top down i.e. taste the beer, decide it's a 39 then juggle the numbers to sum to 39. Color is an obvious wiggle room parameter as no one really cares that much and no one in a judging session really has any idea what the actual color is as compared as to what it's supposed to be. Obvious cases like a jet black Pilsner or a blond stout excepted.

This makes some scoresheets I've gotten back make so much more sense. One of my biggest pet-peeves with the scoresheets I've received is that there was basically no explanation given as to why I was dinged points in certain areas. One specifically was for a APA that scored 41 overall and the judges comments at the end were "Great execution. I have no critiques or recommendations for you." Well then what was wrong that caused you to ding me 9 points?!

It didn't take me very long to realize that once your beer is above a certain score (35ish?) the judges are basically looking for that "intangible element" that pushes it into the World Class category. Basically meaning anything above a certain score is basically pushes the scoring into the subjective territory.

Now when I enter contests I just go in hoping there are no off flavors in my beer. As long as there are no detectable off flavors I pretty much assume it is down to luck as to whether I place or not.

 

[ajdelange]

Not generally in the cases I have witnessed time and time again - mostly it is what was set in the budget 5 years prior or a +/-50% cost estimate that is being refined to a better accuracy. The typical answer is "you said it was $5m (+/-50%) 2 years ago and now you are saying it is $6.5M (+/-10%), sorry it has to cost $5M or less"...

If he's that bad the obvious question is "How do you stay in business?" OTOH if the customer has a requirement that it be < $5M then it's not the manager at whom the finger should be pointed but the customer. It then becomes the manager's job to point out to the customer why <$5M is not reasonable. He should use his experience, backed by the individual line items if necessary, to do that.

Trying to spin that back to the BJCP I would equate it to a judge taking only one (or two) of the classic examples of a style and saying that is the 40-50 definition and then saying a competition beer does not deserve to be in that range because it is different from his example (although it is within style, no flaws and a great beer - just different to his experience)... sort of

That would be what it is like but an experienced judge wouldn't do that. He would know the whole gamut of the style and know it well. He would have been to Europe to taste those beers, for example.

I would agree that a judge should know roughly the beer will land... but if expert judges can give a total score without having to actually evaluate the components why bother with all that hassle -

The best (most experienced) judges (and engineers/scientists) know the answer by intuition (or experience if you will) but verify using analysis. As they become better and better at it they rely less on detailed analysis and more on experience.

Perhaps one has to experience this in order to understand it.

 

[mattd2]

If he's that bad the obvious question is "How do you stay in business?" OTOH if the customer has a requirement that it be < $5M then it's not the manager at whom the finger should be pointed but the customer. It then becomes the manager's job to point out to the customer why <$5M is not reasonable. He should use his experience, backed by the individual line items if necessary, to do that.


That would be what it is like but an experienced judge wouldn't do that. He would know the whole gamut of the style and know it well. He would have been to Europe to taste those beers, for example.



The best (most experienced) judges (and engineers/scientists) know the answer by intuition (or experience if you will) but verify using analysis. As they become better and better at it they rely less on detailed analysis and more on experience.

Perhaps one has to experience this in order to understand it.

I get what you are saying and agree, theoretically, that an experienced judge should be able to nail a score in from first impression. I guess the point is what do we define as experienced judge, master / grandmaster? How many of them are judging at each event? By advocating to the not so experienced judges that the bottom up approach is a good way for them to judge is probably more of my issue (sorry for not being clear).


And for the work thing - yes it is the clients that criticise the cost. And most know that the cost given is realistic for the scope but they just do not have the money to spend. Scope gets cut and the end project does not quite meet the expectations of the end users (who were expecting the $1.5M in extras) and somehow the engineers get called up for it even though it was noted at the very beginning. Ahhhh the life of a engineer/consultant

 

[ajdelange]

I guess the point is what do we define as experienced judge, master / grandmaster? How many of them are judging at each event?

That very much depends on the competition. Some clubs pride themselves on the quality of their judge cadre. Others do not so much. This is related to some remarks I made in an earlier post about getting to know who the good judges in your area are.

By advocating to the not so experienced judges that the bottom up approach is a good way for them to judge is probably more of my issue (sorry for not being clear).

I would never advocate telling new judges to work that way. Perhaps I was not clear on that. I brought it up simply because I know that a lot of experienced judges work that way. Some of this is built into the scoring system in that you have 10 points to assign to 'general impressions'.

Scope gets cut and the end project does not quite meet the expectations of the end users (who were expecting the $1.5M in extras) and somehow the engineers get called up for it even though it was noted at the very beginning.

Oh I see. It's government work.

 

[mattd2]

...I would never advocate telling new judges to work that way. Perhaps I was not clear on that. I brought it up simply because I know that a lot of experienced judges work that way. Some of this is built in to the scoring system in th that you have 10 points to assign to 'general impressions'...

Just to clarify I know you are not advocating for all judges to use this method, just highlighting it is used efficiently by experienced judges. My point was the potential for not so experience judges blindly following what an experienced judge does because it is assumed as the right way to do it. I guess I am reminded of the guidelines on how to judge a beer from the BJCP website where it describes both methods

As for scoring, there are two major methods: top-down and bottom-up. Top-down scoring means that you decide
where the beer should score overall, and fill in the sections to add to that total. Bottom-up refers to the practice of
filling in a score for each section and adding them to a final score. Either way, the score should make sense. Use the
Scoring Guide on the lower left hand side of the scoresheet as a sanity check.

Oh I see. It's government work.

 

[weirdboy]

My point was the potential for not so experience judges blindly following what an experienced judge does because it is assumed as the right way to do it.

This doesn't happen, at least in my experience. I guess you have never actually judged in a competition, so it's probably difficult to grasp how it works in practice, but basically the judges don't communicate with each other about their impressions until they have each filled out their respective scoresheets.

 

[Kidagora]

"But again, how would I compare your watermelon cream ale to someone else's jalapeño cream ale vs a kiwi cream ale?"

Isn't this essentially done in the Specialty Category 23? I would assume that the intangible elements that take a beer from 44 and up would apply and I would think this would be easier if a base style is chosen from which to compare rather than comparing a cherry PORTER vs. a watermelon CREAM ALE vs. mint STOUT etc. In essence, does the watermelon detract from the standard definition of the cream ale or enhance/compliment it? How does the beer stand on its own? It seems with the new BJCP recommendations this is where they are leaning. Not saying experimentals are allowed or going to be, but these styles were experimental not long ago and now have their own proposed category of 21.B. They even admit:

"Beers entered as this style are not experimental beers; they are a collection of currently distinct styles that may or may not have any market longevity. This category also allows for expansion, so potential future IPA variants."

I'm not sure how I feel about the influence of "market longevity" on competition entrants. Seems Grodziskie would never be allowed

To your example, the watermelon cream ale I entered had no mention of off flavors that detracted from style as perceived by judging (via the scoresheet) but I thought it was too bitter and too hot (IMHO). There was a jalapeno cream ale (I found out tonight at a beer club) that was entered in the same category (Experimental #99, I can PM details) that was required only to use corn. We both used corn. We both had a cream ale base. There was no restriction on style or ingredients in the category other than it must have corn. I could have made a standard pilsner (which some did) and entered it and it would have been up to the judges to decide which is better. I imagine this is more difficult in the Specialty #23 category where someone may fit 80% of the criteria for a style but diverge by adding a fruit, spice, malt, or hop out of style. How do you judge a porter vs. a cream ale vs. a stout vs an IPA anyway? Would it be more fair to pit them against each other or their basic style? What does everyone else think? Does this make it too unfair, a mess, less 'pure', complicated, burdensome etc.?

Also thought I'd throw in an update.

Attended my first meeting of one of the major local homebrew clubs and it was amazing. Several members were judges at the recent competition and gave a little more insight into the process and structure of the recent competition.

We then tried multiple beers and I was quite impressed with the assessments and ability to describe in detail multiple aspects of each beer. VERY helpful and informative. Many questions were asked and much knowledge was gained. I HIGHLY recommend anyone who considers themselves a frequent, serious, or curious homebrewer to join a homebrew club if you haven't already. The amount of collective knowledge the members have is breathtaking and sheds a whole new light on perception, process, and potential. I'll be soaking up info like a sponge and may begin the process to get more involved in BJCP. I appreciate all the input and recommendations. Your knowledge, opinions, and insight has opened me up to the complexities of this wonderful hobby.

Cheers!

 

[brettwasbtd]

Judges deal with this in 23 all the time. Most of the time you have a few really great beers that are well balanced and you have to place based on their base beer quality, balance of the special ingredient and the drinkability. I just recently judge this category and the judges were split between the top two. One judge conceited their #1 because he knew he did not prefer ginger in general and thought this was affecting his decision. As a judge he agreed the base beer was flawless and the spicing was complimentary and at a nice level. Obviously there is some subjectivity, and you often do get some intense discussion, but usually there is rationale that makes the consensus decision fair.

The new proposed guidelines will help nature done of the specialty categories. I think the biggest changes are allowing the entrant to provide set descriptors that will help judges appropriately assess the beer out in from of them

 

[Darwin18]

If you really want to expand your knowledge on the competition scene volunteer as a steward at the next closest competition. You'll have the opportunity to see the judging process first hand and try the beers that the judges are evaluating. Not only will it help you to determine off-flavors and their sources in your beers but you'll get to taste the winning beers and see how well yours stack up (at least mentally).