Trying To Understand BJCP Judging Process...Long Read

[Kidagora]

I wrote this post and deleted it several times because I thought contrasting and comparing my own criticism with those of BJCP judges was too lengthy. I do find myself disagreeing with their comments to my detriment. Just didn't think my submissions would do well/deserved the attention. I just have a few questions about the process as this is the first time I have submitted to competition.

I noted one of the beers I had received the scores of 31, 31, 34. Are there three judges for a first round and only one round? Do only three people get to try my beer or are there more depending on the event (sometimes 4, etc.)? Is my beer compared against all entries or only a small group of those entries at a time? There were 61 in the IPA category. Are they broken down into groups? To what extent are the entries judged against others in their category and to what extent are they judged against technical/stylistic adherence? I understand these scores are respectable but with those scores it went on to a mini-BOS where I received a final score of 39. How did it go from an average of 32 (between those 3) to a final score of 39? Seems like a steep jump. How did my beer improve to a 39 when compared to others? Who added input and improved the score?

I also noticed some comments from the judges that were in direct contradiction to BJCP guidelines for style. One criticism stated my SRM was off and the judge gave a numerical value for mine...which was exactly on the low end for the style for which I submitted. He then proceeded to recommend a darker appearance. I feel like I may have been docked even though I followed style/guidelines...to which they acknowledged. And the standard descriptions for the hops/fruit/ingredients I added did not show in the judges perception. Mostly vague comments on citrus/pine/sweet/malty. I even found the judges disagreeing between sheets. One said perfect corn flavor. The other said not enough. One said acceptable carbonation. The other said very low. There was also no mention of my use of watermelon in a cream ale (I clearly mentioned in labeling). With so much disparity even among judges, this kind of shakes my faith in the process. Can they talk to each other about the beer? I find myself discovering new perspectives when I hear what other people mention.

I'm just confused. I would really like to put my two cents in and begin studying for BJCP. I just want to be fair in my tasting/assessment and now I don't feel like I can even judge my own beers.

 

[brettwasbtd]

On my phone so I can't do nice cut and paste with ease, but much of the questions in your second paragraph can be answered by reading the literature on the bjcpb website. Find and read the one on how to run a competition.

For the scoresheet particulars there are a lot of variables at play. What was the judges rank? Everyone senses and perceives beer attributes a little differently. Regardless, the judges do discuss and come to a consensus on the beer and is usually pretty accurate.

If def recommend becoming a judge, you stay to understand the system and can help improve the quality of judging in your area

 

[Pappers_]

I'll address the process first.

Beers are grouped within a style and judged within a flight. So, if there are 30 entries in an IPA flight, there might be three sets of judges evaluating that flight - ten each. Each set of judges includes at least two judges, but can include three.

The judges are not comparing entries against each other, but rather are evaluating the beer in front of them. After each judge has filled out the score sheet for a beer, they then discuss the beer briefly to at least assure there is not a huge disparity in the scores.

Each set of judges push forward any beers that they evaluated and think might be place winners - mini BOS. The senior judges in the flight then re-taste the beers and determine 1st, 2nd and 3rd places.

 

[kombat]

There was also no mention of my use of watermelon in a cream ale (I clearly mentioned in labeling).

The judges do not get to see your labeling while judging. They know nothing about your beer except what category it's supposed to fit into. They taste it, and using their own expertise and knowledge of what's appropriate for the style, provide scoring and feedback. But if you added watermelon, and mentioned it on the label, that's meaningless because they never see that label. They only taste the beer.

 

[ajdelange]

I noted one of the beers I had received the scores of 31, 31, 34. Are there three judges for a first round and only one round? Do only three people get to try my beer or are there more depending on the event (sometimes 4, etc.)?

This very much depends on the particular competition, the number of entries, the number of judges that volunteer for it and the qualifications of those judges. If there are a small number of judges you may have only two judging your beer. If there are a lot four. Four is usually the upper limit but there is really no reason there couldn't be more.

Is my beer compared against all entries or only a small group of those entries at a time? There were 61 in the IPA category. Are they broken down into groups?

Yes. In the case of 61 entries is a group the organizer will try to break the beers down into 'flights' of manageable size. Eight is a good size (would take 8 flights in this case requiring 24 judges). Twelve or 13 is the practical upper limit as after judging 13 IPAs many judges can't write. This would require 5 flights and 15 judges.

To what extent are the entries judged against others in their category and to what extent are they judged against technical/stylistic adherence?

That depends entirely upon the judges as influenced somewhat by the direction given to them by the competition organizer. You will find some sticklers for adherence to the guidelines. You will find some who are more interested in controlling the opinions of the other judges than accurately evaluating the beer, you will find levels of experience which vary from the first BJCP level to professional brewers with years of experience in the industry. Each comes with his personal biases and no two have the same instrumentation (some people cannot taste/smell diacetyl for example). Women have better palates than men.

The goal is, of course, absolute evaluation but each panel will be subconsciously if not consciously comparing the beers in the flight to one another. The first beers tasted almost always score higher than the last because of palate fatigue and other factors. This often results in reduction of scores on the early beers at the completion of the flight. There will be no comparison to other beers in the category until BOS and there only to those promoted. If your beer goes to mini BOS it will be compared to the other bests from each flight. In mini-BOS the selection is made mostly by comparison to the other beers as opposed to adherence to the guidelines.

People are suggestible. I recently took two growlers of a Vienna to a party where there were several very senior judges and a couple of pros. One growler has some Sinamar added to it so the beer was a bit darker. I asked many of these guys to compare and contrast. You would be amazed at the answers I got. One guy's S.O. said she really didn't know anything about beer but they both tasted the same to her. The only right answer I got! BTW, some of these guys still aren't speaking to me.

I understand these scores are respectable but with those scores it went on to a mini-BOS where I received a final score of 39. How did it go from an average of 32 (between those 3) to a final score of 39? Seems like a steep jump. How did my beer improve to a 39 when compared to others? Who added input and improved the score?

The mini-BOS panel. Though the mini BOS panel may have included one (or more) people from the panel that promoted your beer it isn't likely that all three were the same group. One or more of the BOS panelists thought you beer was better than 31/31/34. Note also that the 31/31/34 may well indicate that one of the judges on the original panel thought that your beer was a 39 but got beaten down by the other two who thought it was a 28 under a competition policy that the score spread was not to be more than x points (this is common). "Well if you really think it's that great I can come up to a 31 but no more than that'." "I do but I'll come down to a 34 if you guy's will come up to a 31." Is a typical conversation.

I also noticed some comments from the judges that were in direct contradiction to BJCP guidelines for style. One criticism stated my SRM was off and the judge gave a numerical value for mine...which was exactly on the low end for the style for which I submitted.

Unless this guy has extensive experience with instrumentally measured beer color he has only a very rough idea as to what the color is. There are various "guides" that can be used to evaluate beer color but they are very, very crude. It is impossible to print most beer colors using conventional CMYK inks as their gamuts are very limited and beer colors are very pure as the beers get dark.

He then proceeded to recommend a darker appearance. I feel like I may have been docked even though I followed style/guidelines...to which they acknowledged. And the standard descriptions for the hops/fruit/ingredients I added did not show in the judges perception. Mostly vague comments on citrus/pine/sweet/malty. I even found the judges disagreeing between sheets. One said perfect corn flavor. The other said not enough. One said acceptable carbonation. The other said very low. There was also no mention of my use of watermelon in a cream ale (I clearly mentioned in labeling). With so much disparity even among judges, this kind of shakes my faith in the process.

Welcome to beer competition! Few of the judges really have the experience to give you a good evaluation. Look at the judges rank on the scoresheet and take his experience into consideration. If you stick with the competition thing then you will get to know who the judges are and will know how good they are at judging, whose scoresheets to accept and whose to toss. A certified judges advice to use less crystal malt in a recipe when you didn't use any is obviously not very good advice. Understand that many of these guys are grasping at straws and writing down comments for the sake of writing down comments. Remember that what they taste is influenced in large measure by what they see, hear from other judges... That's why tasting in the commercial world is done double blind.

Can they talk to each other about the beer? I find myself discovering new perspectives when I hear what other people mention.

Yes they can and should so that scoresheets are not inconsistent between them. Many competitions do QC on scoresheets to try to prevent this sort of thing. Many don't.

I'm just confused. I would really like to put my two cents in and begin studying for BJCP. I just want to be fair in my tasting/assessment and now I don't feel like I can even judge my own beers.

By all means do this because there are few ways to learn as much about beer as you will by passing the BJCP exams. Remember that taste is a subjective thing and that your tastes may be different from other peoples. Also remember that the BJCP guidelines were written down by a bunch of guys sitting around the table. A different bunch of guys might have come up with different guidelines. The best way to learn to evaluate beer is to drink it with knowledgeable people. Find out who the good brewers in your area are and drink with them. Tell them what you did (process) and what materials you used (this is not information available to BJCP judges) and get their feedback. This will be worth more than a thousand scoresheets from a competition.

 

[Pappers_]

The judges do not get to see your labeling while judging. They know nothing about your beer except what category it's supposed to fit into. They taste it, and using their own expertise and knowledge of what's appropriate for the style, provide scoring and feedback. But if you added watermelon, and mentioned it on the label, that's meaningless because they never see that label. They only taste the beer.

That's true, unless the category involves special ingredients, such as fruit beers, where the special ingredient listed by the brewer is shared with the judges.

 

[kombat]

That's true, unless the category involves special ingredients, such as fruit beers, where the special ingredient listed by the brewer is shared with the judges.

Yes, of course, my mistake, I should've mentioned this exception.

 

[progmac]

Kidagora -- i feel like you've touched on a lot of the difficulties of judging beer, wine, bbq, or anything. It's subjective.

I think it is important to aim for a beer that is in the middle to high end of the style guidelines and high in intangibles. A lot of time when judges start picking at SRM or other things they don't like, the problem is that they know the beer isn't at the top of the flight, but they aren't quite sure how to articulate why.

You also have to consider palate fatigue, sensitivity, and age of the judges. What I consider an IPA, I pretty much have to enter as a pale ale; what I consider a double IPA gets entered as a regular IPA, and what I consider an undrinkable hop bomb might get entered as a double IPA.

But I do think palates could be better trained. I've seen national judges think coriander was hops. I've had judges go on about noble hop flavor in a beer with just bittering hops. Experiences certainly vary, frustratingly so sometimes.

And other times, it is just confusing. I had a Dortmunder receive a combined score of 47 with just gushing praise from both BJCP certified judges and then it got 4th in the round.

Good luck, and I hope your positive experiences outnumber your frustrations.

 

[Kidagora]

Wow...

Thanks for all the great information. I guess I'm just disheartened that there is such a tremendous subjective element to it. The major differences in opinion worries me (31 vs 39). I'm guessing a lot of people experience multiple different awards/scores on the same beer across different competitions. On the one hand, the diversity among scores can be fun but I also see some frustration with inconsistency across competitions.

The judges I had for my IPA were National (31), Certified (31), and Rank Pending (34). I tried to extrapolate scores relative to Round/Final and I'm guessing the other IPA's up for the mini-BOS must have averaged 25-30 in their round/flight and received a final score of 30-37 compared to all of my scores. I just wish I knew the status and comments of those that suggested the 39 for mine. Is it really a 39? Or is it only 39 compared to other IPA's? Or did some senior judges think it was 39 overruling those with less experience? It's almost like having a beer with friends and everyone says "it's pretty good" and then you hear them talk to someone else about it and they say "it was amazing! I loved it". I believe this beer was numbered 2 in the sample/flight but nothing was entered for "out of _". Palate fatigue may have been at play in this one.

Special malts (biscuit/melanoidin) were used in <3% increments for the IPA and I loved the malt profile. Comments made it sound like a flaw of style with reference to English IPA's and the biscuit/cookie flavor. It still scored high...which furthers my confusion. Shouldn't I have been penalized for coming too close to an English IPA and out of category. I think the beer itself was in the same range of say a Zombie Dust or Ballistic.

The judges I had for the watermelon cream ale were National (33) and the second judge did not mark/identify his rank (33). I feel the watermelon may have given the impression of too much hops. I wanted a light/hybrid beer to compliment the light flavor of watermelon and I wanted a more historical approach to cream ale (dry hopped/IBU 30) with a fruit twist. I personally thought this beer too bitter/hot/lacking crispness but the flavor was great. I feel like I was lucky with the category I placed this under.

I appreciate the scoring and feedback but I find myself devising recipes based on a combination of personal preference in ingredients and adherence to style and had no idea they were so rigorous. With so many new styles coming out and the new 2014 additions pending, I'm encouraged that many beers that people "experiment" with or personalize will no longer be docked/penalized. I see it as freeing homebrewers from an antiquated dogma where only those brews that fit a rigid set of rules can succeed. Homebrewing has come a LONG was in the past 6 years and I've only been doing it for 1 1/2. Just my two cents.

I appreciate all the feedback. I'll be scouring the BJCP website and seeing if I can meet up with a local beer club for tastings.

 

[kombat]

I'm encouraged that many beers that people "experiment" with or personalize will no longer be docked/penalized. I see it as freeing homebrewers from an antiquated dogma where only those brews that fit a rigid set of rules can succeed.

BJCP competitions judge beers based on adherence to traditional styles. It's a competition to see who can make the best, most perfect example of each recognized style. If judging discarded those guidelines and instead were based on simply who made the best beer, it would be entirely worthless. Everyone has different tastes. Some judges might like your beer, others might dislike it. What would be the value in such a competition? What could you brag about, other than "Judge #12 said my beer was heavenly! Judge #7 thought it tasted like motor oil, but who cares? #12 said 'heavenly'!"

Say my local baking club is having a cheesecake contest. I enter my grandmother's family recipe for angel food cake. It's the best angel food cake you've ever tasted. All the other cakes entered are cheesecakes, but they're all pretty mediocre. Should my angel food cake win the cheesecake competition?

The guidelines help remove at least a little subjectivity from the competition. It allows the competitors to know whether or not they're able to meet various criteria, or whether their technique is lacking. Personally, I don't like Belgian beers. But if you brewed a perfect example of a Belgian that nailed the style guidelines to a 'T', I'd have to score you accordingly. Despite the fact that subjectively, I'm not a fan of Belgians, I would have to give you a good score, because you objectively met the criteria for that style. At least a degree of subjectivity is removed from the equation, and the playing field is leveled for all competitors, and thus the competition becomes more meaningful.

 

[ajdelange]

You are taking this much too seriously. You are assuming that there are absolutes involved here and there are but only peripherally. The BJCP system was put together by engineers trying to bring engineering discipline to subjective judgement. It can't be done but the result is better than the total chaos that would result if the attempt were not made. If you think beer competitions are ridiculous go to a dog show.

The best way to answer the questions you are asking is to get as deep into the competition thing as you possibly can. Find a club that is involved in the BJCP program i.e. one that trains judges, runs competitions, has several National and Master judges in its membership etc. Get certified and judge as often as you can. Travel to competitions away from your home base. Advance in rank. Participate in the running of competitions first as steward, then as judge and finally as competition organizer. Become an exam proctor and grader.

The one thing you will not get out of this is a lot of useful feedback for improvement of your beers. The fact that the judges don't get any information about your process (except in the case of unusual ingredients) precludes this. Some judges are very skilled and can tell that you used old extract rather than fresh grain but they are rare and ultimately they are still guessing. So what will you get out of it? We hope some fun but we are certain a broad base of brewing knowledge that will definitely make a better brewer out of you.

 

[progmac]

The one thing you will not get out of this is a lot of useful feedback for improvement of your beers. The fact that the judges don't get any information about your process (except in the case of unusual ingredients) precludes this. Some judges are very skilled and can tell that you used old extract rather than fresh grain but they are rare and ultimately they are still guessing. So what will you get out of it? We hope some fun but we are certain a broad base of brewing knowledge that will definitely make a better brewer out of you.

I think that entrants should absolutely expect this. In fact, suggestions for improving beer are, i think, a cornerstone of judging practice.

I have received very valuable comments from judges regarding beer balance, strength, and so on. The token comments like, 'check sanitization' are never useful, but the perception comments are often very valuable.

 

[Kidagora]

Perhaps I am taking this too seriously. I come from a science background and try to be as objective/critical/empirical as possible. I am merely curious and perplexed how my perceptions can be so different amongst professionals and even their opinions can differ so greatly.

I don't envy the role of judging in the least. I find it incredibly difficult. I am somewhat paralyzed with the notion of even going through the rigorous training to attempt it. I received a significant amount of helpful criticism in the beers I presented. But also mixed messages. English malt taste with American hops is what style? How did mine fit? I think I just wanted the judges to understand what I was attempting and BJCP competitions are not the place to do it. I recognize that now. I wanted a depth of criticism that requires interactive discussion of the process a la clubs, informal competitions, etc. and now I know what I should do.

I once read a quote on the forums here that said something to the effect of "brewing beer YOU like and brewing a beer that will WIN a competition are two different things". Is there any truth to that?

I often think of beer as similar to music in that styles/categories are better perceived by certain critics/historians/judges than others. You may be able to recognize basics but unless you are well steeped in the history, tradition, and style you may not be a good judge. Some are well versed in all styles. Both are sense related and have guidelines to their respective categories. Is that a reasonable analogy?

I don't want anyone to think I'm discounting or trashing the competitions, the judging process, or the judges themselves. I am truly grateful for the experience and look forward to many more.

 

[freisste]

I once read a quote on the forums here that said something to the effect of "brewing beer YOU like and brewing a beer that will WIN a competition are two different things". Is there any truth to that?

There is absolutely truth to this.

I entered a few beers into a competition last year. I got pretty good feedback. Some was a little irrelevant and I ignored it. Some was really helpful and I adjusted my process.

I also recently talked to a judge at a homebrew party (he was a friend of a friend and he showed up as a drinker, not a judge...in fact, he shared some of his own brew). He was very candid in his comments toward my beer and toward BJCP. He mentioned things I could change, I showed him my grain bill and he have good criticism. He said the BJCP was good, but criticized its structure in certain ways (his example: my cream ale could be 10x better in every way than cream ale x, but my color is out of style so I get docked and my score is lower because cream ale x was in the middle of every attribute category). But he also mentioned it is really the only way to do it.

It may not be a perfect system, but it is pretty good.

I will say that my favorite remarks from the competition were the ones that didn't have anything to do with style. One judge called my cream ale "a great law mower beer. I'd enjoy another." In the end, who gives a $h!+ about color when the beer is good. You just have to keep in mind the meaning of the scores. The best Belgian on the planet would score a 5 in category 1 (and vice versa).

 

[ajdelange]

Perhaps I am taking this too seriously.

You should take it seriously if you have a passion for beer. What you should not take seriously is a scoresheet that you got from some competition unless you know the judge to the point where you respect him. Just rank is not sufficient. I once got a comment from a Grand Master judge that said "I do not condone the use of the sparkler with southern style ales" (in this competition the beers were served through engines and the organizer had put a sparkler on for some reason). I'm supposed to take this seriously? My ales aren't northern or southern. They are MO plus some crystal with EKG or Fuggles. That scoresheet is a treasure though. I hope I never lose it.

I come from a science background and try to be as objective/critical/empirical as possible.

You are in the right place. "Wine is made by farmers. Beer is made by engineers." The problem is that you can't be totally objective when your 'measurements' are subjective. Take as an example the color you mentioned in your first post. SRM is 12.7 times the absorption of 430 nm light in passing through 1 cm of beer. Can't get much more objective than that. But I can show you two beers that have exactly (within measurement accuracy) the same SRM but are different colors! Not to mention the fact that the color you see depends on the light you are looking at the beer with, the colors of the surround and adaptations. Another example might be bitterness. Even the prescribed ASBC and EBC measurement methods return a result which "adequately represents bitterness for beers made with fresh hops.." I can give you two beers with the same IBU that have very different bitterness profiles. Etc.

I am merely curious and perplexed how my perceptions can be so different amongst professionals and even their opinions can differ so greatly.

The first mistake is to assume that they are professionals. They are by no means professionals. The good ones are few and far between. A guy that just took the test and is waiting to see if he has passed doesn't know squat compared to a professional brewer or some guy who has been judging for 20 years, been to Germany, the Czech Republic and the UK, brews a lot and isn't a BS artist (yes, unfortunately there are some of those). As to how their opinions can differ so much ask your colleagues about some hot button political issue. Unless you are in a university or similar environment where everyone is uniformly pink you'll probably get quite a diversity of opinion. Also go back and reread my little anecdote about fooling some real pro's with a bit of Sinamar.

I don't envy the role of judging in the least. I find it incredibly difficult.

It is! If you do a good job you go home mentally exhausted.

I am somewhat paralyzed with the notion of even going through the rigorous training to attempt it.

Don't be. You will be rewarded with much deeper knowledge of beer and brewing than you can acquire by any other means (short of going to Herriot-Watt or somesuch).

I received a significant amount of helpful criticism in the beers I presented. But also mixed messages. English malt taste with American hops is what style?

Some dramatic changes in brewing science have taken place in fairly recent times. Most significant among these, IMO, is the fact that RO systems are now affordable by home (and craft) brewers. This removes the tyranny that water used to impose. Add to this new malt and hop cultivars which can be shipped anywhere in the world and anyone can brew any beer anywhere. At the same time economics demand that lagers not be triple decocted nor lagered for 3 months and you find 1)That the meaning of styles is fading and 2)The styles that do survive are not what they used to be. New styles are popping up all the time in the same way as the bizarre dishes of fusion cooking. This obviously makes the job of setting forth style guidelines more difficult.

How did mine fit? I think I just wanted the judges to understand what I was attempting and BJCP competitions are not the place to do it. I recognize that now. I wanted a depth of criticism that requires interactive discussion of the process a la clubs, informal competitions, etc. and now I know what I should do.

Yes, face to face is the best way to get this kind of interaction plus clubs can be fun.

I once read a quote on the forums here that said something to the effect of "brewing beer YOU like and brewing a beer that will WIN a competition are two different things". Is there any truth to that?

Yes, absolutely! In your technical work you don't speak of optimality without describing the optimality criterion. Same here. The 'best' beer can be the one
that
1) That you really like
2) That pleases your SO
3) That your friends really like
4) That wins competitions
5) That sells

These can be quite different beers. It is amusing to go to an investors' meeting for the brewpub I'm affiliated with and hear some of the investors (many of whom are very experienced and avid brewers and high ranking judges) say that the Saison isn't really a very good Saison and have the brewer patiently point out that this may be the case but that it is a good seller. It is amazing how many of these guys would rather lose money by selling a beer that meets their personal conception of what a good Saison is than make money by giving the great unwashed what they want.

I often think of beer as similar to music in that styles/categories are better perceived by certain critics/historians/judges than others. You may be able to recognize basics but unless you are well steeped in the history, tradition, and style you may not be a good judge. Some are well versed in all styles. Both are sense related and have guidelines to their respective categories. Is that a reasonable analogy?

I think music is an excellent analogy and use it often. A well made Pilsner is like the voice of a Strad or a first tier soprano. In a Weizen the hops should be like the violas. You don't hear them but if you leave them out there is diminishment. Etc.

But this comment tells me you are going to be dealing with what I call 'intellectual impedance mismatch'. Most of the people who will be filling out scoresheets for you think heavy metal is good music.

I don't want anyone to think I'm discounting or trashing the competitions, the judging process, or the judges themselves. I am truly grateful for the experience and look forward to many more.

There are many flaws in it but as noted before I really encourage you to get into it as deeply as your time and other commitments permit.

 

[ajdelange]

In the end, who gives a $h!+ about color when the beer is good.

As I just finished writing a chapter about beer color for an upcoming text on quality control I DO!!!

 

[freisste]

As I just finished writing a chapter about beer color for an upcoming text on quality control I DO!!!

I agree. I only meant to point out a potential difference between a crowd pleaser and a competition winner.

 

[ajdelange]

That was clear but as the damn thing consumed the last 4 moths of my life (and as I wait in terror to see what the editor is going to do to it, or make me do to it) I couldn't resist commenting.

 

[freisste]

That was clear but as the damn thing consumed the last 4 moths of my life (and as I wait in terror to see what the editor is going to do to it, or make me do to it) I couldn't resist commenting.

Fair enough. Hope it turns out well in the end.

 

[kombat]

my cream ale could be 10x better in every way than cream ale x, but my color is out of style so I get docked and my score is lower because cream ale x was in the middle of every attribute category). In the end, who gives a $h!+ about color when the beer is good. You just have to keep in mind the meaning of the scores.

It's my understanding that "Appearance" only counts for 3 points (out of 50). So if your colour was off, but you had the right level of head retention, lacing, clarity, etc., then you'd still get at least 1 point, probably even 2. Meaning being "out of colour" for the style would only drop your overall score by 1 point. If your beer were truly "10x better in every way than cream ale x," then your final score would be way, way higher than his, even with the 1 point deduction for the colour being off.

 

[freisste]

It's my understanding that "Appearance" only counts for 3 points (out of 50). So if your colour was off, but you had the right level of head retention, lacing, clarity, etc., then you'd still get at least 1 point, probably even 2. Meaning being "out of colour" for the style would only drop your overall score by 1 point. If your beer were truly "10x better in every way than cream ale x," then your final score would be way, way higher than his, even with the 1 point deduction for the colour being off.

Just looked back at my scoresheet. You are correct in that it only counts toward 3 points. (Compared to 12 for aroma and 20 for flavor...the way it should be, I suppose.)

I only meant that to be an example. In fact, both judges awarded me 3/3 for appearance

 

[ajdelange]

I'll let you all in on a little secret. Some judges work from the bottom up (is the color right? is the carbonation right? etc) and add up the scores. Many, if not most, work from the top down i.e. taste the beer, decide it's a 39 then juggle the numbers to sum to 39. Color is an obvious wiggle room parameter as no one really cares that much and no one in a judging session really has any idea what the actual color is as compared as to what it's supposed to be. Obvious cases like a jet black Pilsner or a blond stout excepted.

 

[weirdboy]

I'll let you all in on a little secret. Some judges work from the bottom up (is the color right? is the carbonation right? etc) and add up the scores. Many, if not most, work from the top down i.e. taste the beer, decide it's a 39 then juggle the numbers to sum to 39. Color is an obvious wiggle room parameter as no one really cares that much and no one in a judging session really has any idea what the actual color is as compared as to what it's supposed to be. Obvious cases like a jet black Pilsner or a blond stout excepted.

I would say I fall somewhere in between those two. It is usually plainly obvious when I first taste a beer as to what broad rank it falls into among "problematic", "fair", "good", "excellent", and "outstanding". Of course there are some beers that are on the borders of those ranges, and especially for the lower-scored beers, I pay a lot more attention to the details of exactly what is wrong with it, and what is right, so that I can provide better feedback. But, generally speaking you can put it in a certain range very quickly.

Then I basically just do a bottom-up approach and assign scores based on how well each parameter matches the guidelines. Usually the total score ends up in the range I had in mind when I first tasted it, and that's the end of it. If it doesn't, I then go through and look more carefully at the scores and at my comments, and figure out why the score isn't in the range I thought it should be initially. Usually, that difference indicates to me that I missed something in my comments that I should have provided better feedback on, but forgot to.

Then the judges for that beer all share their scores and discuss any differences of opinion, and maybe re-taste and make score adjustments again after that.

 

[mattd2]

...One guy's S.O. said she really didn't know anything about beer but they both tasted the same to her. The only right answer I got! BTW, some of these guys still aren't speaking to me...

I think the point there is the SO has no ego to prove she is the best judge - and so calls it like it is

... Take as an example the color you mentioned in your first post. SRM is 12.7 times the absorption of 430 nm light in passing through 1 cm of beer. Can't get much more objective than that. But I can show you two beers that have exactly (within measurement accuracy) the same SRM but are different colors! ...

Given that is a measure of the absorption of a specific wavelength (colour) light - all that is being measure is "darkness" not colour, which is why 2 identical SRM values can be different colours.

I'll let you all in on a little secret. Some judges work from the bottom up (is the color right? is the carbonation right? etc) and add up the scores. Many, if not most, work from the top down i.e. taste the beer, decide it's a 39 then juggle the numbers to sum to 39...

I have always thought working from top down (or bottom up if looking at the score sheet ) is a really bad way of judging beer and goes against the point of the scoresheet in the first place (but it is advocated by the BJCP to use this method). I thought the point of the scoresheet was to break down the beers "components" into specific areas and give them a weighted score. As in the Appearance / Flavour example. By straightaway deciding this is a 39 beer filling out the rest of the sheet is just a box ticking exercise, adding little value and potentially the reason for some of the "interesting" comments when a judge is trying to justify an arbitrary score to get it to match his first call.

What would be interesting to look into is the differences in score spread when comparing judges that work top down vs. bottom up. I would hypothesis that top down approach would lead to a greater average spread due to the initial subjective scoring method.

 

[mattd2]

I would say I fall somewhere in between those two. It is usually plainly obvious when I first taste a beer as to what broad rank it falls into among "problematic", "fair", "good", "excellent", and "outstanding". Of course there are some beers that are on the borders of those ranges, and especially for the lower-scored beers, I pay a lot more attention to the details of exactly what is wrong with it, and what is right, so that I can provide better feedback. But, generally speaking you can put it in a certain range very quickly.

Then I basically just do a bottom-up approach and assign scores based on how well each parameter matches the guidelines. Usually the total score ends up in the range I had in mind when I first tasted it, and that's the end of it. If it doesn't, I then go through and look more carefully at the scores and at my comments, and figure out why the score isn't in the range I thought it should be initially. Usually, that difference indicates to me that I missed something in my comments that I should have provided better feedback on, but forgot to.

Then the judges for that beer all share their scores and discuss any differences of opinion, and maybe re-taste and make score adjustments again after that.

I am not BJCP judge (or even an experienced beer taster ) but I think I take a similar approach to you with a minor difference. Yes the first impression will give away a lot, but I try to follow the standard methodology of smell, look, taste - and run through the scoring then once at the bottom add the score and then check it against the "problematic - outstanding" score ranges to make sure my overall impression of the beer fits the score I have given. I also use the 3 keyword / 5 scale for "stylistic / technical / intangibles" as a double check. Usually it is in line, if not I'll revaluate.

 

[ajdelange]

I think the point there is the SO has no ego to prove she is the best judge - and so calls it like it is

On the deck that afternoon that was definitely the case but the same experiment has been done in the laboratory with the same result. The same beer tastes different if it is a different color. We 'drink with out eyes' as Dr. Bamforth says.

Given that is a measure of the absorption of a specific wavelength (colour) light - all that is being measure is "darkness" not colour, which is why 2 identical SRM values can be different colours.

There is a bit more to it than that. Beer absorption spectra normalized to the 430 nm reading are nearly identical. Were they exactly the same then SRM would be a complete description of beer color (under any illuminant, for any observer and for any width of glass) but they do deviate slightly. The actual deviations are small enough that the SRM (or EBC) measure can be used to compute full color (given path, observer and illuminant) i.e. luminance, hue and saturation (or, if you prefer R, G and B wrt to an arbitrary set of primaries) can be obtained and the accuracy is surprising.

I have always thought working from top down (or bottom up if looking at the score sheet ) is a really bad way of judging beer ... I would hypothesis that top down approach would lead to a greater average spread due to the initial subjective scoring method.

Whichever method you use when you are finished with adjustments to obtain a consensus what you have is an illustration of some of the limitations that one finds when trying to 'score' subjective judgements. But as I said in an earlier post if you think beer judging is arbitrary go to a dog show.

 

[mattd2]

...Whichever method you use when you are finished with adjustments to obtain a consensus what you have is an illustration of some of the limitations that one finds when trying to 'score' subjective judgements. But as I said in an earlier post if you think beer judging is arbitrary go to a dog show.

I guess my point was you would expect more variance in spread from giving a score out of 50, compare with the sum of number of defined subcategories.
The first 4 subcat. deal with very defined properties (although still subjectively) aroma / appearance / flavour / mouthfeel. The last subcat is for that how good is this beer - overall impression.
By going top down you are pretty much being subjective to the beer as a whole and then trying to define why. Not define why and then see what comes out.

The top down approach probably bothers me so much because of how it is used in other areas for my job - when preparing cost estimates (for capital work or engineering costs) it is very common to put in all the reasoning why it will cost $X, only for the person signing it off to say "no it should only cost 0.5x $X because that's my gut feeling" with no reasonable basis.

 

[ajdelange]

The top down approach probably bothers me so much because of how it is used in other areas for my job - when preparing cost estimates (for capital work or engineering costs) it is very common to put in all the reasoning why it will cost $X, only for the person signing it off to say "no it should only cost 0.5x $X because that's my gut feeling" with no reasonable basis.

But he does have a reasonable basis: his experience. He should be able to say "That's not a $3M job. Let me see your labor estimates" and then find, for example, that you used junior engineers where you could have used senior techs or something like that. If he isn't right when he speaks, based on his experience, or gut if you prefer, then either he won't be in the position of signing off for very long or the company won't be in business very long.

I imagine this does map over into beer judging pretty closely. An experienced judge knows that a particular beer is a 39 whether he does bottom up or top down. If he does bottom up and gets a score other than 39 then he knows his individual scores are wrong and goes back to find out where the error is.

As you advance in judging and in your career you too will, we hope, be rewarded with the blessings of experience. It's a normal part of your career arc. And it's not just a 'nice to have'. When you get older you can't do all that math in your head the way you did in youth and must rely, therefore, on experience.

 

[Kidagora]

Also, what is the general consensus regarding newer styles? Styles like white IPA, Black IPA, West Coast, Hoppy Ambers, Rye, etc. They've been around for quite a few years and I'm sure judges have been aware...but have people entered these into competition and won? Could they even enter them into competition in a category other than experimental? And why haven't the styles been added? Is it because they are so relatively new and a standard has not been established? Who determines the guidelines associated with the newer styles?

What I was trying to express earlier was that many people love and enjoy these styles and brew outside normal convention to obtain these styles, including professionals. I believe these styles have a base at their core and it seems like that base could/should be established and still allow competition. Similar in the way that I brewed a cream ale with watermelon. The base was cream ale. But a specialty ingredient was added. Would it be feasible to have a category under Light/Hybrid with a sub-category of "Specialty" instead of relegating it to experimental and pitting it against all the other beers that could be based on baltic porter, IPA, pilsner, etc.

 

[freisste]

There are specific categories within BJCP. At some point, new categories are added due to popularity. Great current example is the black IPA. I just entered beers into a competition that allowed them. Last year they allowed them as well and there was a note on the entry page letting people know how exactly to enter them and that they were working on judging criteria.

I think the issue with your watermelon cream is not that it was bad, just that it didn't fit into the style well. Intentional or not, your beer had an off flavor. Now, if people try your beer, decide that it is delicious, brew their own delicious version and share it, etc., then maybe this watermelon thing will catch on. In 10 years there at be another new category entered into the BJCP guidelines.

The issue with that is there are already probably 100 different subcategories within BJCP. If everybody who brewed a beer with their own additions got their own style, the system would fall apart.

This isn't to say I think having different "wildcard" subcategories in each group is a bad idea. (So category 1 gets a new subcategory 1f, category 2 gets 2d, 3 gets 3c, etc.) But again, how would I compare your watermelon cream ale to someone else's jalapeño cream ale vs a kiwi cream ale?

 

[Smellyglove]

The OP is exactly why I never will enter a beer to a competition if it's not for curiosity about how different scores you can get for that one beer. It's like high school. One teacher would give you a 2, another would give you a 5 for the same project.

 

[ajdelange]

From which we may surmise that were it not compulsory you would not have bothered with high school either.

 

[Smellyglove]

From which we may surmise that were it not compulsory you would not have bothered with high school either.[/QUOTE

Nice. Tell me why you believe that please.

 

[ajdelange]

First: check the definition of surmise.

Second: If your reason for avoiding A is that it is like B it almost goes without saying that you will also wish to avoid B.

 

[Wayne1]

Also, what is the general consensus regarding newer styles? Styles like white IPA, Black IPA, West Coast, Hoppy Ambers, Rye, etc. They've been around for quite a few years and I'm sure judges have been aware...but have people entered these into competition and won? Could they even enter them into competition in a category other than experimental? And why haven't the styles been added? Is it because they are so relatively new and a standard has not been established? Who determines the guidelines associated with the newer styles?

The BJCP style guidelines are revised every few years. The last revision was 2008. They are currently in the process of updating the guidelines. The tentative 2014 guidelines are available to peruse here.

The BJCP is an ALL VOLUNTEER organization. There are no professional judges. No one gets paid. It takes time for everyone to get together to make changes.

The Professional Style Guidelines for GABF and World Beer Cup change every year. They are maintained by the Brewers Association and mostly revised by Charlie Papazian. The latest lists 90 styles with subcategories 2014 GABF Style Guidelines while the 2014 BJCP guidelines has 40 styles with sub categories.

 

[Kidagora]

Ah, thanks. I never noticed/paid attention to GAFB or World Beer Cup. Good to know.

I guess I hold the BJCP in high esteem and look to them whenever I tackle a particular style. Every time I approach a style never attempted I look at their guidelines. After a few satisfactory brews, I tailor to my taste. I find the standards are specific and I appreciate the scoring in competition. The variety in each judges comments is surprising given the fact each is to be trained using the same process. The fact that most beers are scrutinized to such a degree is nice to hear. I look forward to seeing its evolution with the growing craft beer scene. As a newb, I'm very excited and eager to know as much as possible and get involved but I stop short of assuming any authoritative/professional status. I feel I'm just too new and inexperienced.

I thought there was mention of compensation for judging in the competition I applied for but not sure if it was for non-BJCP or for all. Or if the compensation mentioned was for status/ranking.

I'll begin the long process of studying and join a local group for guidance. Thanks for all the comments and suggestions!

 

[ajdelange]

The "compensation" is points applied to your judging record. If lucky, you may get lunch.

 

[jsun]

What are entry fees used for, then? Seems like many competitions would rack up thousands of dollars in entry fees.

 

[jekeane]

Entry fees generally cover the cost of the event. Venues, food, advertising, renting a freezer trucks for beer storage, the list could go on. My homebrew club is hosting a 600 entry competition this weekend and I can assure you that the $3,600 in entry fees isn't paying for the full expense of the hotel we are hosting it in for the weekend.

 

[weirdboy]

I would also add that anything over the expense of running a competition generally goes back into the competition in the form of better prizes for the winners.