the BJCP so called certification drives me crazy

[Brooothru]

And the other 10% don't know how to add!

Or to paraphrase the Great Yogi, "90% is skill. The other half is Luck."

 

[Bobby_M]

I was under the impression that beer would be stored chilled and not at room temp.

Some of the drop off locations where I live have walk-in coolers and I try to use those instead of place where my beer might be sitting at room temp. They do say if they get too many they will take them all out, so I know or expect my beer to be at room temp for some amount of time.

I bottle and drop off my beers as close to the deadline as possible to minimize time in the bottle.

I have occasionally still had contest beers on tap to compare against the spare bottle, some beers do taste quite different.

That's the best of intentions usually. I use my shop as a dropoff location for three competitions and I do refrigerate but I can only vouch for the ones that stay here until competition day. I know comps sometimes store at room temp up until the judging day and then ice them down in coolers. I figure if you store that extra bottle in the worst possible conditions, you're at least aware of how bad it can be given poor handling.

 

[dmtaylor]

There's a national judge I know that said he can count the number of 40+ beers he's scored on on hand.

That's horrible. I know a world-class beer when I taste one. Anyone with that level of experience certainly should. This demonstrates that there are National judges out there who DON'T know a world-class beer when they taste one. That's just wrong. I believe you. That is perhaps the saddest thing of all. It's terrible.

 

[Miraculix]

This is ridiculous and indicates that you don't understand what the BJCP does or how the guidelines are developed. The guidelines describe beers that have been or are actively being brewed by commercial and home brewers for a long enough time to have them stick. The contributors to the guidelines are not just Americans. Besides, to suggest that an American is unable to do sensory evaluation on a German or Belgian beer and describe it using commonly accepted adjectives is ignorant. These are beers that have been brewed consistently for hundreds of years. I know with confidence what a German Pilsner or Belgian Dark Strong is supposed to be like within a range of parameters.

Well, I'm glad that you know, bjcp apparently does not

But let somebody more knowledgeable elaborate about, for example, their definitions of British beers.

 

[Bobby_M]

Well, I'm glad that you know, bjcp apparently does not

But let somebody more knowledgeable elaborate about, for example, their definitions of British beers.

Are you suggesting that the current definitions in the guidelines do not accurately describe the beer styles as listed?

 

[Miraculix]

Are you suggesting that the current definitions in the guidelines do not accurately describe the beer styles as listed?

Indeed I do. But I'm not going into details, sorry, no time for that. Northern brewer elaborated big time on these here in the forum, should be easy enough to find.

I read that, I read what mixed feedback people get for their beers... It's nothing real and nobody really cares about it except the participants who get scores to their liking and the guys who invested lots of time into it and now try to defend it for their own sake.

Sure, people learned something during the course and so on, but every time I read somewhere "but according to bjcp, this Amazonian beer from lampukistan should be....", I throw up a little, because who cares what a few guys think a beer should be like which originated tens of thousands kilometres away in a country they most likely never lived in or even visited.

 

[beersk]

It drives me crazy when I submit the same beer to two local competitions; each competition within 2 weeks of each other. And, in one competition my average score is 22 and comments say the beer is out of style. And in the other competition, I receive a score of 37 and get a red ribbon.

Those doing the judging all have BJCP so called certifications.

The same story repeats itself year-over-year for me; at least over the last 15 years.

Having a BJCP certification is sooooo meaningless.

I've been thinking and saying this for years. To me, it means nothing. Whenever someone says anything about a cicerone or sommolier or wanting to be one I immediately say to myself or maybe outloud, "Big deal" or "Why?" Such a freakin' pointless endeavor.

What makes me laugh the most are those pranks where people change the labels on cheap bottles of wine and give them to these so called "Sommoliers" and simply because of the label, they deem it a good wine. But as soon as they're told it wasn't what the bottle said it was, they get maaaaaaaaaaaaaaaaaaaaaad. Well, it's like, serves you right dude for being such a freakin' snob.

This is why I don't do competitions.

 

[Qhrumphf]

Indeed I do. But I'm not going into details, sorry, no time for that. Northern brewer elaborated big time on these here in the forum, should be easy enough to find.

I read that, I read what mixed feedback people get for their beers... It's nothing real and nobody really cares about it except the participants who get scores to their liking and the guys who invested lots of time into it and now try to defend it for their own sake.

Sure, people learned something during the course and so on, but every time I read somewhere "but according to bjcp, this Amazonian beer from lampukistan should be....", I throw up a little, because who cares what a few guys think a beer should be like which originated tens of thousands kilometres away in a country they most likely never lived in or even visited.

Their guidelines for European styles across the board leave much to be desired, that's for sure. The 2015 guidelines are a massive improvement in this regard. But there are still some deep flaws.

But it's more than "a few guys" and many of the big heads in the BJCP do travel internationally. Regularly.

It is fair to say that BJCP guides for euro styles more closely approximate a description of an American attempt to recreate European styles as found in on shelves in America. Ie "here's this Strong Bitter imported from England, old and stale, but I like it so I'm gonna brew it this way", and as most American examples grow from this, and entered into comps as such, guidelines get written as such leaving the impression that's how the real thing is, fresh and at home.

 

[kevin58]

The BJCP is not the end-all-be-all and their style definitions are only meant as an attempt to make sure everyone who enters and judges competitions are on the same page. So you have wasted how much time, effort, postage and in the end, frustration for what? The answer is to ignore competitions and make the beer that makes you happy.

 

[Bobby_M]

I guess the biggest take away from this is that if BJCP competitions frustrate you and you think it's a scam, ignore them completely. No one forces anyone to compete. If you have honest criticisms about the way something is done and you think it can be done better, the only acceptable answer is to get involved and make the change. Volunteer YOUR time like every other BJCP member does. I want the judge quality and quantity in my local circle of competitions to increase and there is no way to complain that into reality. I have to hold study sessions and foster education.

 

[Horseflesh]

I am not a judge, but I took a BJCP class for fun. I opted not to take the test because I don’t think I’m good enough to pass, and if I did, I still don’t think I’d be a good judge. Judging is hard and I appreciate the efforts of those who do it. My club and local community is always desperate for judges.

I have never competed, and probably never will. I pretty much know what’s wrong with my beer and know that even as I slowly improve, it will never be world-class. If I want to measure myself against others, I have beer at a club meeting. That shows me I’m better than some, not as good as others. That’s OK, and it’s all the validation I need for now.

 

[jekeane]

That's horrible. I know a world-class beer when I taste one. Anyone with that level of experience certainly should. This demonstrates that there are National judges out there who DON'T know a world-class beer when they taste one. That's just wrong. I believe you. That is perhaps the saddest thing of all. It's terrible.

I would be interested in what your definition of world class is. 40 and above is a very good score it means that you are getting down to the finer details of style to make a perfect (world class) beer.

I have probably judged a thousand or so beers. I’ve given 1 50 ever to a beer that was truly world class it was an eisbock. I have given dozens of 40’s but the bulk of homebrew falls in the 30’s.

There is nothing wrong with most beers that score 30’s many are good and I would be happy to have another. 40’s are reserved for excellent beers that are commercial in quality and nail down large sections of the guidelines.

There aren’t many homebrews or commercial beers for that matter that are world class.

 

[Qhrumphf]

I've certainly judged my share of beers as well. I've never given a 50 in the real world while judging (and can think of only three beers I'd give it to at all), but I've given quite a few in the 40s (40-42 usually). I've only given 45+ twice in comps, once to a beer that went on to win BOS of a very large comp, and once to another at NHC finals that won a medal. Both were either 45 or 46 IIRC. I've done practice sheets of classic world class examples where I've gone up to 49, but that wasn't done blind so bias has to be acknowledged.

 

[Miraculix]

I guess the biggest take away from this is that if BJCP competitions frustrate you and you think it's a scam, ignore them completely. No one forces anyone to compete. If you have honest criticisms about the way something is done and you think it can be done better, the only acceptable answer is to get involved and make the change. Volunteer YOUR time like every other BJCP member does. I want the judge quality and quantity in my local circle of competitions to increase and there is no way to complain that into reality. I have to hold study sessions and foster education.

Personally, I don't mind if people have a competition for fun or whatever and that they make up their own rules for this. I mean, that's literally how every competition works, sports, cats, hot dog eating.

The only thing that bothers me, is that the bjcp rules or guidelines or however you call it, are often used as a set in stone thing if people want to recreate European beers.
They are treated like some kind of authority who can define European beers, which is just wrong and often even heavily incorrect.

Using it for their competition game thing, no problem. Treating them like a beer authority however is bad.

 

[BrewnWKopperKat]

page v of the BJCP 2015 guidelines.

"Using the Style Guidelines

"When we created previous versions of the style guidelines, we had no idea how prevalent and pervasive they would become. We believed we were creating a standardized set of style descriptions for use in homebrew competitions, but then found they were widely adopted worldwide to describe beer in general. Many countries with emerging craft beer markets were using them as handbooks for what to brew. Consumers and trade groups began using the styles to describe their products. And, unfortunately, many made astounding leaps of logic well beyond what was our original intent, and subsequently used the guidelines as a sort of universal Rosetta Stone for beer."

 

[dmtaylor]

I would be interested in what your definition of world class is. 40 and above is a very good score it means that you are getting down to the finer details of style to make a perfect (world class) beer.

I have probably judged a thousand or so beers. I’ve given 1 50 ever to a beer that was truly world class it was an eisbock. I have given dozens of 40’s but the bulk of homebrew falls in the 30’s.

There is nothing wrong with most beers that score 30’s many are good and I would be happy to have another. 40’s are reserved for excellent beers that are commercial in quality and nail down large sections of the guidelines.

There aren’t many homebrews or commercial beers for that matter that are world class.

Well this is easy.

There are two kinds of judges out there. The first type tries to find flaws in every beer even if there are none. The second type tastes the beer and judges accordingly and isn't afraid to score a 40+. I've judged about 15 competitions. In that time I have scored beers in the 40s about a dozen times. Twice I have scored beers at like 46 and 47. And yes each time I was within 5-7 points of the other judge.

There are outstanding beers out there. I'm not going to deny an outstanding beer or make up reasons why it needs to be improved if there is in fact nothing that needs to be improved.

 

[Miraculix]

page v of the BJCP 2015 guidelines.

"Using the Style Guidelines

"When we created previous versions of the style guidelines, we had no idea how prevalent and pervasive they would become. We believed we were creating a standardized set of style descriptions for use in homebrew competitions, but then found they were widely adopted worldwide to describe beer in general. Many countries with emerging craft beer markets were using them as handbooks for what to brew. Consumers and trade groups began using the styles to describe their products. And, unfortunately, many made astounding leaps of logic well beyond what was our original intent, and subsequently used the guidelines as a sort of universal Rosetta Stone for beer."

I'm glad to see that the bjcp itself shares my point of view.

 

[dmtaylor]

I'm glad to see that the bjcp itself shares my point of view.

If by "the bjcp itself" you mean "Gordon Strong"... well I'm not him and I don't know him so I don't know what he actually thinks in real life, but I tend to question the sincerity in the statements made above. They know what they're doing. A disclaimer doesn't change truth, whatever the truth might be.

 

[jekeane]

Well this is easy.

View attachment 668397

There are two kinds of judges out there. The first type tries to find flaws in every beer even if there are none. The second type tastes the beer and judges accordingly and isn't afraid to score a 40+. I've judged about 15 competitions. In that time I have scored beers in the 40s about a dozen times. Twice I have scored beers at like 46 and 47. And yes each time I was within 5-7 points of the other judge.

There are outstanding beers out there. I'm not going to deny an outstanding beer or make up reasons why it needs to be improved if there is in fact nothing that needs to be improved.

Im not trying to be snide or sarcastic here, Im just trying to understand your point of view. This reads to me like you are saying you are the right and better judge which is a very subjective statement. 5-7 points is a very large margin of difference 5 should be the max difference and 3 is generally the ideal I see followed.

Saison Dupont, Pilsner Urquell, Coors Original are "world class" beers in their respective styles. I rarely get those types of beer in a home brew or pro brewer competition for that matter. It seems to me you have a harder time just letting a beer be very good or excellent of which there are many.

Literally the whole point of competitions is to acknowledge success and look for flaws without fabricating either. You seem to feel that judges in general fabricate flaws, I might suggest you a fabricating success. There is room for subjectiveness but there are rules to be followed. Would be a competition without them.

Maybe people choice is the way to go for everything. Then we can drown in hazy boys and pastry beers! (ok that part was a little snide)

 

[dmtaylor]

Im not trying to be snide or sarcastic here, Im just trying to understand your point of view. This reads to me like you are saying you are the right and better judge which is a very subjective statement. 5-7 points is a very large margin of difference 5 should be the max difference and 3 is generally the ideal I see followed.

Saison Dupont, Pilsner Urquell, Coors Original are "world class" beers in their respective styles. I rarely get those types of beer in a home brew or pro brewer competition for that matter. It seems to me you have a harder time just letting a beer be very good or excellent of which there are many.

Literally the whole point of competitions is to acknowledge success and look for flaws without fabricating either. You seem to feel that judges in general fabricate flaws, I might suggest you a fabricating success. There is room for subjectiveness but there are rules to be followed. Would be a competition without them.

Maybe people choice is the way to go for everything. Then we can drown in hazy boys and pastry beers! (ok that part was a little snide)

If we're getting picky about the 5-7 thing, I'll confess I simply follow whatever range the head judge tells me to. If that's 3 points, fine. 5 points, fine. We can make it work. With a good judging partner (I always pray for that), I'm almost always within 3-5 points anyway and we don't even need to adjust our scores.

I'm not afraid to use the entire range from 13 or 17 or again whatever the head judge decides is minimum all the way up to apparently 47. I think I agree with you that most homebrews fall between roughly 27 to 37, vast majority.

Yes, I KNOW many judges seek flaws where there are none. I have no question of it. That's my biggest beef. And as many others have already mentioned, probably all judges, ourselves included, do insert some level of subjectivity as well which accounts for more variability. I hate to say it is wrong to do so, but on some level it is and some it isn't. This is a real struggle for the judge who's so sure of himself/herself that he/she knows better than what the guidelines spell out for them. Some judges don't even look at the guidelines. Personally I write down what *I* taste first, THEN I review the guidelines for anything that I got wrong or missed. I think that's a good way to go about it, never got any flack for it anyway besides maybe that it takes a longer time to do the job right and comps are usually cracking the whip at us.

I don't know where we're going with all this except that I would like to indicate that there are facets of all our stories that are right, and when mosaicked all together should form a picture of what the BJCP should look like in an ideal universe. Except that we don't live in an ideal universe. We live in the real one. At least, as far as I know.

 

[kingmatt]

I know there are judges out there that feel EVERY beer can be improved on in some way, and so they will never give a 50 and always provide at least one suggestion on how they feel the beer can be improved when judging. Is that looking for flaws? I'm not sure.

I don't necessarily agree with that POV but I also have never scored a 50 beer yet.

 

[S-Met]

There are just too many variables that you can't control and don't know about to blame the judges or the BJCP certification.
Did the judges go to a beer blast the night before and were hung over?
Did the judges sample 50 beers before yours?
Where was your beer in the flight?
What was the serving temperature?
Was a really exceptional version of the style served right before yours?
Having said all that, I agree that there are problems with beer competitions but I disagree that BJCP certifications are completely meaningless. At least the BJCP judges have met SOME minimal standard for qualification.
I don't think you'll ever be able to get personal likes/dislikes out of beer judging.
Your beer might not have any flaws, but the "overall impression" is something that is highly subjective and thus consistent scoring really can't be expected.
I believe the best you can get out of competitions is to take each judge's comments and scores individually and learn what you can from that.

+1
Also not mentioned, unless I missed it...
A bad beer, be it flawed or just over-the-top in some way in a lineup can wreck your palate. Your tastebuds may not be able to fully recover buly the time you get to the next beer, or even the next round.

I am not a judge. But I am an experienced taster and realize how to cleanse my palate between samples, but its more like rinsing a dirty dish with tap water than really cleaning your palate.

Just for perspective, I was a judge for a friendly neighborhood wine tasting, 20 wines, 5/hr. I recall being somewhere around sample 10 and thinking, "this might be a really great wine, or maybe its sh!tty. bless, I can't tell, I'm pretty sure I can still taste that nasty wine #3 from 90 minutes ago."

 

[Hwk-I-St8]

Yes, I KNOW many judges seek flaws where there are none. I have no question of it

I see this type of behavior in many settings. Food critics, it seems, always need to find some flaw. They're a "critic" so finding something to be critical about seems to be a requirement. For some it's a badge indicating how discerning they are.

I also see this in people who are paid to review others work. Where I work, we have a team that does nothing but review documents, software, etc. There are a number of them who always find something. Always. I've watched them review a document and approve it after revisions. Then they review it again for another update and find some fault in the portion of the document they approved previously. They just need to find something.

I'm confident there are beer judges that are this way.

I used to score all the beers I've had in untappd (rarely do anymore). I never scored a 5 because it left no room for a beer to be better. I would, I think, have the same issue giving out a 50.

 

[dmtaylor]

I used to score all the beers I've had in untappd (rarely do anymore). I never scored a 5 because it left no room for a beer to be better. I would, I think, have the same issue giving out a 50.

This comment is not aimed at you, Hwk-I, but rather is just a continuation of the discussion from my own perspective, as I think it is good discussion (and hope that others think so too)...

I have thousands of beers scored in Untappd. I have given 5's to dozens of them. On such a small scale, well why not. I wish the world was full of 5's. It's not, but if a beer is really world-class with zero flaws, then that's how I score it. It deserves the recognition. Same can be said for 1's and 2's as well of course, dozens of those too unfortunately. I figure, if we're going to give a range of 1 to 5, or 13 to 50, or whatever, then the full range is intended to be used, not just the middle. Not every beer is just a 3 or a 30 plus or minus a fraction of a point. I am all about normalized distribution of data. I like math, too, so I guess to me it just makes good sense to use the entire range as it is intended, not just start in the middle and work up or down by tiny amounts. By spreading it out I feel I have a better sense of truly how great or how horrible a beer really is, relative to all others. I think a lot of judges aim for the center then add or deduct points from there. But I don't really agree with that method. Starting in the middle might be okay, IF you also challenge yourself to consider whether your 30 is really a 35 or a 25. Or is your 3 really a 3.5 or a 2.5. Or whatever. Use the whole range. Everything is not mediocre. Many beers are mediocre, yes. But sometimes there is an aspect or two that stands out for one reason or another, good or bad. Don't be afraid to spread the scores out a bit so they better characterize what you really taste, rewards or dings for good and bad.

I know I'm just talking in circles now so I might just duck out for a while. Cheers all.

 

[Hwk-I-St8]

This comment is not aimed at you, Hwk-I, but rather is just a continuation of the discussion from my own perspective, as I think it is good discussion (and hope that others think so too)...

I have thousands of beers scored in Untappd. I have given 5's to dozens of them. On such a small scale, well why not. I wish the world was full of 5's. It's not, but if a beer is really world-class with zero flaws, then that's how I score it. It deserves the recognition. Same can be said for 1's and 2's as well of course, dozens of those too unfortunately. I figure, if we're going to give a range of 1 to 5, or 13 to 50, or whatever, then the full range is intended to be used, not just the middle. Not every beer is just a 3 or a 30 plus or minus a fraction of a point. I am all about normalized distribution of data. I like math, too, so I guess to me it just makes good sense to use the entire range as it is intended, not just start in the middle and work up or down by tiny amounts. By spreading it out I feel I have a better sense of truly how great or how horrible a beer really is, relative to all others. I think a lot of judges aim for the center then add or deduct points from there. But I don't really agree with that method. Starting in the middle might be okay, IF you also challenge yourself to consider whether your 30 is really a 35 or a 25. Or is your 3 really a 3.5 or a 2.5. Or whatever. Use the whole range. Everything is not mediocre. Many beers are mediocre, yes. But sometimes there is an aspect or two that stands out for one reason or another, good or bad. Don't be afraid to spread the scores out a bit so they better characterize what you really taste, rewards or dings for good and bad.

I know I'm just talking in circles now so I might just duck out for a while. Cheers all.

Regarding distrubution of scores...I'd normally expect a bell curve and I would guess mine probably fall into that...maybe skewed a bit high because I don't drink many crappy beers. I use untappd a lot when choosing beers and also when considering a visit to a brewery. I also try to score based on style, not just my preference and most commercial beers aren't awful for their style.

If I were scoring a beer for a comp (never done that) I'd have no issue giving high or low scores, but as an aggregation of sub-scores for different aspects (I believe that's how it works: appearance, aroma, flavor, carbonation, etc.), it's pretty hard to get a perfect score across the board.

 

[Deadalus]

...

View attachment 668397

...

I am quoting you solely for the attachment.

The scoring system is part of the problem. The system uses numbers to indicate qualitative categories. Entrants may not be aware of this and so think getting above mid-range scores (30-40) is a poor showing, whereas the judges are thinking this is Very Good. 20-29 is still considered Good by the judges. To be specific, the judges are translating numbers to an ordinal scale. I could be wrong, but I don't think there is a well defined definition of what changes in value in the sub-categories of the scores means. For instance, we know that 4 is one more than 3 and 5 is one more than 4. But within a sub-category on the score sheet, when the judge changes from 3 to 4 or 4 to 5 that change probably isn't a value of 1. That's because it's really an ordinal system, it's ordered by quality not quantity. This is why there is variability between the judges. On top of which, as a previous poster alludes, is the idea that the scores follow a normal distribution. It's more than likely not, and I think there is strong anecdotal evidence to say the scores don't follow a normal distribution (bell curve). The scores are very likely not symmetric. Scores are skewed with a lot more scores on the lower end versus the rest of the distribution. There aren't as many world class beers as problematic ones.

As far as competitions, I've entered a few. They've always had 3 judges. I look for consistency. If two judges are close and one off, I lean towards the pair as far as what I think my score was. Unless the of course the lone judge rates it higher, then of course that's my real score.

(I did actually have a score sheet added incorrectly once but it was a very local event and the judge may not have been certified. They forgot to carry the one and my score was 10 points off. So out of maybe out of 100 cases the 101th case had a math problem.)

 

[Spartan1979]

I'm a BJCP National judge and I rarely enter competitions, just a couple of local ones to support the local clubs and occasionally NHC (but I finally won a medal last year, so I'm probably done with that). I decided many years ago that entering a competition is a crapshoot. It depends which judges you get on what day. I do think most judges genuinely try to do a good job.

 

[Bobby_M]

Another way to look at this is that the organic human nature of beer judge palates could be completely overridden by using mass spectrometers and other gear to compare a homebrew with the top commercial example of the style and rate you exactly on how close you got. That would be incredibly FAIR and you couldn't complain about variation anymore. It would also be expensive and boring.

I don't know if judges reach to find flaws. The variation is mostly due to the various concentration thresholds people have. I have had beers that both have Diacetyl and Acetaldehyde at levels that were not even questionable to me but my partner didn't pick up either. If we just left the sheets like that, the entrant would just accuse me of fabrication or bias. What do you do there? In my opinion, generally I'd want to defer to the higher rank. It's not perfect but it's better than flipping a coin.

 

[Qhrumphf]

I know there are judges out there that feel EVERY beer can be improved on in some way, and so they will never give a 50 and always provide at least one suggestion on how they feel the beer can be improved when judging. Is that looking for flaws? I'm not sure.

I don't necessarily agree with that POV but I also have never scored a 50 beer yet.

This is me. I believe in using the entire range. I have given everything from 13 to 46 in competition. 13 is about as rare as 45-46. I think I've given three 13s and two 45-46. I've also probably evaluated 1000 or so beers in comps. But 18-42 gets used regularly. I'll probably hit both extremes once a competition.

If I can offer you a *single* concrete suggestion to improve it, it's not a 45+ beer. Any improvement beyond that is some intangible magic I couldn't identify. A 50 pointer should be a *life changing* beer. My only 3 I have had, of every beer I've ever had in comps or otherwise, are (fresh and undamaged) Weihenstephaner Hefe, Saison DuPont, and Cantillon Fou'Foune. Plenty of world class beers a couple points lower than that. Admittedly at that point it's purely intangible and purely subjective.

 

[dmtaylor]

generally I'd want to defer to the higher rank.

I totally disagree with this. It's a slice of the problem with BJCP. Lower ranks get less respect, even if the lower ranked guy is a supertaster or even if the Master just had liver & onions or who knows what else for lunch. Any judge of any rank can add a lot of value. I respect all equally (...or equally poorly maybe!).

 

[Qhrumphf]

Another way to look at this is that the organic human nature of beer judge palates could be completely overridden by using mass spectrometers and other gear to compare a homebrew with the top commercial example of the style and rate you exactly on how close you got. That would be incredibly FAIR and you couldn't complain about variation anymore. It would also be expensive and boring.

I don't know if judges reach to find flaws. The variation is mostly due to the various concentration thresholds people have. I have had beers that both have Diacetyl and Acetaldehyde at levels that were not even questionable to me but my partner didn't pick up either. If we just left the sheets like that, the entrant would just accuse me of fabrication or bias. What do you do there? In my opinion, generally I'd want to defer to the higher rank. It's not perfect but it's better than flipping a coin.

When high ranking judges proctor exams, they're not permitted to alter sheets or scores in the consensus process, precisely for that reason. When there are significant variations, flaw sensitivity is definitely at play. Its built into the exam grading structure deliberately because it is a VERY real and very present phenomenon. Graded sets where a proctor was clearly diacetyl blind.

A couple years ago I proctored an exam and the other proctor and I were bang on agreement with everything except one where we varied by something like 15 pts. It was a Wit where I'm prone to detecting the ham/hot dog character if cheap coriander (or too much) is used. He thought it was delightful.

I wouldn't necessarily defer to higher ranks with flaw sensitivity. Rather it's knowing what you're abnormally sensitive to and when to give a pass, vs what you're lower in sensitivity. If you're mostly diacetyl blind but your judging partner isn't, even if you outrank em, defer. If you know you're good with diacetyl and you detect it, stick by your guns (though if you know you're extremely sensitive and it's faint, perhaps give the examinee the benefit of the doubt).

For example, I'm ridiculously sensitive to roast-derived pyrazines. Almost ALL coffee beers taste like green pepper or jalapeno to me. Even some darker beers without coffee can give me that character. I'll often note it on a scoresheet, note that I'm super sensitive as well, and not knock any points for it unless it's extreme enough to be present to others.

 

[Qhrumphf]

I totally disagree with this. It's a slice of the problem with BJCP. Lower ranks get less respect, even if the lower ranked guy is a supertaster or even if the Master just had liver & onions or who knows what else for lunch. Any judge of any rank can add a lot of value. I respect all equally (...or equally poorly maybe!).

I've seen plenty a low ranking judge paying as much attention to my sheet as to their own.

Yes, I've worked to train my palate. That hasn't made me any more or less sensitive to anything, just better at identifying what it is.

The worst individual BJCP judges I've seen are National ranked (though on average National judges are solid and quality does align with rank). Folks who passed the legacy exam in the 90s and haven't bothered to grow. Stagnant palate. Poor biased style knowledge. And lazy arrogant sheets.

I honestly think Provisional judges are phenomenal for the most part, since they're often in the midst of studying and preparing for the exam and give it their all. Good sheets. Lots of freshly studied style knowledge. Often off-flavor training fresh as well.

 

[Bobby_M]

I totally disagree with this. It's a slice of the problem with BJCP. Lower ranks get less respect, even if the lower ranked guy is a supertaster or even if the Master just had liver & onions or who knows what else for lunch. Any judge of any rank can add a lot of value. I respect all equally (...or equally poorly maybe!).

Ok, but the rank is the only measuring stick that is somewhat objective. Of course by comparing rankings you don't know everything but the higher rank has judged more, has scored higher on the exam which shows some level of parity with other high ranking judges. Lower level judges have less experience and have not scored beers similarly with other high scoring judges.

If you totally disagree, what measuring stick would you prefer to use when two judges are far apart on a beer? I'm not talking about where one guy acknowledges they have a super high diacetyl threshold and voluntarily defers. I'm talking about two judges who stand their ground but are still 10 points apart? One is recognized and the other is national. You "totally disagree" that the final score should be more influenced by the higher rank judge? If you were the judge coordinator, tell us how you fix that problem on the spot.

If a lower ranked judge really is a superstar, they could easily move up the ranks without breaking a sweat so they should.

 

[dmtaylor]

what measuring stick would you prefer to use when two judges are far apart on a beer? I'm not talking about where one guy acknowledges they have a super high diacetyl threshold and voluntarily defers. I'm talking about two judges who stand their ground but are still 10 points apart? One is recognized and the other is national. You "totally disagree" that the final score should be more influenced by the higher rank judge? If you were the judge coordinator, tell us how you fix that problem on the spot.

If a lower ranked judge really is a superstar, they could easily move up the ranks without breaking a sweat so they should.

If two judges cannot come to agreement, get a couple more judges to taste the same beer and then gang up against the one who either missed something or detected a flaw that doesn't actually exist. This can be humbling for any judge of any rank. Some fellow judges and I (all Recognized or Certified) had to do this once against a National who lost the argument, needless to say he was not happy and actually stormed out, but the rest of us shrugged and I remember saying something to the effect of, truth hurts sometimes. In this particular case the National actually admitted to us that he had a known insensitivity to diacetyl which was the problem. At least he had the decency to admit that he might be wrong, even though he obviously hated us for having to point it out. Typically when another judge points out to me "hey this is oxidized" or "don't you taste DMS?" then after I taste again, I'll either agree or I'll hold my ground and say no I just don't taste it and refuse to budge. It happens to all of us at one time or another. All you can really do is bring in another judge or two or three to figure out who's right & who's wrong. And if it's me, well okay, I've learned something about my own palate. I know I'm not super sensitive to DMS, but I can usually detect it after someone else points it out to me. Diacetyl and oxidation on the other hand I often pick out before others do. Stuff we all learn as we go along on the journey. But rank has absolutely NOTHING to do with our relative ability to pick up flavors, other than *maybe* the higher ranked guy has had more of these experiences so they know when to back off and not get overly argumentative, just suck it up and admit they aren't sensitive to this or that... but still should only record on the scoresheet what they themselves are able to taste, except maybe to note with an asterisk: "the other guys said DMS, but personally I don't get it".

If a lower ranked judge really is a superstar, but doesn't give a **** about rank, he/she won't go up in rank. I have enough points to be National. But I don't care. They don't have exams for National in my area, I'd have to drive 100 miles like I did last time to get to Certified. I'm content to stay at Certified at this moment in time. Does this mean I'm not as skilled a taster as a National? I dunno. Get a scoresheet from me, then you can be the judge of the judge. I think if I took the exam I would likely pass. Meh. Maybe someday. Maybe not.

 

[mwilcox]

I am quoting you solely for the attachment.

The scoring system is part of the problem. The system uses numbers to indicate qualitative categories. Entrants may not be aware of this and so think getting above mid-range scores (30-40) is a poor showing, whereas the judges are thinking this is Very Good. 20-29 is still considered Good by the judges. To be specific, the judges are translating numbers to an ordinal scale. I could be wrong, but I don't think there is a well defined definition of what changes in value in the sub-categories of the scores means. For instance, we know that 4 is one more than 3 and 5 is one more than 4. But within a sub-category on the score sheet, when the judge changes from 3 to 4 or 4 to 5 that change probably isn't a value of 1. That's because it's really an ordinal system, it's ordered by quality not quantity. This is why there is variability between the judges. On top of which, as a previous poster alludes, is the idea that the scores follow a normal distribution. It's more than likely not, and I think there is strong anecdotal evidence to say the scores don't follow a normal distribution (bell curve). The scores are very likely not symmetric. Scores are skewed with a lot more scores on the lower end versus the rest of the distribution. There aren't as many world class beers as problematic ones.

As far as competitions, I've entered a few. They've always had 3 judges. I look for consistency. If two judges are close and one off, I lean towards the pair as far as what I think my score was. Unless the of course the lone judge rates it higher, then of course that's my real score.

(I did actually have a score sheet added incorrectly once but it was a very local event and the judge may not have been certified. They forgot to carry the one and my score was 10 points off. So out of maybe out of 100 cases the 101th case had a math problem.)

for what it's worth, I don't worry about the "fair/good" labels as I don't think the words meet the description used to the right of the score ranges. In my opinion 0-13 is basically Awful, 14-19 is Problematic, 20-25 are generally Fair, 26-32 tend to be Good/Solid, 33-38 Very good. Or thereabouts; the exact number doesn't matter. Beer is on a continuum, not in little boxes within a scale.
Source: Grandmaster and Guidelines Reviewer/Editor who gives plenty of 40's where they are deserved.

 

[dmtaylor]

for what it's worth, I don't worry about the "fair/good" labels as I don't think the words meet the description used to the right of the score ranges. In my opinion 0-13 is basically Awful, 14-19 is Problematic, 20-25 are generally Fair, 26-32 tend to be Good/Solid, 33-38 Very good. Or thereabouts; the exact number doesn't matter. Beer is on a continuum, not in little boxes within a scale.
Source: Grandmaster and Guidelines Reviewer/Editor who gives plenty of 40's where they are deserved.

Prost to you, Michael.

Personally I would define 0-22 as Awful, 23-27 as Less Than Good, 28-32 as Pretty Good, 33-38 Great, 39-42 Wow, and 43+ World-Class Awesomeness.

However, quantitative averages are still better than subjective qualitative bins like these. The aboves are just approximations and very subjective.

 

[Qhrumphf]

Given a courtesy score of 13, I go by: 13-16 unpalatable/nauseating. 17-20 seriously flawed and unpleasant though not entirely undrinkable. 21-25 generally not great but some good parts. 26-29 decent but flawed. 30-35 solid but not perfect, flaws are pretty minor. 36-39 excellent and without flaws but lacks bits depth/complexity/balance, or excellent with a very slight flaw. 40-44 flawless and bang on style, excellent beer though missing the magic. 45+ unbelievable in every regard, couldn't improve it if I tried.

 

[jswillbrewforbeer]

I've learned to pay more attention to the comments as opposed to the actual score. Scoring is almost subjective because there is no manual which tells you "subtract 2 points for having medium low diacetyl instead of low in the aroma of a Czech Premium Pale Lager." The score is really more of a range anyway.

 

[Hwk-I-St8]

Prost to you, Michael.

Personally I would define 0-22 as Awful, 23-27 as Less Than Good, 28-32 as Pretty Good, 33-38 Great, 39-42 Wow, and 43+ World-Class Awesomeness.

However, quantitative averages are still better than subjective qualitative bins like these. The aboves are just approximations and very subjective.

I struggle with descriptors like awful, less than good etc. only because a person might get low scores solely due to something not being to style. It may be a very good beer, but get a low score due to style variances, right?

 

[Deadalus]

In social science surveys (think of where you are asked disagree strongly, disagree, neutral, agree, agree strongly) one has to be very careful using numbers for the categories. The categories aren't necessarily evenly spaced and without strict design of the responses should only be treated as categorical. You can't average neutral, neutral, and agree for instance.

Now consider a beer competition. The beers get a score which is averaged by several judges. The numbers are being derived from what is probably more of a categorical grading system. I suspect this influences some of the variability in scores.