Homebrew Competition Scoring

[Pancoastbrewing]

So I've just gotten the results back from the second homebrew competition I've entered a beer in, and I'm somewhat miffed...

This was a local competition with 162 entires. I brewed a Vienna lager and there were 23 lagers entered. I was super happy with this beer. The least flawed one I've ever made, imo.

This competition stated that beers would be judged "loosely on the 2008 bjcp standards", so I got the feeling scoring wouldn't be super harsh. It's not a NHBC qualifier, just a fun competition. I received a score of 46 from one judge and 43 from another, for an overall score of 44.5...obviously I'm beyond thrilled with those numbers. But incredibly, I didn't place in the top 3 lagers. From what I understand, many judges rarely give out 40s, so it's hard for me to believe a 44.5 didn't even place! I'm not upset at all, I understand that you get far more out of the score sheet feedback than actual numbers. I was just stunned that I didn't place with that score.

Is this something common? The other competition in which I entered I received a 32 (still considered "very good") with a flawed beer. Are some competitions more about building a brewer's confidence with inflated scores? As good as my 44.5 beer was, it was hardly what I would consider "world class"! I'm happy either way, just a bit curious.

 

[beernutz]

What were the qualifications of the judges who rated your beer? That information should be on the score sheet.

 

[Pancoastbrewing]

What were the qualifications of the judges who rated your beer? That information should be on the score sheet.

One was ranked an "experienced judge" and a bjcp apprentice. The other was also ranked an "experienced judge" but did not put down a bjcp rank or status.

 

[dmtaylor]

One was ranked an "experienced judge" and a bjcp apprentice. The other was also ranked an "experienced judge" but did not put down a bjcp rank or status.

I was going to ask the very same question. And sure enough..... These judges really had no idea what they were doing, so it's anybody's guess what the scores would have been in a real sanctioned competition. BJCP judge formal ranks actually start at Recognized, and all your judges were below that rank.

 

[Hoppy2bmerry]

They were generous enough with scoring that your very very good beer was out ranked by three or more other excellent beers? That seems to be the only explanation.

 

[Pancoastbrewing]

I was going to ask the very same question. And sure enough..... These judges really had no idea what they were doing, so it's anybody's guess what the scores would have been in a real sanctioned competition. BJCP judge formal ranks actually start at Recognized, and all your judges were below that rank.

Yeah, that's what I figured. I don't live in a big metro area and the competitions around here always struggle to find enough judges. I guess it was a "just for fun" event and I shouldn't expect too much. But it would be nice to get feedback from some legit judges. Otherwise, I might as well give one of my beers to my dad and get his opinion!

 

[dmtaylor]

Even real comps are a crapshoot. I have always recommended to anyone wanting good feedback on their beer, to enter at least 3 or 4 competitions with each beer, knowing that approximately 60% of the feedback you will receive is garbage. So throw out all the outliers and forget they ever existed, then take the average of the sheets with the good feedback, and that is your true score. One comp alone is almost never enough to get good feedback unless you get super lucky.

Also recognize that a National or Master judge is not necessarily always better or "more correct" than a Recognized or Certified rank judge, as any judge can have good days or bad days or whatever, and all have slightly different experience bases, preferences, etc. All of these ranks also have qualified opinions, but it's still somewhat of a crapshoot as described previously, even though it probably shouldn't be. But below these ranks, opinions have zero qualifications so the crapshootiness is much worse.

My qualifications: I am BJCP Certified #A0511, been a judge since 2007(?) and I have entered probably ~50 competitions since way before that even.

 

[decoleur]

Some comps award medals based purely on points but most I have been in will do a miny BOS for the category to determine the medals, if there isn't a clear winner. So you can easily have a lower scoring beer get the gold because it suits the judges pallet more.

I just had one of mine beat a friends much better product because it struck the judges fancy. at the end of the day it is kind of arbitrary but you shouldn't sneeze at those scores.

Now do it again with a new brew of the same recipe at a different comp and I will say you need to send me a bottle with the recipe attached.

 

[Homercidal]

What were the scores of the beers that placed?

 

[Pancoastbrewing]

Even real comps are a crapshoot. I have always recommended to anyone wanting good feedback on their beer, to enter at least 3 or 4 competitions with each beer, knowing that approximately 60% of the feedback you will receive is garbage. So throw out all the outliers and forget they ever existed, then take the average of the sheets with the good feedback, and that is your true score. One comp alone is almost never enough to get good feedback unless you get super lucky.

Also recognize that a National or Master judge is not necessarily always better or "more correct" than a Recognized or Certified rank judge, as any judge can have good days or bad days or whatever, and all have slightly different experience bases, preferences, etc. All of these ranks also have qualified opinions, but it's still somewhat of a crapshoot as described previously, even though it probably shouldn't be. But below these ranks, opinions have zero qualifications so the crapshootiness is much worse.

My qualifications: I am BJCP Certified #A0511, been a judge since 2007(?) and I have entered probably ~50 competitions since way before that even.

Good info. Unfortunately for my area, there may only be 3 competitions a year MAX, unless of course I mail them out to regional comps. To be honest, it's a bit disappointing that both competitions I entered had nearly 200 entries and neither seemed to have ranked judges. At least not for the lager category. The comments I received on this last one, I'm pretty sure I could have described an entry in more descriptively correct terms. There seems to be a big shortage of BJCP certified judges. Oh well, it's still all for fun at the end of the day and I'm still a fairly novice brewer with only a dozen or so finished beers to my credit. I'll still enter the next time competitions come around just for the heck of it.

 

[Pancoastbrewing]

What were the scores of the beers that placed?

I wasn't able to attend the awards ceremony, and I'm not sure if they announced them there either, but the winning scores were never posted on the competition website.

 

[Pancoastbrewing]

Some comps award medals based purely on points but most I have been in will do a miny BOS for the category to determine the medals, if there isn't a clear winner. So you can easily have a lower scoring beer get the gold because it suits the judges pallet more.

I just had one of mine beat a friends much better product because it struck the judges fancy. at the end of the day it is kind of arbitrary but you shouldn't sneeze at those scores.

Now do it again with a new brew of the same recipe at a different comp and I will say you need to send me a bottle with the recipe attached.

Ha, it's interesting...I went to the LHBS intending to brew a marzen with totally different ingredients. But my LHBS is incredibly hit or miss with availability to the point that my entire grain bill and hop schedule had to be changed. They were even out of 34/70 dry yeast. But this was the cleanest lager I've ever done and it was the first time that I used the saflager s-189 swiss dry yeast. I always seem to have trouble cooling down my wort pre-pitching and other dry yeasts have given me some off flavors due to the higher starting ferm temp. The s-189 was super clean though. Will definitely use it again.

 

[myelo]

There have been a number of studies showing wine competitions are basically a crapshoot. One study looked at wines submitted to multiple competitions and found that when a wine scored a gold medal at one competition, the likelihood it would score a gold medal at another competition was no different than chance. It's random.

The same applies to individual judges evaluating the same wine from the same bottle at different times. The intraobserver variability is high -- he or she will score the same wine high one time and low the other. Realize these are not random wine drinkers, they are experienced wine judges.

I've not seen similar studies of beer judges but would be shocked if the they are any better. So if you want to submit beer to competitions because it's fun, go for it. But if you think the results are scientifically meaningful ... not so much.

 

[Qhrumphf]

Even real comps are a crapshoot. I have always recommended to anyone wanting good feedback on their beer, to enter at least 3 or 4 competitions with each beer, knowing that approximately 60% of the feedback you will receive is garbage. So throw out all the outliers and forget they ever existed, then take the average of the sheets with the good feedback, and that is your true score. One comp alone is almost never enough to get good feedback unless you get super lucky.

Also recognize that a National or Master judge is not necessarily always better or "more correct" than a Recognized or Certified rank judge, as any judge can have good days or bad days or whatever, and all have slightly different experience bases, preferences, etc. All of these ranks also have qualified opinions, but it's still somewhat of a crapshoot as described previously, even though it probably shouldn't be. But below these ranks, opinions have zero qualifications so the crapshootiness is much worse.

My qualifications: I am BJCP Certified #A0511, been a judge since 2007(?) and I have entered probably ~50 competitions since way before that even.

This is excellent advice. Almost assuredly your score you were given was inflated. Enter the beer in multiple comps (especially any with larger and more qualified judge pools) and see how it fares. A 44+ point beer should not only place in a small competition, it should place in the NHC first round, and possibly even place in NHC Finals.

Two possible scenarios, I think.

1) Your judge pair judged the entire flight, overscored everything, and three other brewers may be misled into thinking they brewing nearly 50 point beer.

2) Your judge pair rated all their beers highly, but your entry got beaten by other lower scoring entries in mini-BOS (by more qualified judges who more fairly scored the other flights)

And yes, always take judge opinions with a grain of salt. Even National, Master, and Grand Master judges (and I say that as a National judge on the road to hitting Master). While higher ranked judges will usually be more reliable in terms of accurate sensory feedback, appropriate scoring, and actual style knowledge, as said above judges can have bad days. Or even if nothing else, judging is still done blind, judges have no info about what you did and can only make educated guesses, and there could always be damage in shipment, or even flat out competition error. It's not unheard of for a competition to inadvertently swap beers, or to enter data incorrectly.

So like he says, enter widely, and filter out the noise.

 

[Pancoastbrewing]

This is excellent advice. Almost assuredly your score you were given was inflated. Enter the beer in multiple comps (especially any with larger and more qualified judge pools) and see how it fares. A 44+ point beer should not only place in a small competition, it should place in the NHC first round, and possibly even place in NHC Finals.

Two possible scenarios, I think.

1) Your judge pair judged the entire flight, overscored everything, and three other brewers may be misled into thinking they brewing nearly 50 point beer.

2) Your judge pair rated all their beers highly, but your entry got beaten by other lower scoring entries in mini-BOS (by more qualified judges who more fairly scored the other flights)

And yes, always take judge opinions with a grain of salt. Even National, Master, and Grand Master judges (and I say that as a National judge on the road to hitting Master). While higher ranked judges will usually be more reliable in terms of accurate sensory feedback, appropriate scoring, and actual style knowledge, as said above judges can have bad days. Or even if nothing else, judging is still done blind, judges have no info about what you did and can only make educated guesses, and there could always be damage in shipment, or even flat out competition error. It's not unheard of for a competition to inadvertently swap beers, or to enter data incorrectly.

So like he says, enter widely, and filter out the noise.

All great advice. It might be a better idea to just join to local home brew club and have other brewers test out your stuff. Although they may not be honest face to face!

 

[IkeKrizzule]

There have been a number of studies showing wine competitions are basically a crapshoot. One study looked at wines submitted to multiple competitions and found that when a wine scored a gold medal at one competition, the likelihood it would score a gold medal at another competition was no different than chance. It's random.



The same applies to individual judges evaluating the same wine from the same bottle at different times. The intraobserver variability is high -- he or she will score the same wine high one time and low the other. Realize these are not random wine drinkers, they are experienced wine judges.



I've not seen similar studies of beer judges but would be shocked if the they are any better. So if you want to submit beer to competitions because it's fun, go for it. But if you think the results are scientifically meaningful ... not so much.

I know nothing of wine judging, but I would very much argue that results from a legitimate Bjcp contest with well trained judges is far different than "random chance" or a "crapshoot".

The contest the OP entered was not this type of competition and should not be treated as such. But at legitimate contests run by well trained judges I am a firm believer that talent and quality win out. Maybe not every time, judges are human and there are many factors that can influence judging, but not to anywhere near what you describe.

 

[Qhrumphf]

I know nothing of wine judging, but I would very much argue that results from a legitimate Bjcp contest with well trained judges is far different than "random chance" or a "crapshoot".

The contest the OP entered was not this type of competition and should not be treated as such. But at legitimate contests run by well trained judges I am a firm believer that talent and quality win out. Maybe not every time, judges are human and there are many factors that can influence judging, but not to anywhere near what you describe.

Clearly you've never entered the same beer in multiple competitions. Routinely things swing widely. I've had the same beer from the same batch judged the same day in two different comps. One will score high 30s to low 40s and win me a gold. The other will score low 20s or even teens. And these are both by qualified judges.

The rampant bias and variability of wine judging had been pretty well documented, and I too would like to think/hope that beer judging is more reliable, but I'd still say the word "crapshoot" is a valid descriptor.

 

[dmtaylor]

I've had the same beer entered in two competitions, with the same judge!, and he judged it like 20 points differently depending on who he was judging with. Influence, bias, crapshoot, call it whatever you like, but it's very real. I repeat: The only way to know how your beer truly scores is to enter 3-4 comps, throw out the garbage, and average the rest.

I'm not saying it *should* be like this, because obviously in a perfect world, certified judges should know what they are talking about. The unfortunate reality is, we're all human and all biased and easily influenced by dozens of factors. More:

I know that when I judge or taste a beer multiple times over the course of days or weeks, my perceptions can change VERY significantly between tastings. And I'm a friggin perfectionist and brutally honest at all times, perhaps even to a fault. But such is life. None of us is perfect. Not even a Master judge or JZ or GS or anybody. Nobody. We are not robots.

But anyway. Sorry for the tangential ranty thing.

 

[myelo]

I know nothing of wine judging, but I would very much argue that results from a legitimate Bjcp contest with well trained judges is far different than "random chance" or a "crapshoot".

Wine experts in general, and wine judges in particular made the same argument ... until the data showed otherwise. Of course it could be much different in beer judging, but I would be interested in seeing some actual data.

I should point out that the wine studies were judging wines from professional, commercial vintners. So it's not a matter of telling the difference between spoiled grape juice and a well made wine. I have no doubt that a beer judge can consistently pick out poorly made homebrew with substantial flaws.

 

[dmtaylor]

I have no doubt that a beer judge can consistently pick out poorly made homebrew with substantial flaws.

I do have doubt. When judges (or non-judges alike) are BLINDLY presented with a highly acclaimed beer, they'll often describe all sorts of off-flavors that really aren't there, and I'm not certain why -- maybe to show off their knowledge prowess or whatever. And then if you show them the label, they'll immediately flip-flop and tell you it's the best beer on earth, even if off-flavors are purposely added by the experimenter to see if they can detect them. I can't provide a specific reference but I know such experiments have been run many times. It seems there's nothing any human can really do to avoid this odd behavior, which might be a form of confirmation bias or some such term. The objective truth lies someplace in between a lot of the time. But again.... we digress.