ABV Bias in Beer Ratings

[VikeMan]

Mods: Please move this if it doesn't fit in Brew Science.

Back in January 2020, I analyzed a bunch of beer rating data painstakingly copied and pasted from Beer Advocate, hoping to confirm or disprove something I thought I "knew" already, i.e. that there is a bias in beer ratings associated with (or at least correlating with) ABV. I meant to share it back then, but must have been distracted by something, because I didn't. I looked at the data several ways (by style, by hoppy styles only, by non-hoppy styles only, etc.), but I saw the same phenomenon no matter how I sliced it. Here is a scatter plot of every beer with at least 500 ratings (2,652 beers). X axis is ABV. Y axis is the rating from BA.

Now, does this mean that beers with higher ABV are better than beers with lower ABV? Is it the impact of more malt flavor? Or body? Does it mean we're alcoholics? Does it mean that people don't really rate "to style" despite their best intentions? If a perfect Imperial Stout is a "5," shouldn't a perfect pub style Blonde Ale also be a "5?" Please feel free to discuss and debate. It's the main reason for this post.

When I visit a big beer destination, I (like most people I would think) try to prioritize the best breweries. Beer ratings are a reasonable starting point, I think. But the ABV bias thing bothers me. I'd rather drink an excellent pilsner than a pretty good DIPA. So for the past few trips, I have used the analysis from back in 2020 to adjust raw beer ratings. Note the line of best fit in the chart. The slope is y=0.0913x+3.2563. So, on average, each increase in 1% ABV gives an increase in score of 0.0913. And the median ABV from the data was 6.8%. So, leaving the raw scores for 6.8% beers as-is, subtracting 0.0913 per each 1% ABV above 6.8, and adding 0.0913 per each 1% ABV below 6.8% is accomplished as follows:

AdjustedScore = RawScore + ((6.8- ABV) x 0.0913)

In my mind, applying that formula for each beer "levels the playing field." (Imagine driving a "Z" axis right through the center of mass of the scatter plot, and rotating until the line of best fit is parallel to the X axis.)

So what happens when applying the formula to the beers on tap at any given brewery? Well, not much, but when comparing adjusted averages for a set of breweries (in a beer destination), it has sometimes had the effect of moving a brewery up or down a spot or two in the pecking order. Which makes me feel better about my final beer destination itinerary, if nothing else.

I also did a sort of similar analysis of NHC winning beers, looking at ABV tendencies vis-a-vis style guideline ranges, and found that beers with ABVs in the upper part of the style guideline range (or often above) tended to win more often. This won't come as any surprise to most experienced competitors, I suspect. But that's a story for another day.

 

[mac_1103]

Bigger beers hide their flaws better? Lighter beers can't hide their flaws at all?

Just the first thing that popped into my head.

 

[sibelman]

One question: does the raw number of beers in any given category skew toward higher ABV? If so, these more numerous entries would be expected to show up more in the winner's circle. So, does your analysis correct for that in some way? Of course, I may have misunderstood; but the way we gather and present data can inadvertently shape the conclusions we draw.
All that said, the observation seems intuitively reasonable.

An anecdote: someone once told me she wouldn't bother with, e. g., an IPA below a particular ABV (I don't recall her number - possibly 6.x%) because it was presumptively not gonna be a good beer.

Another anecdote: my second cucumber pilsner is better (IMHO) than the first because the ABV (and maltiness) is lower.

As to the meaning of the 'pro-alcohol' judgments reflected in ratings/competitions: even if "we" aren't all/mostly alcoholics, affection for alcohol clearly factors prominently into why we drink beer.

Thanks, @VikeMan , for taking the time to gather and present these data.

 

[kurds_2408]

Doesn’t surprise me. Talking to my friends and coworkers, none of which brew, everyone seems to just want hops that get you drunk. I have a friend who told me once he doesn’t ever buy beer below 6%. Thinks it’s a waste of money. He loves drinking craft beer but has 0 knowledge of style and how beers should taste. Just wants something hoppy and dark. Which seems to be a trend in this area. Several of the local breweries have beers that don’t quite match the style they are labeled. The common theme seems to be take a known style and bump up the IBUs and ABV above what’s traditional for that style. They’re good beers though, so unless rated by a true beer enthusiast they probably aren’t rated right.

 

[Dland]

Just from my experience brewing, it is easier to brew full flavored beer when using sufficient amount of grist. For example, 20# grain works well in my rig to brew a 10+ gallon batch, that usually achieves an ABV of between 5.5-6.8.

When I try to scale back grain bill to get a lower ABV beer, for example using 17# grain for a 10 gallon batch, the resulting beers may be good, and often seem more "refreshing", but is also usually less flavorful.

Assuming normal attenuation is achived in both cases, the beer with more grain per gallon is going to have higher ABV,

And by flavorful, I don't mean over hopped, (in reaction to above post) but the full spectrum of malt, hops & body.

 

[Holden Caulfield]

Now, does this mean that beers with higher ABV are better than beers with lower ABV? Is it the impact of more malt flavor? Or body? Does it mean we're alcoholics? Does it mean that people don't really rate "to style" despite their best intentions? If a perfect Imperial Stout is a "5," shouldn't a perfect pub style Blonde Ale also be a "5?" Please feel free to discuss and debate. It's the main reason for this post.

Vikeman - great analysis.

I believe that there is a strong bias towards the more robust (stronger, bitter, maltier, roastier, American hop -Mosaic, Citra, etc -flavored) beer styles by the people who actually rate beers. It's not the ABV, but rather the styles and that those styles are most enjoyed, at least when drinking 1 by those that rate beers, are highly correlated to ABV.

Those that rate beers are not representative of the actual population of beer drinkers and they tend to rate classic lagers like Pilsners lower than the IPAs, Double IPAs, Imperial Stouts, Wilds, Quads, etc, despite the fact that most beer drinkers prefer more classic-like styles of beer. Bitburger, Pilsner Urquell, Bavik, London Pride, Boston Lager are as well made and deserving of the best of these heavy styles and at the end of the day most beer drinkers would actually prefer to drink these beers and styles day to day - its just that they do not rate beers, they just enjoy what they like.

Some additional data from Beeradvocate to shed light on the style bias by those that rate beers...

Highest rated Pale Lager is #9598
Pilsner Urquell is #26,150
Tucher Helles is #15,256
Top 250 beers did not include a single Pale Lager or Bitter (at least that my quick scan could tell)
Top 250 beers mostly include - Heavier stouts, IPAs, Wilds, Saisons, Quads, Fruit lambics

 

[huckdavidson]

To level the playing field you'd need the number of ratings considered for each beer to be the same.

Lower alcohol beers cost less, have more consumers with less invested by the consumer thus are more likely to be distributed over a wider population.

Higher alcohol beers cost more, have less consumers and more invested by the consumer thus are more likely to rate higher and are less likely to be distributed over a wider population. (Specialty beers, eisbock, doppelbock, etc... sought out by the consumer.)

 

[MaxStout]

Thanks, @VikeMan for putting in the time and thought to compile these data. Quite interesting--I've never thought of that angle, but it seems intuitive. Since many of those high-ABV beers tend to cost more, maybe people have the impression it's automatically better.

This is anecdotal and only one data point, but a local brewer once told me he'd never brew anything under about 5% ABV. We were talking about beers like Gratzer, and he said "people don't buy low-ABV beers...they want to know they're getting something." Understandable. When you brew 15 bbls per batch, you have to know what sells, and you have to sell it fast enough to free up those tanks for the next brew.

 

[Wrendelar]

Higher alcohol beers cost more, have less consumers and more invested by the consumer thus are more likely to rate higher and are less likely to be distributed over a wider population. (Specialty beers, eisbock, doppelbock, etc... sought out by the consumer.)

I think the cost correlation of higher abv craft beers also skews ratings. The mentality of "I paid $XX more for this high abv IPA than that low abv IPA, it must certainly taste better". What people spend on something and in part the rarity of it drives some individual's ratings higher. Whether they consciously are doing it or not.

 

[VikeMan]

One question: does the raw number of beers in any given category skew toward higher ABV? If so, these more numerous entries would be expected to show up more in the winner's circle. So, does your analysis correct for that in some way? Of course, I may have misunderstood; but the way we gather and present data can inadvertently shape the conclusions we draw.
All that said, the observation seems intuitively reasonable.

When I look at various categories (styles or style groups), I don't see any trend (in numbers of beers) toward higher vs lower ABVs.

I believe that there is a strong bias towards the more robust (stronger, bitter, maltier, roastier, American hop -Mosaic, Citra, etc -flavored) beer styles by the people who actually rate beers. It's not the ABV, but rather the styles and that those styles are most enjoyed, at least when drinking 1 by those that rate beers, are highly correlated to ABV.

When I look by individual styles, I see similar slopes (within each style) as for the whole universe.

 

[day_trippr]

My first thought on the bias thing would be the need to eliminated price as an influence...

Cheers!

 

[chipotlemonger]

I love this post, OP. Great thread! I try to rate to style as much as possible, but I wouldn't doubt that that sometimes does not happen.

 

[jrgtr42]

It’s a hell of a good question.
I think it does have to do with a lot of beer drinkers combined with the American tendency (and while I don’t have numbers, I’m guessing the majority of people rating are American) towards bigger is better. The more extreme the beer, the better it gets rated, no matter any flaws in the brewing, the concept etc.
I think part of it also comes from craft beer fans rebelling against the mass produced ones, the BMC, which are designed to be as unassuming as possible.
Personally, over the years, I’ve gone that route myself- early on in my beer drinking career, I would only drink stouts, thinking that if you can see through it, it’s not worth drinking.
But over the past few years, I’ve definitely developed an appreciation for a good Pilsner, kolsch and do on, the lighter, lower AbV styles.
Not only because as I get older, my tolerance has gone down, but also respect of those styles that there’s no mass hopping, huge roasted grains etc to hide behind; any flaws are right out in the open.

 

[VikeMan]

I decided to take a look at my own last 100 beers rated (excluding beers from brewers I know personally). I used my Untappd ratings rather than BA, because I don't normally rate on BA. I was kind of hoping for a very flat slope. What I got was flatter than the one at the top of the thread, but still showing a bias that correlates to ABV. So while I like to think that I rate strictly to style, the data suggests that may not be completely true.

 

[chipotlemonger]

I decided to take a look at my own last 100 beers rated (excluding beers from brewers I know personally). I used my Untappd ratings rather than BA, because I don't normally rate on BA. I was kind of hoping for a very flat slope. What I got was flatter than the one at the top of the thread, but still showing a bias that correlates to ABV. So while I like to think that I rate strictly to style, the data suggests that may not be completely true.

I do not think that your trend is noteworthy. You likely do rate flat as you can without an ABV bias (I am guessing that the Rsqu is pretty low for this graph). I would argue that the prevalance of 4% - 6% beer (beer's sweet spot) means that there are going to be more "poor" beers in this ABV range.

 

[Holden Caulfield]

So while I like to think that I rate strictly to style, the data suggests that may not be completely true.

If by what you mean is that you may have a bias toward certain styles to rate them higher than other styles, despite their relative quality, this is what I believe is very prevalent with those rating beers.

It not that those beers are better made or that the style is better, its just that the stouts, IPAs, Dark Belgian's, Sours, etc... are just given higher ratings. Hence the fact that not a single Pilsner, Helles, Bitter, etc... is in the Top 250 Beers on Beer Advocate. This explains why the correlation to ABV is weak.

If a regression were run assigning a 1 to the styles represented in the top 250 beers against all other styles that were assigned a 2, my hypothesis is that there would be a significant correlation.

 

[VikeMan]

It not that those beers are better made or that the style is better, its just that the stouts, IPAs, Dark Belgian's, Sours, etc... are just given higher ratings.

But, as I mentioned before, when I looked at the data within each style (i.e. separately), the correlation to ABV was still there. So there must be more to it than some styles being preferred over others.

 

[wsmith1625]

I would argue that the prevalance of 4% - 6% beer (beer's sweet spot) means that there are going to be more "poor" beers in this ABV range.

Great point.

I also believe that a majority of beer drinkers (including BA and Untapped members) need to be slapped in the face with flavor in order to appreciate a beer. Their palates aren't trained to pick up the subtle grain and hop flavors in lighter low ABV beers. Brewing taught me difficulty in brewing a clean and delicious light beer such as a lagers, cream ales, or kolsch that don't taste like macro brewed adjunct beers.

 

[Protos]

I always knew those "ratings" have little to do with the actual qualities of the beer. Just a tiny slice of unorthodox personal preferences of a tiny group.

As long as you don't brew Triple Soured NEIPAs or Imperial Pumpkin Potato Stouts you're never a good brewer enough

 

[bike2brew]

So a multi variable logistic regression would facilitate identification of the variables with the greater probability of association with the overall ranking