Hypin' Hypin' Homers III: Revenge of the Homers

Homebrew Talk - Beer, Wine, Mead, & Cider Brewing Discussion Forum

Help Support Homebrew Talk - Beer, Wine, Mead, & Cider Brewing Discussion Forum:

This site may earn a commission from merchant affiliate links, including eBay, Amazon, and others.

stupac2

Well-Known Member
Joined
Apr 12, 2013
Messages
10,070
Reaction score
19,372
Location
Oakland, CA
I've been sitting on this for a little while because the numbers I was getting were different from before, and I couldn't figure out why. Turns out the data wizarded up for me was review-only. I thought for a second and realized that there were worse things that cutting out the hads, so I went with it. The one important thing to note is that this left some of the data fairly scant, so I combined Massachusetts and Vermont into a single state. Turns out most Hill Farmstead beers have 0-2 reviews from Vermont, so that was a way to keep them in the dataset, beers with 1 home review obviously aren't significant (and, it turns out, crash my stats library!).

If you want to see the whole damn thing of 240-some beers, it's right here.

Let's Get Significant

Since this dataset is so huge, I'm started out with only beers where the difference is statistically significant. For my definition of significant I went with P<0.05 because if it's good enough for medical research, it's good enough for this. The sheet of significant beers is here.

The Hype Civil Wars

I created a list off to the side of the states with the most significant beers, and you'll notice that California tops the list by far. But, at the same time, California has the most beers on the list, so I also divided by the total. In this light, it's actually Minnesota that's the worst, with all 6 of their top-250 beers having a statistically significant increase in scoring. The only other 100%-er with more than 1 beer on the list is North Carolina. The other 5 (Connecticut, New Mexico, Oklahoma, Utah, and Washington) only have 1 beer.

Moving past the 100%-ers, we see that Ohio, with 8 of its 9 beers being statistically overrated, is pretty bad. Illinois at 3/5 is too, but that's not all that many beers to be judging this off of, really. Oregon, Michigan, Mass, and Vermont/Mass are all around 40%, which is high but not ridiculous.

Florida is the shock to me on this sheet, with only 3/8 beers being significant. I think that would change with Hads included, since previously every single beer I looked at from Florida was not only significant, but among the most overhyped in the sample. That's not the case here, making Florida's reviewers seem pretty even-keeled, especially compared to their tickers.

Finally, California may have over double the next state in overhyped beers, but it's second-to-last on the sheet, trailing only Germany (Germany!) in hype-resistance. Go team!

In terms of average percentage overrating, most states are in the 2-5% range, with the only ones over 6% being Utah, Texas, and Wisconsin. But none of them have a lot of beers in the sample, so I don't know if we should read into that too much.

IPAs

As with last time, people were interested in this so I'm looking at it. (Just a note, I'm being pretty liberal with the term "IPA" and including Pale Ales, Pale Lagers, etc.) The IPA page is here. The top 10 overrated IPAs with significant differences are:

Citra Ninja- Pipeworks Brewing Company
Winter Ale (2012) / 5th Anniversary Ale- Kern River Brewing Company
Citra Pale Ale- Hill Farmstead Brewery
White Rajah- The Brew Kettle Taproom & Smokehouse / Production Works
Burton Baton- Dogfish Head Brewery
Jai Alai IPA- Cigar City Brewing
Gandhi-Bot Double IPA- New England Brewing Co.
Elevated IPA- La Cumbre Brewing Co.
Kiwi Rising - Double IPL- Jacks Abby Brewing
Alchemy Hour Double IPA- Great Lakes Brewing Company

Nothing on there seems wildly out of whack to me. Except Kiwi Rising, that **** is gold.

Anyway, nothing super interesting is jumping off of this page to me, except PsuedoSue, which is underrated by Iowans by over 4%. It's not significant, but it's interesting. None of the other underrated IPAs are anywhere near that magnitude.

Finally, the average difference in home vs away scores is only 2.34%. Really not all that big, and in line with what I got last time.

HEADY VS PLINY BECAUSE **** YOU THAT'S WHY

The home crowd definitely likes Pliny the Elder better than the away crowd, but the difference is tiny, just less than 1%. The enormous number of reviews makes that small difference significant. Compare this to Heady Topper, because everyone is always comparing these beers and no one ever gets tired of it, where the Vermont/Mass crowd rates it 2.4% higher than everyone else. Clearly, Pliny is the more consistent and therefore better beer. Clearly.

Stouts

Last time stouts were the more controversial style. Here's the data.

As before, here are the top 10 significantly overrated beers:

Fifteen- Central Waters Brewing Company
Chocolate Rain- The Bruery
Murdad Out Stout- Three Floyds Brewing Co. & Brewpub
Czar Jack Imperial Stout- Minneapolis Town Hall Brewery
Black Tuesday- The Bruery
Darkness- Surly Brewing Company
Big Bad Baptist- Epic Brewing Co.
Hunahpus Imperial Stout- Cigar City Brewing
Barrel Aged Sexual Chocolate- Foothills Brewing Company
Uncle Jacobs Stout- Avery Brewing Company

I'm not too surprised by anything on this list. Although you non-CA dudes are all clearly wrong about BT and CR, that **** is awesome.

The bottom of the list here is interesting, too, the following beers are all underrated by 2% or more:

Life Is Like...- Cigar City Brewing
Peche Mortel (Imperial Stout Au Cafe)- Brasserie Dieu Du Ciel
Assassin Imperial Stout- Toppling Goliath Brewing Company
Old Rasputin XV Anniversary Barrel Aged Stout- North Coast Brewing Co.
AleSmith Speedway Stout - Vietnamese Coffee- AleSmith Brewing Company

Of course, none of them are statistically significant, so it could just be by chance. But it's an interesting list to me.

Finally, the average disagreement is now only 2.43%, basically in line with the IPAs. Last time it was almost twice as high, so I guess I just picked controversial beers for that one.

Sours

Sours are funny. There aren't as many, and there are very few statistical differences. The presence of all the foreign ones compounds things in funny ways, too. Anyway, here's the data, and here are the ones with significant differences:

Cascade Sang Noir- Cascade Brewing / Raccoon Lodge & Brewpub
Atrial Rubicite- Jester King Brewery
Peche n Brett- Logsdon Farmhouse Ales
Oude Tart With Sour Cherries- The Bruery
Cuvee De Castleton- Captain Lawrence Brewing Co.

That's not the list I would've guessed, personally. Also kind of interesting to me is that all three 3F beers on the top 250 are in the top 7 biggest differences, just none are significant. I wouldn't have guessed that.

There are also a lot more underrated sours than the other ones (at least compared to the total number of beers), so the average only comes out to 1.4% overrating. Which is pretty small compared to IPAs and Sours. There are a couple of interesting underrated beers, first is Cable Car, which I guess just tastes better outside of the sticky, dirty confines of Toronado. The second one, 1v1willwreku and dontdrinkbeer will be interested to hear, is Fantome Saison, the most underrated sour, with Belgians liking it nearly 5% less than non-Belgians.

For Next Time

Since this data is only reviews, I think next time I'd like to look at only Hads, to see if those are different in any way. I'm out of things to examine with this dataset, but if you think of something, let me know!
 
Last edited by a moderator:
Thanks for all the work that went into this, really interesting stuff.

Finally, California may have over double the next state in overhyped beers, but it's second-to-last on the sheet, trailing only Germany (Germany!) in hype-resistance. Go team!

Amazing that Germany is so hype-resistant, with herrburgess's posts on BA, I would have assumed everyone that has ever been to Germany hates all non-germanic beers and rates true-germanic beers as the best ever.

edit: I guess this means that herrburgess is right and that German beers really are superior ;)
 
Amazing that Germany is so hype-resistant, with Hurbergress's posts on BA, I would have assumed everyone that has ever been to Germany hates all non-germanic beers and rates true-germanic beers as the best ever.
Yeah, that was funny. What was also funny is that with 9 beers on the list, Germany has more beers on there than most states. Making his inane crusade look even more ridiculous.
 
Also, that's a whole lot of data to digest...that's some serious work (at least for a math moron like me)!
 
Just curious as to what planet Backwoods Bastard is an IPA on? Srs.


LYMI, but srs.?
I must've just missed it. I had to filter those out by hand.

I can't edit, so could Gene or someone remove Backwoods bastard from the list and add Alchemy Hour Double IPA- Great Lakes Brewing Company to the bottom?
 
You also wrote "Which is pretty small compared to IPAs and Sours" when I'm guessing you meant IPAs and Stouts as you were comparing sours.

Still pretty interesting stuff and thanks for putting it together for sure.
 
You also wrote "Which is pretty small compared to IPAs and Sours" when I'm guessing you meant IPAs and Stouts as you were comparing sours.

Still pretty interesting stuff and thanks for putting it together for sure.
Why do you guys have to go and point out all my errors after it's too late for me to correct them?

Worst. Editors. Ever.

(Gene could you fix that one too?)
 
Cool stuff, I'll take a look at the full report later once my laptop is recharged and I'm not in my phone.
 
I couldn't resist and checked the spreadsheet on my phone anyway. I think it's instructive to observe the p-values and how these carry the most information about whether or not the observed differences are significant. The reason being that, if the sample sizes are small, the observed differences could appear large (relatively) yet not be significant because of the higher professionally probability they are due to random chance. On the other hand, smallish differences can be highly significant if the samples that produced them are large enough. Mathematically this all chimes down to the fact that the standard deviation of the sampling distribution is s/sqrt(n), where s is the observed standard deviation and n is the sample size. So for large values of n we would not expect very large differences in the data sets.
 
You and me both! It was totally not intentional for them to be left out. I think I'll be able to do that sometime next week, it depends on bensw.


Yeah sorry about that, I think I just ****ed up the URL that I was scraping so the hads=true was not set :(
 
I couldn't resist and checked the spreadsheet on my phone anyway. I think it's instructive to observe the p-values and how these carry the most information about whether or not the observed differences are significant. The reason being that, if the sample sizes are small, the observed differences could appear large (relatively) yet not be significant because of the higher professionally probability they are due to random chance. On the other hand, smallish differences can be highly significant if the samples that produced them are large enough. Mathematically this all chimes down to the fact that the standard deviation of the sampling distribution is s/sqrt(n), where s is the observed standard deviation and n is the sample size. So for large values of n we would not expect very large differences in the data sets.
I'm a bit confused, are you trying to say that I did something wrong, or just pointing this out for the benefit of the masses because I didn't? I sort of talked about it with regard to the tiny Pliny difference being significant because of the huge dataset, but definitely never fully explained it. (I figure people either know this already or don't care, so I haven't been dwelling on methods much.)
 
I regularly have to interpret data for my job. For example, to figure out if I want to recommend X drug over the new Y drug from looking at clinical studies. My own conclusion would be Minnesota and Oregon are, in fact, hypin' homers because they have an overall high number of significantly hyped beers and a high average difference. Ohio, Michigan, Vermont/Mass and California have a high number of significantly hyped beers, but the average difference is lower. Wisconsin, Texas and Utah have a low number of significantly hyped beers, but the average difference is very high. Take home point for me would be to second guess myself if I want something from Minnesota and Oregon.

Personally, I loved seeing The Bruery Chocolate Rain and Black Tuesday near the top. I can continue my hate and spamming "The Bruery" on any threads about, "what beers are overpriced or overrated?"

Another personal boner would be seeing Oregon with 6 beers and a 5.39%. Now I can bring this up when a PNW freak from Oregon explains they have no rare beer because they aren't "hypin' homers." There's some guy on those "other" forums (drahem?) that constantly portrays this attitude, so now he can go suck a big fat chubby.

 
I regularly have to interpret data for my job. For example, to figure out if I want to recommend X drug over the new Y drug from looking at clinical studies. My own conclusion would be Minnesota and Oregon are, in fact, hypin' homers because they have an overall high number of significantly hyped beers and a high average difference. Ohio, Michigan, Vermont/Mass and California have a high number of significantly hyped beers, but the average difference is lower. Wisconsin, Texas and Utah have a low number of significantly hyped beers, but the average difference is very high. Take home point for me would be to second guess myself if I want something from Minnesota and Oregon.

Personally, I loved seeing The Bruery Chocolate Rain and Black Tuesday near the top. I can continue my hate and spamming "The Bruery" on any threads about, "what beers are overpriced or overrated?"

Another personal boner would be seeing Oregon with 6 beers and a 5.39%. Now I can bring this up when a PNW freak from Oregon explains they have no rare beer because they aren't "hypin' homers." There's some guy on those "other" forums (drahem?) that constantly portrays this attitude, so now he can go suck a big fat chubby.


Draheim is from Washington, IIRC, so I think insulting Oregon would only be pleasing to him.
 
I'm a bit confused, are you trying to say that I did something wrong, or just pointing this out for the benefit of the masses because I didn't? I sort of talked about it with regard to the tiny Pliny difference being significant because of the huge dataset, but definitely never fully explained it. (I figure people either know this already or don't care, so I haven't been dwelling on methods much.)

The latter, just trying to shed some light on what those numbers represent and why the two columns are different. All the work looks good to me. :)
 
The top 10 overrated IPAs with significant differences are:

Citra Ninja- Pipeworks Brewing Company
Winter Ale (2012) / 5th Anniversary Ale- Kern River Brewing Company
Citra Pale Ale- Hill Farmstead Brewery
White Rajah- The Brew Kettle Taproom & Smokehouse / Production Works
Burton Baton- Dogfish Head Brewery
Jai Alai IPA- Cigar City Brewing
Gandhi-Bot Double IPA- New England Brewing Co.
Elevated IPA- La Cumbre Brewing Co.
Kiwi Rising - Double IPL- Jacks Abby Brewing
Alchemy Hour Double IPA- Great Lakes Brewing Company

Not at all surprised to see White Rajah on the list, but not because of homer hype. White Rajah (and the Brew Kettle's Pale Ale: 4 C's) seem to lose their flavor faster than any beers I've ever had. Fresh, I think they're tremendous, but not fresh they're mediocre.

I also wonder how people rank high-variation beers when they have them frequently. Take White Rajah, which I frequently buy and find to be high variation, either because of the freshness issue or some other factor. I'm inclined to think of White Rajah in terms of its peak value/potential, but other people wouldn't have that ability. In numbers, if I buy White Rajah 5 times and those 5 times are ranked:
5
4.75
3
2
2

I'm going to rank it (in my single ranking) as closer to 4.5 than the mean from that distribution. But if those 5 bottles were sent out to different people, the mean of their ranks (assuming identical tastes to mine) will be substantially below 4.5.

I would guess that both of these factors will lead to the appearance of hyping homers, but I would say the first definitely shouldn't (because these scores aren't freshness adjusted), and I'm not sure the second should either.

Anyway, good stuff.
 
I think looking at IPAs and stouts presents an interesting comparison, too, given that freshness is critical for one style and not for the other. Locality is of course not a perfect proxy for freshness, but it's pretty good for what we have.

I also think there's probably a difference in out-of-towner ratings between people who drank a beer while visiting someplace and people who drink it at home (i.e., they traded for it, bought it and had it shipped, etc.). Not that we have any real way of figuring this out.

Beer is, in a lot of ways, a situation-dependent experience. If you have a good beer in the middle of an amazing vacation, you're going to rate it higher than if you open one at home after your dog died. This isn't a criticism of the methodology -- average ratings probably account for this as well as is possible -- but a reminder that there's no true value that the average ratings estimate. It's not like taking the average heights of a bunch of people to estimate which country has the tallest people. It's actually a lot more complicated than that.

We can probably guess at some of the factors that influence ratings besides the actual quality of the beer (hype, difficulty of obtaining, style, freshness, personal rating styles) but not others (subjective experiences). It would be really interesting to try to isolate some of the first list.
 
I also wonder how people rank high-variation beers when they have them frequently. Take White Rajah, which I frequently buy and find to be high variation, either because of the freshness issue or some other factor. I'm inclined to think of White Rajah in terms of its peak value/potential, but other people wouldn't have that ability. In numbers, if I buy White Rajah 5 times and those 5 times are ranked:
5
4.75
3
2
2

I'm going to rank it (in my single ranking) as closer to 4.5 than the mean from that distribution. But if those 5 bottles were sent out to different people, the mean of their ranks (assuming identical tastes to mine) will be substantially below 4.5.

I suspect this is a significant issue. I'd definitely wager it's in play with some of the Hair of the Dog beers. (And maybe Hop Venom as well.)

I'd also be curious to see some analysis of temporality and serving format. I know that I thought that the first batches of Sang Noir (available only at the Holiday Ale Festival) were amazing beers; whereas the bottled version (year later and presumably what most non-homers have rated) was not nearly as good.
 
I've had 3 of the 5 significant sours (Sang Noir, AR, Oude Tart w/ sour cherries), and they're some of the best beers I've ever had. Didn't look up their scores or really know of any hype before trying them (couldn't really avoid it for AR living in Austin, but I distanced myself as much as possible and have had it 'many' times now). Sang Noir was my first Cascade.

Anyway, interesting project, thanks for doing this!
 
Also I'm not sure if it's been mentioned but for Atrial Rubicite I think the in-state/out-of-state variation could be due to batch differences. Most people who reviewed this out-of-state had Batch 1, which was older than Batch 2 and even people from Texas said they liked Batch 2 much better. I'd be willing to bet most of the in-state reviews are from Batch 2 since that's what was on tap for a while.
 
Also I'm not sure if it's been mentioned but for Atrial Rubicite I think the in-state/out-of-state variation could be due to batch differences. Most people who reviewed this out-of-state had Batch 1, which was older than Batch 2 and even people from Texas said they liked Batch 2 much better. I'd be willing to bet most of the in-state reviews are from Batch 2 since that's what was on tap for a while.
That has been mentioned before, but I'd be pretty surprised if it were a significant factor.
 
I think overall seeing the numbers has been very helpful. I said this in the first thread but I think there will always be a perception of homerism due to the actions of a few people.

Earlier I saw this ISO:

http://www.TalkBeer.com/communit...woodys-iso-ba-abraxas-rare-blabaer-list.3029/

I hadn't heard of the beer so I look it up:

http://beeradvocate.com/beer/profile/21154/100242/?ba=Skrypt

(Sorry for BA Link). First review for the beer and it just happens to be a 4.99 with the exact same picture in the avatar?

Now, overall I don't know much about Swamp Heads and that beer sounds like it is pretty good. But I can't help but feel like there is a bit of a hyping homer when the first review is 4.99 and then he wants to trade his 1 bottle per person, 60 bottle release. There might be another explanation for this (got to try it on tap, significant other also got a bottle so one for me one to trade...) but on the surface, for me, it definitely perpetuates the image of hyping homers.

edit: add this question to the ISO post:

Happy to hear a response, nothing wrong with using a small local limited release beer to try to land some bigger wants but I would be interested to hear where to draw the line between respectable trading/reviewing and hyping.
 
Last edited by a moderator:
I think overall seeing the numbers has been very helpful. I said this in the first thread but I think there will always be a perception of homerism due to the actions of a few people.

Earlier I saw this ISO:

http://www.TalkBeer.com/communit...woodys-iso-ba-abraxas-rare-blabaer-list.3029/

I hadn't heard of the beer so I look it up:

http://beeradvocate.com/beer/profile/21154/100242/?ba=Skrypt

(Sorry for BA Link). First review for the beer and it just happens to be a 4.99 with the exact same picture in the avatar?

Now, overall I don't know much about Swamp Heads and that beer sounds like it is pretty good. But I can't help but feel like there is a bit of a hyping homer when the first review is 4.99 and then he wants to trade his 1 bottle per person, 60 bottle release. There might be another explanation for this (got to try it on tap, significant other also got a bottle so one for me one to trade...) but on the surface, for me, it definitely perpetuates the image of hyping homers.

edit: add this question to the ISO post:

Happy to hear a response, nothing wrong with using a small local limited release beer to try to land some bigger wants but I would be interested to hear where to draw the line between respectable trading/reviewing and hyping.
Grats! You found my beeradvocate username! I was really trying to keep it a secret, you know, what with making the same exact post on both forums, moving references/trades from there to here, etc...

I've had the beer multiple times in the tasting room. I wrote what I think is a fair review of the beer, and it appears to be in line with every other review.

Because I have two bottles (you guessed correctly on the SO), I'd like to trade for something of equal quality and rarity. I've seen this get done 1:1 with a rare on that other forum (and proxied a similar trade for a friend), so I'm thinking I'm not too far off.
 
Last edited by a moderator:
Grats! You found my beeradvocate username! I was really trying to keep it a secret, you know, what with making the same exact post on both forums, moving references/trades from there to here, etc...

I've had the beer multiple times in the tasting room. I wrote what I think is a fair review of the beer, and it appears to be in line with every other review.

Because I have two bottles (you guessed correctly on the SO), I'd like to trade for something of equal quality and rarity. I've seen this get done 1:1 with a rare on that other forum (and proxied a similar trade for a friend), so I'm thinking I'm not too far off.

Wasn't trying to imply that you were doing anything underhanded or attempting to hide your identity. As I said at the bottom, it sounds like a great beer and bottle count/limits/PVW factor seem to make it a fair trade for most of what you are ISO. I think it is just a situation where from the outside it could easily look like hyping. Which further reinforces stupac's statistical analysis which seems to point to the direction that the actual amount of "hyping" that takes place is relatively small.
 
Wasn't trying to imply that you were doing anything underhanded or attempting to hide your identity. As I said at the bottom, it sounds like a great beer and bottle count/limits/PVW factor seem to make it a fair trade for most of what you are ISO. I think it is just a situation where from the outside it could easily look like hyping. Which further reinforces stupac's statistical analysis which seems to point to the direction that the actual amount of "hyping" that takes place is relatively small.
To be fair, I can't really make judgments about situations like this one. However, I would be positively shocked if this weren't a case of hyping (intentional or not, and by this guy or not). 60-bottle releases are just going to have that happen to them, but the sample size will be far too small for anyone to be able to really prove it (unless it's hilariously egregious, of course).

Anyway, 60 bottles? What the **** is that? Did they have one of those 10-gallon barrels? Did they say "aw **** it, let's keg 80% of this and watch the plebs dance for the 60 bottles we'll make"? That's a really bizarre number of bottles to release. If you only have 1 barrel, either bottle it all or don't bottle any. I don't get going for a medium there, except to build hype.
 
Anyway, 60 bottles? What the **** is that? Did they have one of those 10-gallon barrels? Did they say "aw **** it, let's keg 80% of this and watch the plebs dance for the 60 bottles we'll make"? That's a really bizarre number of bottles to release. If you only have 1 barrel, either bottle it all or don't bottle any. I don't get going for a medium there, except to build hype.
According to a bartender in the tasting room, they attempted to hand bottle and ended up under filling a bunch, which they can't legally sell.
 
I should have read this thread before posting. It's awesome. Great work stupac.

Fwiw, re: my accusation of homerism, you'll find my ba/untappd ratings to be rather critical of my home town brewery, often below the average, quirkzoo
 
I should have read this thread before posting. It's awesome. Great work stupac.

Fwiw, re: my accusation of homerism, you'll find my ba/untappd ratings to be rather critical of my home town brewery, often below the average, quirkzoo

Again, that further supports what I was trying to say, which is that when the data is taken in aggregate the actual amount of "homerism" is very little. But if you look at a single beer (often a limited release and therefore a small sample size) the appearance (or perhaps the possibility) of homerism goes up.
 
To be fair, I can't really make judgments about situations like this one. However, I would be positively shocked if this weren't a case of hyping (intentional or not, and by this guy or not). 60-bottle releases are just going to have that happen to them, but the sample size will be far too small for anyone to be able to really prove it (unless it's hilariously egregious, of course).

Anyway, 60 bottles? What the **** is that? Did they have one of those 10-gallon barrels? Did they say "aw **** it, let's keg 80% of this and watch the plebs dance for the 60 bottles we'll make"? That's a really bizarre number of bottles to release. If you only have 1 barrel, either bottle it all or don't bottle any. I don't get going for a medium there, except to build hype.
Interesting case. I agree with stupac2 about a 60 bottle release. And for them to under-fill a bunch, well they've bottled before right? But it's already been addressed.

IMO it's unintentional hyping. The beer has 5 ratings and is stellar so far, great. Maybe it got rated a little higher by one or all reviewers because of the rarity, but I doubt it was an intentional, concerted effort. I'm taking the glass half full look here. I don't think ShelbyTheGuy did anything wrong (note his review was also a month ago).
However, if someone else were to be hyping average homer beer to be world-class, then quirkzoo has a point.
Homers gonna homer, unless everyone starts approaching beer completely indifferent and with few preconceived notions. But that sucks the fun away from beer. Be subjective, enjoy it, promote what you like, and share it with others!
 
Back
Top