Beer data collection ideas

[naptown1]

SHORT VERSION:
Looking for ideas on beer related data I could scrape/collect.

LONG VERSION:
I am working on building a scraping pipeline with Apache Airflow and Scrapy and may try running it on my Raspberry Pi once i get it up. I need some ideas on the type of data i may collect. In the past when playing with scraping i grabbed the top beers from each style along with their reviews and did some analysis and visualizations with Bokeh on that data. I was thinking of taking what i did before and just build it out a bit and add some data to augment what I can already grab.

Basically i am a data nerd and looking to build something out to work with some new tech.

 

[JulianB1]

 

[Hanzo1]

Nice try Cambridge Analytica.

 

[naptown1]

Damn you have found out my plan to influence the elections by coopting the all important neckbeard vote.

 

[reverseapachemaster1]

Can you design an effective way to measure poops without including urine weight? That's what this site really needs.

#talkpoop

 

[Hanzo1]

Damn you have found out my plan to influence the elections by coopting the all important neckbeard vote.

I assumed you were just hired by AB InBev to gain intel on craft beer drinkers but seems this runs deeper than I ever expected.

 

[NeedsMoreDog]

is naptown a place? or a man?

 

[naptown1]

is naptown a place? or a man?

not quite sure what you are asking. it is just me. naptown is also the nickname for indianapolis. i am **** at coming up with usernames.

 

[ASUBeer]

Scrape all of the Barleywine checkins on untappd and then cross reference it with purchase volume at Taco Bell drive through after mid night.

 

[Lutter]

Can you design an effective way to measure poops without including urine weight? That's what this site really needs.

#talkpoop

 

[Hanzo1]

Those holes look a little big, not sure how that would work after a TB trip. Maybe cheese cloth instead? You could even squeeze out water for further accuracy.

 

[jivex5k1]

25 messages, member since 2013. What the hell...

 

[naptown1]

25 messages, member since 2013. What the hell...

not sure what you are saying

 

[jivex5k1]

not sure what you are saying

25 messages, member since 2013. What the hell...

 

[Fingolfin]

Arbitrator has already done it.

 

[naptown1]

it is the 'what the hell' part. do i need someone to vouch for me?? sorry i used to lurk for every and then stop visiting the site much cause life. we all can't just sit and drink barleywines in our parents basement playing video games and hitting refresh on TB

 

[Fingolfin]

we all can't just sit and drink barleywines in our parents basement playing video games and hitting refresh on TB

You got something against dontdrinkbeer?

 

[Arbitrator]

Friendly advice: sounds like you're a data scientist. Imo, a DS without project ideas is missing a critical skill, like if a mathematician popped into this forum saying he had just learned Fourier transforms and needed ideas for what he could use them on. Ideally, you'd have a problem in mind and would have learned those tools to solve it.

On to your tools, scraping isn't hard -- I did it with Scrapy alone. You really only need Airflow if you're trying to set up a regular incrementer (which raises the question, why?). So you're probably fine on just using Scrapy, and it's a useful skill. Instead of Airflow, I used a cron job -- which you can dump onto a remote instance. Airflow is the kind of thing I'd use to manage workflows, not a basic task; it's probably overkill.

What interests you about beer? And about this site in particular? Start from there and see if some data questions naturally bubble up. There's no shortage of them.

Here's a simple one: what are the top 10 most liked beers (from posts in DDT) on a daily basis?

 

[naptown1]

Close i am a data engineer. Yeah would be better if I had a problem I was wishing to solve but really looking to play with some stuff and I learn more if i build something myself then just doing tutorials.

I have done a bunch a scraping, that is just a means to end. I would prefer just using an API or something. And the use of Airflow is to really see in ongoing basis how it functions. In addition the scraping would be just a part of the workflow.

thanks for the help it does have me thinking. I like the idea of starting with the question "what interests you about beer"

 

[RyanG1]

Close i am a data engineer.

Can I **** on your stand up desk?

 

[Arbitrator]

Close i am a data engineer. Yeah would be better if I had a problem I was wishing to solve but really looking to play with some stuff and I learn more if i build something myself then just doing tutorials.

I have done a bunch a scraping, that is just a means to end. I would prefer just using an API or something. And the use of Airflow is to really see in ongoing basis how it functions. In addition the scraping would be just a part of the workflow.

thanks for the help it does have me thinking. I like the idea of starting with the question "what interests you about beer"

Ah, if you're an engineer then disregard my first para. Definitely learn your tools. People like me depend on people like you to do our jobs.

Also: plot the trend of various TB phrases over time, with the viz showing the avatar of the person who originated it. That'll use Scrapy, pandas, ntlk, and Bokeh.

E.g.

"Julian, please" would be MarkIntihar?
"**** on ____ desk" would be MordorMongo?

Etc

 

[jivex5k1]

we all can't just sit and drink barleywines in our parents basement playing video games and hitting refresh on TB

Sucks for you.

 

[naptown1]

Can I **** on your stand up desk?

i am too fat and lazy for a stand up desk