Add Breweries with missing/not enough data

[openbrewerydb]

Add Breweries with missing/not enough data

[chrisjm]

@Johnnyk737 Thank you so much for submitting this issue for discussion!

I'm still thinking this over but I currently foresee two challenges:

1. Most of the requests I get via Github and the Discord channel are to make the data cleaner. I worry that allowing incomplete data will cause more issues with staleness than solve for discovery reasons.
2. We are planning to do more automated data gathering in the future so I worry that a separate file will inevitably get stale.

Granted these are just initial thoughts and I'm open to discussing this further and/or brainstorm other ways in which to solve this issue. Thanks again and for the Ireland contributions! 🍻

[Johnnyk737]

Gotcha, well if the data is being used that way, then I don't see any good reason to add breweries with missing data at this time. I'd be happy to track down that data for the Ireland breweries once the other data points have been decided.

[Johnnyk737]

Closing this as we will not be moving forward with breweries with missing data at this time.

[Johnnyk737]

Glad to contribute!

Yeah that all makes sense. I do understand that the data could become stale. If it's allowed, it may turn into a "set it and forget it" situation. On the other side, I would hate to lose those breweries.

It would help to know how users of this information are using this. If they are wanting to know what is around them, all that is needed is a name and the lat/long values (and possibly type). At that point, their preferred Maps can handle the rest in getting them there (accuracy is another issue, I suppose).

As an aside, I'm curious to know more your automation plans.

[chrisjm]

@Johnnyk737 Sorry for the delay in response!

Automation plans mostly involve scraping brewery associations, brewers guilds, etc. for up-to-date information. We'd want to do this ethically (i.e., get permission, only scrape every month or so, space out requests to every 1 second, etc.).

It's still being worked out, so let me know if you have any thoughts or ideas!

[Johnnyk737]

I'm actually working on something similar (scraping mixed drink data for a bartending-type project). I just wrote a simple python script that grabs the main list (400+ items), then GETs each drink from the main page. After each GET, I sleep for 1 second. It's a little time consuming, but it keeps them from thinking it's a DDOS. For now, this is fine doing it once (and mixed drinks don't really change).

As for ideas on how to automate this, I haven't got any good ones aside from extremely manual processes initially. This would be finding the site to scrape from, separate scraper scripts per site as we can't count on the same format for each one. If you plan to extend this to other countries, then different languages and addresses have to be managed.

I think I this got away from the main topic though. I'd be happy to discuss this further separately from this topic! I can create one, it doesn't look like there is one specifically about adding automation.

How are other users of this information using this?

[chrisjm]

Re: the scraping, I added a new discussion issue: #78

As for how others are using this, I only have limited data but it's mostly beginner developers accessing the API by way of school or personal projects. Some companies are using it for technical interviews too.

AFAIK, it's not currently being used for any analysis other than mapping/geolocation. However, I'd like to change this by adding more data such as production capacity and other features. Along those lines, I have a half-finished NLP project I did while finishing a data science bootcamp, which I'd like to finish and put into the pipeline.