openbrewerydb/openbrewerydb-rails-api

Official v1 Open Brewery DB REST API built with Ruby on Rails

🍻 Open Brewery DB Dataset

This is the open-source dataset for the Open Brewery DB API which is served by a REST API built with Ruby on Rails

🎯 Purpose

Provide an approval-based pipeline to update the dataset and API.

🗄 Data Formats

• CSV - Full Dataset
• JSON
• PostgreSQL SQL

API

Access the dataset programmatically via the Open Brewery DB API. Use the following tools to get started without any code:

databar.ai

If you don't know how to use APIs, you can use Brewery DB without code through the databar.ai platform.

Postman

A shared Postman collection containing all the API requests to fetch breweries information from the open-source dataset.

🚀 Getting Started

1. git clone [email protected]:openbrewerydb/openbrewerydb.git
2. cd openbrewerydb && npm install

🤝 Contributing

For information on contributing to this project, please see the contributing guide and our code of conduct.

1. Fork the repository
2. Add or update breweries in the CSV (Excel, Google Sheets)
3. Submit a Pull Request

Tips

First and foremost, don't worry about messing up! 🙂 Thank you so much for contributing! 🙌

• CSVs are organized by data/[country]/[state_province]
• Required fields/columns: name, brewery_type, city, state_province, postal_code, and country
• When adding a brewery, do not include an id. This will be created after review.
• Please either add to breweries.csv (preferred if adding breweries for a new country) or the individual state/province CSV file. Adding to both at the same time may introduce duplicates/errors.

⚙️ Scripts

These are the npm scripts used to maintain this dataset.

• npm run csv:combine - Combine CSVs from country/state-region folders into breweries.csv
• npm run csv:split - Split breweries.csv into country/state-region/city CSVs
• npm run contributors:add - Add contributor (interactive CLI)
• npm run contributors:check - Check if there are any missing contributors
• npm run contributors:generate - Generate contributors into README.md
• npm run generate:ids - Generate unique OBDB IDs based on the brewery name and city and overwrite breweries.csv
• npm run generate:json - Generate JSON from breweries.csv output to breweries.json
• npm run generate:sql - Generate PostgreSQL SQL from breweries.csv output to breweries.sql
• npm run validate - Validate CSVs based on JSON Schema
• npm run workflow:maintain - Combine, generate, split (used when updating individual CSVs)

👾 Community

• Join the Newsletter
• Join the Discord

📫 Feedback

Any feedback, please email me.

Cheers! 🍻

Contributors ✨

Thanks goes to these wonderful people (emoji key):

Mike Putnam 🔣	Andrew A. Barber 🔣	Jason Allen 🔣	Juicob 🔣	Will Karnasiewicz 🔣	Dylan T. Vavra 🔣	Madison Martinez 🔣
Daniel Eremchuk 🔣	Alex Chong 🔣	Matt S 🔣	Samuel Rusher 🔣	Evan Caraway 🔣	Tyler K Kuromiya Parker 🔣	kendellmendoza 🔣
Johnnyk737 🔣	James Schuler 🔣	Creighton Leif 🔣	Vitaly Tomilov 💻	Kyle Scudder 🔣	Chris Mears 💬 💻 🔣 🚧 📆 🔧 ✅	donkeyslaps 🔣
Pranav Davar 🔧	Alexandre Hernandes Barrozo 🔣	Resten 🔣	Matt Higgins 🔣	Alex Justesen 🔣	Craig Kelly 🔣	Krzysztof Rewak 🔣
John Baumert 🔣	Charlie Cox 🔣	Miles Kane 🔣	Anthony Laflamme 💻	Georg Engelsmann 🔣	Clinton Williams 🔣	Brent Busby 🔣
kenster89 🔣	Adilet Sarsembayev 🔣	Pranav Davar 🔣	b-mc2 🔣	Nicole 🔣	Nicholas Hance 🔣	Joachim Nilsson 🔣
Alejandro Lopez Rocha 🔣	zshapleigh 🔣	Praval Visvanath 🔣	JohnHenry 🔣	Alfredo Garcia 🔣

This project follows the all-contributors specification. Contributions of any kind welcome!

openbrewerydb-rails-api's Issues

Handle `Elasticsearch::Transport::Transport::ServerError` exception

Bug Report

Describe the bug

Elasticsearch::Transport::Transport::ServerError
[429] {"code":429,"message":"Concurrent request limit exceeded. Please consider batching your requests, or contact [email protected] for help."}

Return code 429 Too Many Requests.
Message: Concurrent request limit exceeded for ElasticSearch. Please delay concurrent calls using debounce or throttle.

Accept a list of types

• Here:

  openbrewerydb-rails-api/app/models/brewery.rb

  Line 26 in 99c31da

  scope :by_type, ->(type) { where('lower(brewery_type) = ?', type.downcase) }

• Maybe other places as well?
• Need to handle Nil case that keeps erroring

Add Tags and associate with Brewery

Feature Request

• As a User I want to be able to retrieve and search breweries using one or more tags so that I can further filter my search results.

Implementation Details

Initial tags will be:

• pet-friendly
• patio
• food-truck
• food-service
• tours

The endpoint call should be something like:

https://api.openbrewerydb.org/breweries?by_tag=pet-friendly

with a response of:

{
  id: 5494,
  name: "MadTree Brewing",
  brewery_type: "regional",
  street: "3301 Madison Rd",
  city: "Cincinnati",
  state: "Ohio",
  postal_code: "45209-1132",
  country: "United States",
  longitude: "-84.4239715",
  latitude: "39.1563725",
  phone: "5138368733",
  website_url: "http://www.madtreebrewing.com",
  tags: [
    "pet-friendly",
    "food-service",
    "patio",
  ],
  updated_at: "2018-08-24T15:44:22.281Z"
}

Checklist

• Add Tag and Taggings model
• Add by_tag filter to BreweriesController::index
• Set up ElasticSearch
• Unit tests
• Integration tests

RFC: Move from relational to document store database?

Discussion

Reasoning

• In discussing a way to allow for community-driven dataset updates, I had a thought to create another repository with a collection of JSON files (probably organized by country then state/providence) which could then be compiled and update a DB via Github Actions or another CI/CD process. With this process, it would likely be easier to update a DB like Mongo or DynamoDB.

To Research 📚

• Is key-value Document Store / NoSQL database able to do location-based queries like PostGIS?
  • #34 (comment)
  • MongoDB Geospatial Queries

Pros and Cons

Relational DB (PostgreSQL, MySQL, MariaDB, etc)

Pros

👍 Able to perform more complicated queries such as the full-text search and PostGIS / geospatial searching (not fully researched; key-value might be able to do this as well)
👍 Relational databases are more robust for data management

Cons

👎 Somewhat complicated to set up
👎 Expensive (~$180/year with AWS RDS reserved)

Document Store DB (MongoDB, DynamoDB, etc)

Pros

👍 Easier to set up from a development standpoint
👍 Cheaper (<$12/year, with current stats; maybe even free)
👍 Can be very performant if data is organized properly

Cons

api test suite

Hello.
I train the skill of writing tests for api. Api.openbrewerydb.org was chosen as an api for tests. You may be interested to know the test results. Or you can apply these tests to improve api. Link to the project with tests https://github.com/vshagur/openbrewerydb-api-tests. Work on the tests is not finished yet. If you want to add checks to the test suite, please write which ones. I will add.
Sorry for my English.

Add ability to filter to `random` endpoint

Feature request

Currently, we randomize all breweries. It would be nice to be able to filter by the usual filters and then select the random brewery from that.

Add pagination information to the response headers

Not sure if I should raise this as a bug or feature request. It would be great if the pagination information was added to the response headers to be used in conjunction with the the page and per_page params`. Stuff like total_count so if using this to build a UI it can be used to build the pagination controls.

Thanks :)

Add Github Actions for CI

Overview

It's been a while, but I think we were using Travis CI for the continuous integration (CI).

Continuous deployment (CD) is handled by Heroku, but isn't currently working because CI isn't set up.

Tasks

• Add Github Actions for testing repo
• Remove .travis.yml
• Update README
• Update build status badge

Implementation Notes

• This Stack Overflow answer was very helpful in figuring out how to set up PostgreSQL in a container.

Add ability to query by postal code

Overview

There isn't an ability to query by postal code, but there should be. ZIP+4 should be optional.

Tasks

• Add by_postal query option to /breweries

Issues

#32

Implement rate-limiting

Add meta data to headers (v1)

Feature request

Overview

There's not currently a way to get a total count along with the results for high-level information.

For example:

{
  "results": [],
  "total": 84
}

Tasks

• Include total with results

Implementation notes

This section will be used by the assigned developer for any additional notes.

geospatial indexing / more granular location queries

It’d be great to have the ability to search by location.

eg, [url]?by_location=34.3,76.5

I’d be happy to take up this issue

Upgrade Ruby

Feature request

Overview

Upgrade to the most recent ruby version (though probably < 3.0 unless we can get away with it; 2.7?)

Upgrade Ruby on Rails to 6.1

Feature request

Overview

Investigate upgrading to 6.1

Better caching

Feature request

Overview

We need better caching.

Tasks

???

Add Beer Information

Would you consider expanding the scope of this project to include beer information as well?

Example User Scenario:
As an API user, I want to be able to query by beer names and optionally brewery information to return information like:

• The Brewery
  • name: ,
  • brewery_type: ,
  • street: ,
  • city: ,
  • state: ,
  • postal_code: ,
  • country: ,
  • longitude: ,
  • latitude: ,
  • phone: ,
  • website_url: ,
  • updated_at: ,
  • tag_list: []
• The beer
  • beer_name: ,
  • beer_style: "",
  • beer_abv: ,
  • pDev: ,
  • tasting_notes: ,
  • description: ,
  • breweryId : (or name?)

`by_state` not working for some abbreviations (e.g. "ny")

Describe the bug
The by_state filter isn't working for some inputs.

To Reproduce
https://api.openbrewerydb.org/breweries?by_state=ny

Expected behavior
https://api.openbrewerydb.org/breweries?by_state=new_york

Incorrect search for multiple words (underscores)

Bug Report

Describe the bug

From the documentation for search queries - "Note: For the query parameter, you can use underscores or url encoding for spaces." Any query for multiple words (underscores are used as a separator) returns an empty dictionary.

Steps to reproduce bug

1. We make a request to api (use underscore) - https://api.openbrewerydb.org/breweries/search?query=san_brewery
2. Server sends an empty response - screenshot
3. We make an API request with the same parameters (use url encoding) - https://api.openbrewerydb.org/breweries/search?query=san%20brewery
4. Server sends response - screenshot

Expected behavior

It was expected that search results do not depend on which separator is used.

Screenshots

The request is made from the console.

Device information

Desktop

• OS: [Linux Samsung 4.9.0-8-amd64 #1 SMP Debian 4.9.144-3.1 (2019-02-19) x86_64 GNU/Linux]
• Browser [chromium]
• Version [Версия 73.0.3683.75 (Сборка для разработчиков), built on Debian 9.8, running on Debian 9.11 (64 бит)]

Smartphone

None

Additional context

We get a similar behavior when we use an autocomplete request.

For example -

https://api.openbrewerydb.org/breweries/autocomplete?query=the_brewery

https://api.openbrewerydb.org/breweries/autocomplete?query=the%20brewery

Return random brewery

Feature request

Overview

Return a random brewery when the /breweries/random endpoint is called.

• Bonus points: n param to return a random number of n breweries?

URL encoded filters and sorts not working

Bug Report

Describe the bug

Filter and sort methods don't seem to work with url encoding where a space is encoded as a +.

Steps to reproduce bug

I'm using broad brook as a test string in the PHP example below but was able to replicate this issue with Postman as well.

$uri = '?by_name=broad+brook' // returns 0 results

Expected behavior

$uri = '?by_name=broad+brook' // returns 1 result (expected)

Add Authentication & Authorization

Feature Request

• As an Admin or Moderator, I want to be able to authenticate my requests via JWT, so that I can make authorized updates.
• As an Admin, I want to be authorized for all actions, so that I can create, update, and delete breweries, tags, and users.
• As a Moderator, I want to be authorized for some brewery actions, so that I can create and update all information about breweries.
• As a User, I don't want to be authorized, so that I can just query the API.

Implementation Details

• Authentication: devise
• Authorization: CanCanCan

Checklist

• Implement Authentication
• Implement Authorization
• Implementation Tests

Add ability to exclude `by_type`

Overview

Use case: "I am trying to get a list of all the breweries in Austin, TX that are not in "Planning" type"

Implementation Notes

• exclude_types parameter name?
• Multiple types may be included via comma-separation (ex. ?exclude_types=planning,regional)

Add CRUD endpoints for moderators and admin users

Update `sort` filter to use `asc` and `desc` parameters

Feature request

Overview

The current + and - parameters for sorting are succinct and fancy, but very confusing. Let's instead use asc and desc which is more inline with database vocabulary.

• Pros - Easier to understand
• Cons - We lose the ability to sort by multiple features, though I'm not sure that worked anyway.

Import task for breweries CSV

Overview

Add import task for db/breweries.csv. This will replace the old method of import via the Brewers Association scraped HTML files.

Tasks

• Create new import task in app/services/import/

Related issues

#17 #23

Provide brewery data dump

Is your feature request related to a problem? Please describe.
I'd like a way for developers and data analysts to easily pull brewery information data and to contribute updates via pull requests while a proper user-contributor app is being worked on.

Describe the solution you'd like
I want to pick a format of data that will be easily generated via the app and imported on each deploy. Some considerations are SQL, YAML, and JSON.

Describe alternatives you've considered
I've considered using a NoSQL DB for this project but thought a relational DB would be better. I'm open to a discussion on this.

Use `obdb_id` instead of `id` for getting a single brewery

Feature request

Overview

• Currently, the API gets the brewery by id. However, this ID is no longer persistent. (it's changed on each DB update).
• We are starting to use a Yelp-esque unique ID for breweries in the form of obdb_id.
• Use obdb_id for searching instead of ID
• Or maybe allow for both? (probably not necessary)

Add Brewery Slugs

Feature Request

• As an API user, I want to be able to query by brewery slug, so that I don't need to remember the ID (???)
• As a Brewery Owner, I want to edit my breweries's slug, so that I can keep consistent branding and SEO (???)

Remove Brewers Association import task

It's fairly irrelevant now that there's a CSV export included in the repository.

Better ElasticSearch index?

Overview

We want to return more expected results from the /breweries/search and /breweries/autocomplete endpoints.

Goals

• Return more expected results or at least be able to explain results.

Tasks

• Look into the ElasticSearch index creation via Searchkick (gem/library) and Bonsai (hosting) and report findings

Random request being cached by cloudflare and returns same result

Bug Report

Describe the bug

Random endpoint being cached by Cloudflare (according to headers in response for 24h). Which makes this API call useful once a day.

Not sure if it is intentional, I believe that makes the random endpoint a bit useless.

Steps to reproduce bug

1. Go to Open Brewery DB
2. Click on Run for a random request
3. It will always return the same result, in dev tools response headers you will see:

cache-control: public, max-age=86400
cf-cache-status: HIT

Expected behavior

Disable caching for /random endpoint

Screenshots

Additional context

Passing Cache-Control: 'no-cache' with a request does not help

BUG: List breweries sorting; `+` not working as expected

Bug Report

Describe the bug

The documentation for api does not correctly indicate the direction of sorting

Steps to reproduce bug

1. Go to 'https://www.openbrewerydb.org/documentation/01-listbreweries'
2. We look at the sorting section.
3. Copy the link from the second example and send the request.
4. We get the result in descending order.

Expected behavior

We expect the result in ascending order. The documentation says: "- for ascending order (default)"

Screenshots

screenshots_1, screenshots_2,

Device information

Desktop

• OS: [Linux Samsung 4.9.0-8-amd64 openbrewerydb/openbrewerydb-gatsby#1 SMP Debian 4.9.144-3.1 (2019-02-19) x86_64 GNU/Linu]
• Browser [Chromium]
• Version [Версия 73.0.3683.75 (Сборка для разработчиков), built on Debian 9.8, running on Debian 9.11 (64 бит)]

Smartphone

None

Additional context

In addition to the error with the direction of sorting in the description, there is no clarity. You may get the impression that you need to use the "+" key to sort in ascending order. I suggest that you explicitly indicate that you do not need to use the "+" key and add an example of a default sorting link (now there is only a combined case for two fields in the description). Using the "+" key in a query leads to an incorrect result. Screenshot.

Filter by lat/long and distance

Feature request

Add the ability to search for breweries within a radius given a lat/long point.

See https://github.com/alexreisner/geocoder#geospatial-database-queries

Utilize PostgreSQL's Full Text Search, (ie, remove Elasticsearch dependency)

Overview

The idea here is to remove the dependency on Elasticsearch since it relies on another service and costs extra $$$. PostgreSQL has the ability to do a full-text search across multiple tables so we might as well use that.

Resources

• PostgreSQL Full Text Search Intro
• PostgreSQL tsquery datatype

Tasks

TBD

Add columns `address_2`, `address_3`, and `county_province`

Feature request

Overview

To get prepped for worldwide domination, the DB needs some new columns to handle international addresses.

Reference:
openbrewerydb/openbrewerydb#39

Add the following columns to the DB:

• address_2 and address_3
• county_province

Tasks

• Create migration

Screenshots & Diagrams

Additional context

Implementation notes

Documentation

• Reindexing Elasticsearch
• Heroku deploy: https://devcenter.heroku.com/articles/heroku-button I don't think this is actually necessary
• Rails version

Gemfile

• Add versions to avoid confusion

Remove search?

Curious if adding the Elasticsearch is beneficial and a useful and necessary architecture addition.

Can search and autocomplete be solved in a better way without the need for an additional resource?

adding sorting to filtering queries changes the list of returned items

Bug Report

Describe the bug

adding sorting to filtering queries changes the list of returned items

Steps to reproduce bug

1. We will make a request for receiving data with a filter by name with the value "town" - https://api.openbrewerydb.org/breweries?by_name=town
2. We repeat the previous query, but with the addition of sorting by name - https://api.openbrewerydb.org/breweries?by_name=town&sort=name
3. Compare the id brewing of the first request with the second.
   First -
   [189, 398, 535, 618, 624, 735, 736, 880, 926, 975, 1067, 1134, 1328, 2338, 2476, 3663, 3802, 6245, 7601, 7674]
   Second -
   [398, 2338, 2740, 3107, 3247, 3802, 3984, 4675, 5054, 5660, 5661, 5740, 5749, 6000, 6046, 6387, 7339, 7358, 7758, 7802]

We see that the second query returned a different list of breweries.

Expected behavior

When the api user adds sorting to filtering requests, he expects to receive exactly the same data set, but sorted by the field defined by him. This is a natural behavior. This natural behavior can be seen in search queries and autocompletion queries. For example queries -
https://api.openbrewerydb.org/breweries/autocomplete?query=town and
https://api.openbrewerydb.org/breweries/autocomplete?query=town&sort=name will return the same list - [ '624', '2338', '2524', '2688', '2922', '3584', '3984', '5183', '5895', '5906', '6387', '6827', '7615', '7749', '7765']

Screenshots

The natural behavior.
Bug.

Device information

Desktop

• OS: [Linux Samsung 4.9.0-8-amd64 #1 SMP Debian 4.9.144-3.1 (2019-02-19) x86_64 GNU/Linu]
• Browser [Chromium]
• Version [Версия 73.0.3683.75 (Сборка для разработчиков), built on Debian 9.8, running on Debian 9.11 (64 бит)]

Smartphone

• None

Additional context

• None

Validate arguments

Bug Report

Describe the bug

Most of the errors via Sentry are due to invalid arguments sent to the endpoints. Validate the arguments and inform the user before moving on to processing the request.