mainmatter/breethe-server

Air Quality Data for Locations around the World

breethe-server's Issues

Ignore outdated locations

There are a bunch of locations that do not have any fresh data, e.g. "lastUpdated": "2017-07-20T19:00:00.000Z". We should ignore all locations that we get from OpenAQ that have not been updated in at least the past 7 days.

It's better to not show locations than showing them but then having no data.

Cache PPM data

This is the second step where we if there is data in the database (that is still relatively recent), return that and still load the new data in the background and write it to the database for later requests.

Setup Travis CI

obviously

Load PPM data proactively

I think we should load a location's ppm data whenever that location is returned for a location search (either by lat/lon or by name). We can simply kick that off in a task and if that task finished before the request for the location data comes in that's good - if not than we simply load the data synchronously.

Remove ppm from UnitEnum

We are converting all :so2, :no2, :o3, :co values from ppm to µg/m^3 as needed.
We cannot convert :pm10, :pm25 and :bc, but these measurements should already be in µg/m^3. If not the app throws an error.

We should remove :ppm from the custom Enum type UnitEnum as it's never used.

Elixir.MatchError for nil OpenAQ measurement results

https://sentry.io/simplabs/ppm-server/issues/552262233/

Elixir.MatchError: no match of right hand side value: %{"meta" => %{"found" => 0, "license" => "CC BY 4.0", "limit" => 100, "name" => "openaq-api", "page" => 1, "website" => "https://docs.openaq.org/"}, "results" => []}
  File "lib/airquality/sources/open_aq/measurements.ex", line 43, in Airquality.Sources.OpenAQ.Measurements.query_open_aq/1
  File "lib/airquality/sources/open_aq/measurements.ex", line 7, in Airquality.Sources.OpenAQ.Measurements.get_latest/1
  File "lib/airquality_web/controllers/measurement_controller.ex", line 8, in AirqualityWeb.MeasurementController.index/2
  File "lib/airquality_web/controllers/measurement_controller.ex", line 1, in AirqualityWeb.MeasurementController.action/2
  File "lib/airquality_web/controllers/measurement_controller.ex", line 1, in AirqualityWeb.MeasurementController.phoenix_controller_pipeline/2
...
(3 additional frame(s) were not displayed)

(MatchError) no match of right hand side value: %{"meta" => %{"found" => 0, "license" => "CC BY 4.0", "limit" => 100, "name" => "openaq-api", "page" => 1, "website" => "https://docs.openaq.org/"}, "results" => []}

move sentry config to general configuration file

Switch to better data source

Unfortunately the data that we get from openaq.org is relatively poor. We should switch to a different data source as breethe only has real value if it shows good, up-to-date data.

controller error handling

Add more precise error handling for controllers
This should probably be a later step (after #9)

Add data source for US

Currently, we only get the data from the EEA which obviously only covers the EU. We need to connect a data source that covers the US as well (and likely/maybe more after that).

Tasks

• find free data source with good coverage and up-to-date data for all of the US (we don't want to connect to multiple data sources)
• write connector like we have for the EEA (do not rely on live data that's loaded within the request/response cycle but update periodically)
• test

Web endpoint for PPM data

This is the endpoint for returning the air quality data for a particular location. We can probably just return all data that we have at once.

Add quality_index to measurement

Stores computed value of air quality (good, bad, etc...)

Include measurement data in location payload

If the data is available in the db.
If not, return locations and start task to retrieve measurement data.

Add show endpoint for locations

i.e., api/locations/1

Elixir.Ecto.InvalidChangesetError: could not perform insert because changeset is invalid.

https://sentry.io/simplabs/ppm-server/issues/570417059/

Elixir.Ecto.InvalidChangesetError: could not perform insert because changeset is invalid.

Applied changes

    %{
      available_parameters: [:o3, :pm25, :bc, :co, :no2],
      city: "San Francisco-Oakland-Fremont",
      coordinates: %Geo.Point{coordinates: {37.864765, -122.30274}, srid: 4326},
      country: "US",
      identifier: "Berkeley Aquatic Par",
      last_updated: #DateTime<2018-06-02 12:00:00.000Z>
    }

Params

    %{
      "available_parameters" => ["o3", "pm25", "bc", "co", "no2"],
      "city" => "San Francisco-Oakland-Fremont",
      "coordinates" => %Geo.Point{coordinates: {37.864765, -122.30274}, srid: 4326},
      "country" => "US",
      "identifier" => "Berkeley Aquatic Par",
      "last_updated" => #DateTime<2018-06-02 12:00:00.000Z>
    }

Errors

    %{identifier: [{"has already been taken", []}]}

Changeset

    #Ecto.Changeset<
      action: :insert,
      changes: %{
        available_parameters: [:o3, :pm25, :bc, :co, :no2],
        city: "San Francisco-Oakland-Fremont",
        coordinates: %Geo.Point{coordinates: {37.864765, -122.30274}, srid: 4326},
        country: "US",
        identifier: "Berkeley Aquatic Par",
        last_updated: #DateTime<2018-06-02 12:00:00.000Z>
      },
      errors: [identifier: {"has already been taken", []}],
      data: #Breethe.Data.Location<>,
      valid?: false
    >

  File "lib/ecto/repo/schema.ex", line 128, in Ecto.Repo.Schema.insert!/4
  File "lib/breethe/data/data.ex", line 42, in Breethe.Data.create_location/1
  File "lib/enum.ex", line 1294, in Enum."-map/2-lists^map/1-0-"/2
  File "lib/breethe/sources/open_aq/locations.ex", line 7, in Breethe.Sources.OpenAQ.Locations.get_locations/2
  File "lib/breethe/sources/open_aq.ex", line 22, in Breethe.Sources.OpenAQ.get_locations/2
...
(3 additional frame(s) were not displayed)

(Ecto.InvalidChangesetError) could not perform insert because changeset is invalid.

Applied changes

    %{
      available_parameters: [:o3, :pm25, :bc, :co, :no2],
      city: "San Francisco-Oakland-Fremont",
      coordinates: %Geo.Point{coordinates: {37.864765, -122.30274}, srid: 4326},
      country: "US",
      identifier: "Berkeley Aquatic Par",
      last_updated: #DateTime<2018-06-02 12:00:00.000Z>
    }

Params

    %{
      "available_parameters" => ["o3", "pm25", "bc", "co", "no2"],
      "city" => "San Francisco-Oakland-Fremont",
      "coordinates" => %Geo.Point{coordinates: {37.864765, -122.30274}, srid: 4326},
      "country" => "US",
      "identifier" => "Berkeley Aquatic Par",
      "last_updated" => #DateTime<2018-06-02 12:00:00.000Z>
    }

Errors

    %{identifier: [{"has already been taken", []}]}

Changeset

    #Ecto.Changeset<
      action: :insert,
      changes: %{
        available_parameters: [:o3, :pm25, :bc, :co, :no2],
        city: "San Francisco-Oakland-Fremont",
        coordinates: %Geo.Point{coordinates: {37.864765, -122.30274}, srid: 4326},
        country: "US",
        identifier: "Berkeley Aquatic Par",
        last_updated: #DateTime<2018-06-02 12:00:00.000Z>
      },
      errors: [identifier: {"has already been taken", []}],
      data: #Breethe.Data.Location<>,
      valid?: false
    >

Cache location data

This is the second step where we if there is data in the database (that is still relatively recent), return that and still load the new data in the background and write it to the database for later requests.

Improve location description

Either by computing it on the fly or querying google API.

Elixir.Ecto.InvalidChangesetError: could not perform insert because changeset is invalid.

https://sentry.io/simplabs/ppm-server/issues/564726955/

Elixir.Ecto.InvalidChangesetError: could not perform insert because changeset is invalid.

Applied changes

    %{
      available_parameters: [:co],
      city: "צפון",
      coordinates: %Geo.Point{coordinates: {32.91575, 35.29302}, srid: 4326},
      country: "IL",
      identifier: "תחנה:נאות הכיכר",
      last_updated: #DateTime<2018-05-31 21:00:00.000Z>
    }

Params

    %{
      "available_parameters" => ["co"],
      "city" => "צפון",
      "coordinates" => %Geo.Point{coordinates: {32.91575, 35.29302}, srid: 4326},
      "country" => "IL",
      "identifier" => "תחנה:נאות הכיכר",
      "last_updated" => #DateTime<2018-05-31 21:00:00.000Z>
    }

Errors

    %{identifier: [{"has already been taken", []}]}

Changeset

    #Ecto.Changeset<
      action: :insert,
      changes: %{
        available_parameters: [:co],
        city: "צפון",
        coordinates: %Geo.Point{coordinates: {32.91575, 35.29302}, srid: 4326},
        country: "IL",
        identifier: "תחנה:נאות הכיכר",
        last_updated: #DateTime<2018-05-31 21:00:00.000Z>
      },
      errors: [identifier: {"has already been taken", []}],
      data: #Breethe.Data.Location<>,
      valid?: false
    >

  File "lib/ecto/repo/schema.ex", line 128, in Ecto.Repo.Schema.insert!/4
  File "lib/breethe/data/data.ex", line 42, in Breethe.Data.create_location/1
  File "lib/enum.ex", line 1294, in Enum."-map/2-lists^map/1-0-"/2
  File "lib/enum.ex", line 1294, in Enum."-map/2-lists^map/1-0-"/2
  File "lib/breethe/sources/open_aq/locations.ex", line 7, in Breethe.Sources.OpenAQ.Locations.get_locations/2
...
(3 additional frame(s) were not displayed)

(Ecto.InvalidChangesetError) could not perform insert because changeset is invalid.

Applied changes

    %{
      available_parameters: [:co],
      city: "צפון",
      coordinates: %Geo.Point{coordinates: {32.91575, 35.29302}, srid: 4326},
      country: "IL",
      identifier: "תחנה:נאות הכיכר",
      last_updated: #DateTime<2018-05-31 21:00:00.000Z>
    }

Params

    %{
      "available_parameters" => ["co"],
      "city" => "צפון",
      "coordinates" => %Geo.Point{coordinates: {32.91575, 35.29302}, srid: 4326},
      "country" => "IL",
      "identifier" => "תחנה:נאות הכיכר",
      "last_updated" => #DateTime<2018-05-31 21:00:00.000Z>
    }

Errors

    %{identifier: [{"has already been taken", []}]}

Changeset

    #Ecto.Changeset<
      action: :insert,
      changes: %{
        available_parameters: [:co],
        city: "צפון",
        coordinates: %Geo.Point{coordinates: {32.91575, 35.29302}, srid: 4326},
        country: "IL",
        identifier: "תחנה:נאות הכיכר",
        last_updated: #DateTime<2018-05-31 21:00:00.000Z>
      },
      errors: [identifier: {"has already been taken", []}],
      data: #Breethe.Data.Location<>,
      valid?: false
    >

fix measurement creation

Two different locations are returning same measurement ids (inconsistently)

fix: use location_id in find_measurement.

Release

In order to relase this we need to

• update README without license: https://choosealicense.com/no-permission/
• setup cloudflare
• open source
• rename repo to breethe-server

Configure Sentry to report Task crashes and errors

Error reporting for async tasks may require the use of Sentry.capture_exception/2

Deploy

@marcoow Any idea where you'd want to host this? Heroku has limitations: connections are limited, in memory state is lost every 24 hours, etc...

refactor OpenAQ.Locations

Rename files to match new app names

Replace `Dependabot Preview` with `Dependabot`

In theory it shuts down the 3rd of August 2021

https://github.blog/2021-04-29-goodbye-dependabot-preview-hello-dependabot/

add db constraint for location.identifier

This is to avoid duplicates.

Camel-case attribute names

http://orbitjs.com does not deal with attribute names the same way in the browser as in Node so we should just camel-case all attribute names to circumvent that problem.

use insert_or_update! for measurement and location create

This avoids throwing an error if we are fetching a record we already have from the openAQ api.

Narrow down CORS configuration

require json-api mime type

Require Accept: "application/vnd.api+json" header

remove translated null-records from measurement payload

Add error tracking

We need to setup Sentry so we get notified when things don't work: https://github.com/getsentry/sentry-elixir

remove static manifest from _web app config/prod.exs

We don't need it as we don't have any static files.

Web endpoint for locations

This is the web endpoint for location data which supports search by position as well as free text.

Load locations on demand

This is the simple first step where we only load data on demand and don't apply any caching etc.

Load PPM data on demand

This is the simple first step where we only load data on demand and don't apply any caching etc.

Optimise serializer to include data attribute on a per location basis

Remove data attribute if it is an empty array (location doesn't have any measurements) as discussed in #57

Serve a location's measurements from a nested route

We should serve locations from a nested route under /location as that is easier to handle on the client:

/locations/:id/measurements

Translate non existing measurement records

Translate non-existent records into

"attributes" => %{
  "parameter" => "pm10",
  "unit" => "ppm",
  "value" => null,
  "measured-at" => null
},

in the controller

Elixir.MatchError api/locations?filter[name]=sidney

https://sentry.io/simplabs/ppm-server/issues/555895296/

Elixir.MatchError: no match of right hand side value: %{"results" => [%{"address_components" => [%{"long_name" => "Sidney", "short_name" => "Sidney", "types" => ["locality", "political"]}, %{"long_name" => "Capital", "short_name" => "Capital", "types" => ["administrative_area_level_2", "political"]}, %{"long_name" => "British Columbia", "short_name" => "BC", "types" => ["administrative_area_level_1", "political"]}, %{"long_name" => "Canada", "short_name" => "CA", "types" => ["country", "political"]}], "formatted_address" => "Sidney, BC, Canada", "geometry" => %{"bounds" => %{"northeast" => %{"lat" => 48.671867, "lng" => -123.389405}, "southwest" => %{"lat" => 48.631178, "lng" => -123.4179581}}, "location" => %{"lat" => 48.6502411, "lng" => -123.399005}, "location_type" => "APPROXIMATE", "viewport" => %{"northeast" => %{"lat" => 48.671867, "lng" => -123.389405}, "southwest" => %{"lat" => 48.631178, "lng" => -123.4179581}}}, "place_id" => "ChIJWSxLC9Fnj1QR8UWcISfWukM", "types" => ["locality", "political"]}, %{"address_components" => [%{"long_name" => "Sidney Center", "short_name" => "Sidney Center", "types" => ["locality", "political"]}, %{"long_name" => "Sidney", "short_name" => "Sidney", "types" => ["administrative_area_level_3", "political"]}, %{"long_name" => "Delaware County", "short_name" => "Delaware County", "types" => ["administrative_area_level_2", "political"]}, %{"long_name" => "New York", "short_name" => "NY", "types" => ["administrative_area_level_1", "political"]}, %{"long_name" => "United States", "short_name" => "US", "types" => ["country", "political"]}, %{"long_name" => "13839", "short_name" => "13839", "types" => ["postal_code"]}], "formatted_address" => "Sidney Center, NY 13839, USA", "geometry" => %{"location" => %{"lat" => 42.2906379, "lng" => -75.2557286}, "location_type" => "APPROXIMATE", "viewport" => %{"northeast" => %{"lat" => 42.2988914, "lng" => -75.23972119999999}, "southwest" => %{"lat" => 42.2823833, "lng" => -75.27173599999999}}}, "place_id" => "ChIJKyOmm66e24kRq9NFVajuZoI", "types" => ["locality", "political"]}, %{"address_components" => [%{"long_name" => "Sidney", "short_name" => "Sidney", "types" => ["locality", "political"]}, %{"long_name" => "Clinton Township", "short_name" => "Clinton Township", "types" => ["administrative_area_level_3", "political"]}, %{"long_name" => "Shelby County", "short_name" => "Shelby County", "types" => ["administrative_area_level_2", "political"]}, %{"long_name" => "Ohio", "short_name" => "OH", "types" => ["administrative_area_level_1", "political"]}, %{"long_name" => "United States", "short_name" => "US", "types" => ["country", "political"]}, %{"long_name" => "45365", "short_name" => "45365", "types" => ["postal_code"]}], "formatted_address" => "Sidney, OH 45365, USA", "geometry" => %{"bounds" => %{"northeast" => %{"lat" => 40.324103, "lng" => -84.1180299}, "southwest" => %{"lat" => 40.2539969, "lng" => -84.216291}}, "location" => %{"lat" => 40.2842164, "lng" => -84.1554987}, "location_type" => "APPROXIMATE", "viewport" => %{"northeast" => %{"lat" => 40.324103, "lng" => -84.1180299}, "southwest" => %{"lat" => 40.2539969, "lng" => -84.216291}}}, "place_id" => "ChIJP_iExpEMP4gRzOa4U9qCKAY", "types" => ["locality", "political"]}, %{"address_components" => [%{"long_name" => "Sidney", "short_name" => "Sidney", "types" => ["locality", "political"]}, %{"long_name" => "Richland County", "short_name" => "Richland County", "types" => ["administrative_area_level_2", "political"]}, %{"long_name" => "Montana", "short_name" => "MT", "types" => ["administrative_area_level_1", "political"]}, %{"long_name" => "United States", "short_name" => "US", "types" => ["country", "political"]}, %{"long_name" => "59270", "short_name" => "59270", "types" => ["postal_code"]}], "formatted_address" => "Sidney, MT 59270, USA", "geometry" => %{"bounds" => %{"northeast" => %{"lat" => 47.7295171, "lng" => -104.1360509}, "southwest" => %{"lat" => 47.6960878, "lng" => -104.206216}}, "location" => %{"lat" => 47.7166836, "lng" => -104.1563253}, "location_type" => "APPROXIMATE", "viewport" => %{"northeast" => %{...
  File "lib/airquality/sources/google/geocoding.ex", line 13, in Airquality.Sources.Google.Geocoding.find_location/1
  File "lib/airquality/sources/open_aq.ex", line 7, in Airquality.Sources.OpenAQ.get_locations/1
  File "lib/airquality_web/controllers/location_controller.ex", line 12, in AirqualityWeb.LocationController.index/2
  File "lib/airquality_web/controllers/location_controller.ex", line 1, in AirqualityWeb.LocationController.action/2
  File "lib/airquality_web/controllers/location_controller.ex", line 1, in AirqualityWeb.LocationController.phoenix_controller_pipeline/2
...
(3 additional frame(s) were not displayed)

(MatchError) no match of right hand side value: %{"results" => [%{"address_components" => [%{"long_name" => "Sidney", "short_name" => "Sidney", "types" => ["locality", "political"]}, %{"long_name" => "Capital", "short_name" => "Capital", "types" => ["administrative_area_level_2", "political"]}, %{"long_name" => "British Columbia", "short_name" => "BC", "types" => ["administrative_area_level_1", "political"]}, %{"long_name" => "Canada", "short_name" => "CA", "types" => ["country", "political"]}], "formatted_address" => "Sidney, BC, Canada", "geometry" => %{"bounds" => %{"northeast" => %{"lat" => 48.671867, "lng" => -123.389405}, "southwest" => %{"lat" => 48.631178, "lng" => -123.4179581}}, "location" => %{"lat" => 48.6502411, "lng" => -123.399005}, "location_type" => "APPROXIMATE", "viewport" => %{"northeast" => %{"lat" => 48.671867, "lng" => -123.389405}, "southwest" => %{"lat" => 48.631178, "lng" => -123.4179581}}}, "place_id" => "ChIJWSxLC9Fnj1QR8UWcISfWukM", "types" => ["locality", "political"]}, %{"address_components" => [%{"long_name" => "Sidney Center", "short_name" => "Sidney Center", "types" => ["locality", "political"]}, %{"long_name" => "Sidney", "short_name" => "Sidney", "types" => ["administrative_area_level_3", "political"]}, %{"long_name" => "Delaware County", "short_name" => "Delaware County", "types" => ["administrative_area_level_2", "political"]}, %{"long_name" => "New York", "short_name" => "NY", "types" => ["administrative_area_level_1", "political"]}, %{"long_name" => "United States", "short_name" => "US", "types" => ["country", "political"]}, %{"long_name" => "13839", "short_name" => "13839", "types" => ["postal_code"]}], "formatted_address" => "Sidney Center, NY 13839, USA", "geometry" => %{"location" => %{"lat" => 42.2906379, "lng" => -75.2557286}, "location_type" => "APPROXIMATE", "viewport" => %{"northeast" => %{"lat" => 42.2988914, "lng" => -75.23972119999999}, "southwest" => %{"lat" => 42.2823833, "lng" => -75.27173599999999}}}, "place_id" => "ChIJKyOmm66e24kRq9NFVajuZoI", "types" => ["locality", "political"]}, %{"address_components" => [%{"long_name" => "Sidney", "short_name" => "Sidney", "types" => ["locality", "political"]}, %{"long_name" => "Clinton Township", "short_name" => "Clinton Township", "types" => ["administrative_area_level_3", "political"]}, %{"long_name" => "Shelby County", "short_name" => "Shelby County", "types" => ["administrative_area_level_2", "political"]}, %{"long_name" => "Ohio", "short_name" => "OH", "types" => ["administrative_area_level_1", "political"]}, %{"long_name" => "United States", "short_name" => "US", "types" => ["country", "political"]}, %{"long_name" => "45365", "short_name" => "45365", "types" => ["postal_code"]}], "formatted_address" => "Sidney, OH 45365, USA", "geometry" => %{"bounds" => %{"northeast" => %{"lat" => 40.324103, "lng" => -84.1180299}, "southwest" => %{"lat" => 40.2539969, "lng" => -84.216291}}, "location" => %{"lat" => 40.2842164, "lng" => -84.1554987}, "location_type" => "APPROXIMATE", "viewport" => %{"northeast" => %{"lat" => 40.324103, "lng" => -84.1180299}, "southwest" => %{"lat" => 40.2539969, "lng" => -84.216291}}}, "place_id" => "ChIJP_iExpEMP4gRzOa4U9qCKAY", "types" => ["locality", "political"]}, %{"address_components" => [%{"long_name" => "Sidney", "short_name" => "Sidney", "types" => ["locality", "political"]}, %{"long_name" => "Richland County", "short_name" => "Richland County", "types" => ["administrative_area_level_2", "political"]}, %{"long_name" => "Montana", "short_name" => "MT", "types" => ["administrative_area_level_1", "political"]}, %{"long_name" => "United States", "short_name" => "US", "types" => ["country", "political"]}, %{"long_name" => "59270", "short_name" => "59270", "types" => ["postal_code"]}], "formatted_address" => "Sidney, MT 59270, USA", "geometry" => %{"bounds" => %{"northeast" => %{"lat" => 47.7295171, "lng" => -104.1360509}, "southwest" => %{"lat" => 47.6960878, "lng" => -104.206216}}, "location" => %{"lat" => 47.7166836, "lng" => -104.1563253}, "location_type" => "APPROXIMATE", "viewport" => %{"northeast" => %{"lat" => 47.7295171, "lng" => -104.1360509}, "southwest" => %{"lat" => 47.6960878, "lng" => -104.206216}}}, "place_id" => "ChIJq3eWf4ZxJFMRhAMlC94gOm4", "types" => ["locality", "political"]}, %{"address_components" => [%{"long_name" => "Sidney", "short_name" => "Sidney", "types" => ["locality", "political"]}, %{"long_name" => "Gurley", "short_name" => "Gurley", "types" => ["administrative_area_level_3", "political"]}, %{"long_name" => "Cheyenne County", "short_name" => "Cheyenne County", "types" => ["administrative_area_level_2", "political"]}, %{"long_name" => "Nebraska", "short_name" => "NE", "types" => ["administrative_area_level_1", "political"]}, %{"long_name" => "United States", "short_name" => "US", "types" => ["country", "political"]}, %{"long_name" => "69162", "short_name" => "69162", "types" => ["postal_code"]}], "formatted_address" => "Sidney, NE 69162, USA", "geometry" => %{"bounds" => %{"northeast" => %{"lat" => 41.15650780000001, "lng" => -102.9405759}, "southwest" => %{"lat" => 41.109549, "lng" => -102.998219}}, "location" => %{"lat" => 41.1448219, "lng" => -102.9774497}, "location_type" => "APPROXIMATE", "viewport" => %{"northeast" => %{"lat" => 41.15650780000001, "lng" => -102.9405759}, "southwest" => %{"lat" => 41.109549, "lng" => -102.998219}}}, "place_id" => "ChIJtVzoBn-tcYcR83vYoP_1Pyk", "types" => ["locality", "political"]}], "status" => "OK"}

Update measurement data every night or so

We should have a simple mix task that updates the data for all the locations we have in the database every night or so so that the data is always fresh (enough) and can be returned right away without loading it synchronously.