openaq/lcs-pilot

lcs-pilot's Issues

Discussion around cost of platform

One question that came up in talking to @bitner was how much, in terms of costs, OpenAQ is able to withstand/sustain for the pilot.

Ideally, we'd like to get an understanding of the scope of a couple of different options (e.g. here is what it would look like, ballpark, if we wanted to store data indefinitely, assuming this ingestion rate as opposed to here is what it would look like if we were to host 90 days of LCS data only).

But, also to some concreteness around this, in reality, it would be ideal for OpenAQ to support no more than double its current AWS monthly bill, which is ~2500/month at full bore of data ingestion beyond the pilot. In the short term, say in the first 6 months of the pilot, it'd be nice to be able to stick to no more than 25% more than our current costs.

It would be helpful for us to understand the AWS infrastructure cost sensitivity: # of data sources, rate of data ingest, amount of data accessed, or any other relevant parameters. This will help us understand how many partners we can/should include in the pilot, or a limit to the amount of data that can be shared per partner, a future costing model, etc.

Explore the use of TileDB for archived data access

TileDB is very similar to Parquet + Athena in that it can provide search for data that lives in S3.

TileDB provides more extensive compression and indexing than available with Parquet and Athena.

TileDB is designed to be very readily directly used for data science with direct mappings to data structures in many languages including Python and R.

Create repository for dataset configurations based on EDF pipeline

https://github.com/edf-org/aqdc-data-pipeline/tree/master/aqdc-projects

Explore options for PostgreSQL + TimescaleDB

TimescaleDB is a PostgreSQL extension for managing temporally indexed data. It automatically partitions and adds several enhancements (including massive speed improvements) for dealing with and aggregating temporal data. TimescaleDB also allows for compression that may allow more data to be stored on the "live" system.

Extract core of EDF SIDL conversion into standalone repository

https://github.com/edf-org/aqdc-data-pipeline/tree/master/sidl-converter

Create Crosswalk between OpenAQ and EDF data models

Potential LCS Data Providers / Data Sources

Linked here is a table showing a list of potential low-cost sensor pilot data providing parters.

These are groups/orgs/individuals who have previously expressed interest in sharing data or already provide data access freely.

Included is information on how to access data (eg. if there is an existing API or contact person).