This is a light service that can consume multiple data sources and make a uniform data set.
You should install docker-compose, clone the repo and run $ docker-compose up --build
set up the venv and install dependencies with $ pipenv install
To ingest data, run
$ pipenv run python API_ingest.py && pipenv run python csv_ingest.py && pipenv run python web_scrape_ingest.py
To run reports (total centers, centers in each zipcode) run $ pipenv run python reports.py
- Compare ds Columns - 15 min
- define initial schema - 15 min
- setup database container with test seed script - 1 hour
- env security? - 15 min
- set up and test dedoop - 1 hour
- setup and test scrape ingest- 5 hours
- setup and test csv ingest - 30 min
- setup and test API ingest - 30 min
- SQL scripts - 1 hour
- document!! - 1 hour
- needs way more tests
- multi-state support (not just California!)
- examine errors in logs and fix (looks like off by one errors persist in web scraper)
- Theres extremely large amounts of duplication according to our dedup method (phone number) This is a data smell for sure, work with stakeholders to come up with a better uniqueness system
- see commented TODOs
- data ingestion should be a script or single command