covid19tracking/covid-data-pipeline

Scan/Trim/Extra Pipeline for State Coronavirus Site

corona19-data-pipeline

Scan/Trim/Extract Pipeline for Coronavirus Site

• The code now expects to be run from the root directory of the repo. *
• This includes IDEs like VS Code. *

---

Scanner

1. Gets the data from urls in google sheet.
2. Pulls the raw HTML
3. Creates a clean version without the markup
4. Push it into a github repo.

Backup To S3

1. pulls an image for each page
2. pushed it to an S3 bucket

Specialized_Capture

1. Fire up a captive browser
2. For a list of urls, take a screen shot
3. If they change, push them into git

covid-data-pipeline's Issues

Screen capture isn't capturing full page

Can we capture the full vertical height of the data pages? Seems we're cutting off the bottom of a lot of pages in our screen capture.

cannot import name 'get_available_sources'

1. I just performed a fresh install
2. git checkout dev
3. python tests/test_sources.py
   Returns:

Traceback (most recent call last):
  File "tests/test_sources.py", line 10, in <module>
    from url_source import UrlSource, get_available_sources
ImportError: cannot import name 'get_available_sources' from 'url_source' (<<my file path>>\url_source.py)

For what it's worth:

$ grep -r "get_available_sources" .
./tests/test_sources.py:from url_source import UrlSource, get_available_sources
./tests/test_sources.py:    sources = get_available_sources()

It doesn't seem like get_available_sources is used anywhere outside of test_sources.

Am I missing something or is this an actual issue?

Resources for what data is avaliable for each WA county

Hey I am not sure what the best place to coordinate on pulling in new data, but I thought I'd take a survey of what data is available in each WA county as of time of writing.

Only 8 are currently reporting negative cases, 2 reporting hospitalization & 1 reporting recoveries.

https://docs.google.com/spreadsheets/d/1CS-jkQ7olNC1N0pi43fYsqNnGhq5jEBjHE0i1fhttCg/edit?usp=sharing

If y'all are planning on diving down to the county level I'm happy to work on any of them, but I care most about getting WA, GA, CA, TX & NJ up.

Question: Where does this run on?

Just curious where you have this running from, like on a server, serverless?

Integrate S3 for images

Add a new file/class that pushes image data to S3 using a standard naming convention.

It should be like DirectoryCache with methods like:
read/write/delete/list
but write is all we need for now.

Then make the screenshot routine push to that location.

Let's get the keys from the env for now. if the keys aren't present, print a warning and skip the push