updated for Nov, 2019
- Slowly porting to Python 3
- CouchDB please consult project docs on how to install configure
- copy setting.py.orig to settting.py
- fill in details about couch database server
- run
python settings.py
to create directories, only need to do this once
- will port previous version
- details soon
- previous step will download raw HTML and clean it
- Assure all data ready to be loaded is in:
AN_IN
- run
python load_db.py
to attempt database inserts of all files - successfully loaded files show up in
AN_OUT
- errored out files will end up in
AN_ERROR
- some cleanup is done during download/change/upload to DB process
- TBD for rest of process
- TBD. (Most likely will be a Jupyter Notebook)