agrc/geocode-job

The process for comparing vista data with agrc addresses data in the google cloud

Geocode-job

Create a containerized python geocoding script as a kubernetes job.

Steps to run

1. Prepare data
   1. Run prep_addresses.py
      • Pulls data from VISTA
      • Partions the data into multiple CSVs
2. Create k8s job yaml specifications
   1. Run vista_job_template.py
      • Uploads data to Cloud Storage with service account credenitals
      • Creates k8s job template files
3. Apply seceret for service worker with cloud storage permissions to k8s cluster
   1. authorize kubectl with geocoding api cluster
   2. run kubectl apply -f .secrets/gcs-secret.yml
      • Service account key must first be base64 encoded into gcs-secret.yml
4. Apply job yamls to cluster
   1. run kubectl apply -f job.yaml
5. Download geocoded CSVs from cloud storage

Steps to build

1. Build container from docker file
   1. docker build . -t {container name}
   2. docker tag {container name}:latest gcr.io/{project id}/webapi/{container name}:latest
2. Push to registery
   1. docker push gcr.io/{project id}/webapi/{container name}:latest
   2. User needs project permissions to allow push to gcr

geocode-job's Issues

more residence_id

Is this going to fail looking for UNIQUE_ID

geocode-job/vista/combine_results.py

Line 53 in 01e71b2

arcpy.AddJoin_management(vista_layer, 'RESIDENCE_ID', geocode_layer, 'INID')

align sql query with template

https://github.com/keatonwalker/geocode-job/blob/master/vista/prep_addresses.py#L9

needs to match with

https://github.com/keatonwalker/geocode-job/blob/7673e5641cc8679978ac20523b7b7261ecba3760/vista/vista_job_template.py#L84