adaptiveapplications/carnegie

civics

Git Branching Model: http://nvie.com/posts/a-successful-git-branching-model/

Amazon Developer (Alexa) accounts

• Login --> https://developer.amazon.com/home.html
• Summary of User Permissions
  • Administrator (Aaron)
  • Developer (let me know if you need a new invitation)
  • Marketer
  • Analyst

AWS accounts

• Everyone has an IAM account that is your first name
• Please login and change your password -> https://165284110550.signin.aws.amazon.com/console
• Summary of groups, roles, and roles is below. If you feel that you need additional permissions or to another user with a specific set of permissions, please post it on slack
  • alexa (role) has RDSFullAccess, LambdaFullAccess, S3FullAccess
  • lambda (role) has RDSFullAccess
  • Developers (group) has RDSFullAccess, EC2FullAccess, LambdaFullAccess, S3FullAccess

Build Lambda NodeJS

1. Create a zip archive containing *.js and the entire node_modules directory.
2. Upload the zip archive to the Lambda dev console.
3. Profit.

Geolocation Accounts

• http://www.geonames.org/
  • username: adaptiveapplications
  • password: reach out to ajdrake for password

Python setup (v2.7)

$ pip install numpy
$ pip install quantities
$ pip install geopy
$ pip install Geocoder

$ python gtfs_realtime_client.py <POLL INTERVAL> <BUS NUMBER> <MAX BUSES>

Python setup (v3.*)

$ pip3 install pymsql
$ pip install requests

$ python3 louisvillekygov-to-awscivicdata.py

Package
$ conda install virtualenv
$ virtualenv ./carnegie-ve
$ pip install carnegie-ve pymysql
$ cd carnegie-ve/lib/python3.6/site-packages
$ zip -r -X "carnegie-ve.zip" ./*

Scott's Journey (on a Mac, using homebrew)
$ brew install python3
$ brew link --overwrite python3  # run 'brew link python3' to see warnings
$ python3 -m venv ./carnegie-ve  # create new virtualenv (built-in to 3.6)
$ source carnegie-ve/bin/activate
$ python -m pip install pymysql
$ https://docs.python.org/3/installing/index.html

Upload the zip to AWS Lambda for testing

#Redis and EC2 ssh [email protected] -i ssh-ec2-pair.pem cd redis-stable redis-stable]$ src/redis-cli -c -h incident-cache.aqsren.ng.0001.use1.cache.amazonaws.com -p 6379

Notes and Tools

• https://dev.mysql.com/downloads/workbench/
  • civicdata.crogewynsqom.us-east-1.rds.amazonaws.com:3306
  • username: ro / password: civicdataalliance
• AWS Console https://165284110550.signin.aws.amazon.com/console
• Slack: adaptiveapplications.slack.com
• Package data script
• Zipcode/neighborhood source: https://suburbanstats.org/zip-codes/kentucky (and some other source I forgot + general knowledge)

carnegie's Issues

Scott's Permissions

@ajdrake
My AWS account for AA doesn't have permissions to create the new ElastiCache/Redis instance. I'm happy to do it if you want to alter my permissions or provide service account credentials.

memory usage too high for lambda

03:23:02
START RequestId: 1ebf1b01-c8eb-11e7-9aaa-2dcc1045648d Version: $LATEST

03:23:02
Execution for 2017-11-14 03:23:02.042031

03:23:02
lambda started

03:23:02
aws

03:23:02
Retrieving input file...

03:23:02
http://lky-open-data.s3.amazonaws.com/LMPD/Crime_Data_2017.csv

03:23:02
INPUT_FILE retrieved

03:23:08
END RequestId: 1ebf1b01-c8eb-11e7-9aaa-2dcc1045648d

03:23:08
REPORT RequestId: 1ebf1b01-c8eb-11e7-9aaa-2dcc1045648d Duration: 6487.61 ms Billed Duration: 6500 ms Memory Size: 128 MB Max Memory Used: 128 MB

03:23:08
RequestId: 1ebf1b01-c8eb-11e7-9aaa-2dcc1045648d Process exited before completing request

add support for ORM proxy

http://docs.sequelizejs.com or http://bookshelfjs.org/index.html#

Create Alexa skill in team Amazon account

The testing done during the hackathon was using Scott's account. Need to get it built in a team account.

change js files to use read only account for mysql connection instead of admin

account will be:

username: ro
password: civicdataalliance

Upload Louisville Crime Feed file to AWS

Publish for Alexa Skills test mode

Publish for Alexa Skills production mode

Fix the MySQL connection timeout that is occurring when testing Alexa skills

Pull most recent file from data.json for consistent daily import

pull the most recent file name from the data.json, because the update cycle can append the "_0" etc if someone manually manipulates a file.

in the python script geocache the locations while they sync to mysql database

add parameters to python data sync script so that it can do previous years

Upload Flash feed js file to AWS

Data sync script

script should pull the latest civic data csv file, and post it to the mysql database

import 2016 data into mysql database

change civicdata table column names to match CSV file headers

fill in zipcodelookup table using usps data

Send e-mail to request tester

add geocache field to the database table louisvilleky.crimedata

Upload js files to a central source repo

Correct civid typo in mysql instance