codeforpdx/recordexpungpdx

A project to automate the expungement of qualifying criminal records in Oregon. This project is done in conjunction with Qiu-Qiu Law.

RecordSponge

A project to automate expunging qualifying criminal records. This project is done in conjunction with Qiu-Qiu Law. Learn more in the wiki.

Please read our code of conduct.

If you're interested in learning more about our project and getting involved, please join us at one of our meetup events! You can also request an invite to join our Slack channel by contacting our CodeForPDX brigade leader, Hugh: [email protected]

Table of Contents

• Tech Overview
• Installation
• Running Components
• Testing
• Contributing
• Technical Documentation
• License

Tech Overview

This is a web app built using React for the in-browser interface, and a backend web service implemented with the Flask web framework in Python.

Our dev environment is entirely containerized with Docker, and no other dependencies need to be installed natively. We use Python's pipenv for maintaining backend dependencies. We use mypy to type check any optional typings and pytest to test backend code. We use NPM to develop and build the frontend code. Docker is used to build and deploy the app stack for both local development and for deployment to the web.

Installation

You can get your dev environment up and running with installing only Docker and docker-compose. The npm and backend dev servers run in docker containers, synced with source code directories so that code changes propagate on the local servers right away. If you have any trouble, don't hesitate to ask on our Slack channel!

1. Fork the repo to create a copy for your own github account, and clone your own copy. (Read CONTRIBUTING.md for important info about syncing code your on github)

2. Install docker

   • Mac

     • Follow installation instructions in: Getting Started -- Docker on Mac OS X

       (click on Get Docker for Mac [Stable])

   • Linux

     • First, follow: Docker Installation

     • Configure your user to run docker without sudo: https://docs.docker.com/install/linux/linux-postinstall/

     • Install docker-compose

   • Windows

     • Windows (as always) is a bit more challenging. Docker CE on Windows requires Hyper-V support, which is only available on Windows 10 Pro, Enterprise, or Education. If you have one of those versions, follow Docker's documentation to install Docker Engine.
     • If you have Hyper-V:
       • Install GNU Make for Windows using either Chocolatey or by downloading the executable from sourceforge, running the installer, and then putting the binary on your Path Environment variable.
     • If you have Windows 10 Community, Home, or other versions, create a Linux VM using VirtualBox and run Docker Engine through there. Alternately, you could partition your hard drive and dual-boot Linux to use for development.

     • Important Windows Note

       if your git config changes Linux-style line endings into windows-style line endings using the core.autocrlf config flag, you'll need to disable that in order for the docker builds to work correctly. You can disable this setting for just the recordexpungPDX repo by running git config --local core.autocrlf false from the root directory of this repo.

Running the docker stack

In the project's root directory, run make new. This pulls the dev-tagged "expungeservice" image and launches the containers using docker-compose. Start and stop the running stack with make up and make down.

After this target completes, you can navigate to http://localhost:3000 in the browser and connect to the React dev server with full hot-module reloading. This may take a minute or two to come up before it is available while it installs node modules. Check the service with make frontend_logs.

In the course of backend development, one may not need to be running the React/HMR dev server. To build the frontend static files, and use Flask to serve them, run the following command:

$ make frontend_down frontend_build

Then navigate to http://localhost:5000 to access the backend service directly. This is configured to serve the static files. make up will start the HMR dev server again at 3000 if you stopped it.

Whenever a dependency is added to the backend, someone needs to rebuild the dev-tagged image and push or folks will get errors when trying to run the stack. To do so, rebuild the image, and reload the backend with:

$ make backend_build backend_reload

If it works, push with:

$ make push

More details in DevOps README.

For more project documentation on Docker, some troubleshooting, and some basic commands, see: doc/docker.md

Cleaning

While in the directory of your local repo, run:

$ make clean

in order to remove build artifacts.

To completely remove containers, volumes, and compose networks, run:

$ make clobber

DevOps

See DevOps README.

Testing

Currently using pytest for testing the backend. Run all backend tests by running the following command in the project root directory:

$ make backend_test

This runs a pytest command to execute all the unit tests inside the backend docker container. All of these tests should pass if you have correctly set up the backend dev environment.

To run a subset of test cases without first shelling into the docker container, you can use a docker-compose exec command, which specifies a container by service name and a runnable command in the container in a single step, e.g.:

$ docker-compose exec expungeservice pipenv run pytest tests/[subdir]

To specify and run a subset of the test cases.

Contributing

Please see CONTRIBUTING.md

Technical Documentation

• Project technical design: doc/design.md
• Additional frontend docs, mostly design patterns: src/frontend/README.md.

License

This project is open source under the terms of the MIT License.

recordexpungpdx's Issues

Re-visit the system design doc

The design doc is out-of-sync wrt issues logged since and subsequent discussions. We should sync up:

• endpoint names

• query parameters: what filters are needed?

• POST bodies: what does the JSON sent to the server look like?

• responses: what does the JSON returned by the server look like?

• headers: what headers does the API require?

• status codes: what status codes do endpoints return?

Search component integration

Create a component that allows you to search records and return results.

States:

• Empty search form
• Validations on search form
  • All search fields are required.
  • The date format must be MM/DD/YYYY.
  • Technical error, please try again in a few minutes.
• Loading
• No records found
• Search results
  • See mockup below for details

HTML to use: https://github.com/hmarcks/expunge-pdx-frontend/tree/master/search

implement views as react components

Endpoint: /api/search

Endpoint for searching the remote site. The web app will POST to this endpoint with search params in the body, and the endpoint will make a request of the remote site.

Data format TBD.

Fix errors in auth table

The auth table has an extra comma and is missing a semicolon.

refactor case parser

The case parser could be refactored to a state design pattern.

Create a case parser

Build a parser that returns a list of charges for a case from the case detail page.

Build project wiki on github

The project wiki on github provides external users with documentation about what our task is, what we're doing and ensures continuity when the project is handed off.

Describe the task here, including any supplemental information.
Tag yourself, or whoever is responsible for it.

This task is done when...

• build out the project wiki

Update README for installing Python 3.7

Need instructions for Mac and Linux

Sync parser and analyzer data classes

The parser has some data classes (e.g. Case) which are also duplicated in analyzer. Sync the two so that analyzer can understand objects generated by parser.

OR

Have the main app convert from one to another. The analyzer has predefined enums defined for some of the data types which basically stems from the expunge analysis flowchart while the parser objects are likely to have strings as types mostly.

Child of #9

Update database schema w/suggestions from Tim

Tim made some suggestions Jan 2 about our users table. He recommends moving the columns related to authentication to a separate table.

Build Icon Component

Create a reusable icon component in ReactJS.

We are no longer using React, so we'd need a new way to insert inline svg.

This is low priority as we can use font icons for now, but inline svg is ideal.

Here's relevant article: https://css-tricks.com/pretty-good-svg-icon-system/

Add main backend logic

This would probably implement the logic when user clicks "Search".

This should invoke the crawler/parser to get the data for given input (client name, dob), then call the analyzer, get the results and store it in database.

Update name of analysis table to "analyses" to match pluralization in other tables

Except for auth, the other tables in config/postgres/initdb/scripts/create-tables.sql are pluralized (users, clients, rules, etc.), so the analysis table probably should be analyses so it matches the other tables.

This task is done when:

• The "analysis" table is renamed to "analyses"

Configure integrated tests

Set up CI along with a testing framework and directory

Create Login Form

Build react components that are reusable for building forms and provide form validation configurations.

Create parser to pull valid params from search page

There are a few params that are sent back in the response when visiting navigating to the search. Need to retrieve those params in order to perform a search.
child of #11

Main login integration

Integrate the UI to log in to the app, reset password, and set password.

User flow:

Template w/ error messages: http://dev.huntermarcks.net/log-in
Design system: http://dev.huntermarcks.net/design-system

Update error methods in the crawler

The parsers have methods that will run when an error is encountered. These methods need to be implemented so that when an error does occur the user and the dev team can be notified appropriately.

Files to be updated:

• Node Parser https://github.com/codeforpdx/recordexpungPDX/blob/master/src/backend/expungeservice/crawler/parsers/node_parser.py#L16
• Param Parser https://github.com/codeforpdx/recordexpungPDX/blob/master/src/backend/expungeservice/crawler/parsers/param_parser.py#L34
• Record Parser https://github.com/codeforpdx/recordexpungPDX/blob/master/src/backend/expungeservice/crawler/parsers/record_parser.py#L53

This is done when...

• The methods are updated so that it logs the error
• And the methods raise the error so that the user can be notified of the issue

Class for connecting to database

Write a Python class for connecting to a Postgres database. Constructor should take host, port, name, username, and password and should expose connection and cursor objects. Usage should be:

(in context of HTTP request, such as a GET or POST handler):

import database
g.database = database.Database(db_host, db_port, db_name, db_username, db_password)
g.cursor.execute("""SELECT username FROM users;""")
rows = g.cursor.fetchall()
print(rows)

Create a record parser

Build a parser that collects all the data from the search results page on OJCIN that builds a list of cases. Each row in the results page would be a case.

Account HTML template

Create view in HTML / CSS for the Account view.

This is very similar to the "Add user" view.

Wireframes: https://marvelapp.com/644d2f9/screen/50214083
Design system: http://dev.huntermarcks.net/design-system

Will probably want to build in the prototype for now. You can fork this repo: https://github.com/hmarcks/expunge-pdx-frontend

Endpoint: /api/users/

Endpoint for making and getting users. Endpoint should handle POST and GET requests.

POST /api/users/ to create a user:

{
    "username": "<USERNAME>",
    "email": "<EMAIL>",
    "admin": "<TRUE|FALSE">
}

admin is optional and will default to False

GET /api/users/USER_ID to get a user

The GET response is TBD but will include some set of columns in the users table. It could be:

{
    "user_id": "",
    "username": "",
    "email": "",
    "admin": "",
    "date_created": "",
    "date_modified": ""
}

test issue please ignore

testing

Build a project plan with milestones

Partner requests a project roadmap with timelines for deliverables.

Describe the task here, including any supplemental information.
Tag yourself, or whoever is responsible for it.

This task is done when...

• a project plan is developed and team syncs on deliverables
• plan is shared with partner agency

Automate navigation of the site

Child of #8

Cannot install the itsdangerous dependency

Description

When running pipenv install the following error occurs.
Could not find a version that satisfies the requirement itsdangerous==1.0.0

Possible fix

Update to the latest version

Do not parse charges on Record parser

Context

The record parser extracts the cases from the search results page. It also extracts the charges associated to the case, but this is not necessary since the charges can be extracted from the case details page in more detail.

Tasks

• Update tests to check the parser skips the charge column.
• Remove the sections responsible for parsing the charge columns for each case.
• Ensure all existing functionality does not change.

Create database objects

Add user view

Create view in HTML / CSS for adding a new user

Wireframes: https://marvelapp.com/644d2f9/screen/50214073
Design system: http://dev.huntermarcks.net/design-system

Will probably want to build in the prototype for now. You can fork this repo: https://github.com/hmarcks/expunge-pdx-frontend

Automate expunge decision

Build a tool that will analyze each charge and decide if it can be expunged

Containerize Nginx

Make a Docker container for the webserver. The weberver will:

• - serve the React application
• - route requests to the Flask service
  Involves:
• - writing an Nginx config file config/nginx/nginx.conf with static route to the web app and routes to the Flask services
• - configuring Nginx to use HTTPS
• - adding a stanza for the web server to docker-compose.yml
• - adding a target to Makefile to build the web server image
• - adding a target to Makefile to build the web application as a docker volume that the web server will use to serve static html/css/images/etc

We also need to pick a Docker image to use for this task. Something like this: nginx - Docker Hub seems like a good place to start

Containerize Flask app

Build a Docker container for our Flask application. This container will run the Python/Flask code.

This involves:

• adding a Docker file to src/backend/expungeservice
• adding a stanza to docker-compose.yml
• adding targets to Makefile for building the image, starting and stopping the container (see how the database container is done)
• flesh out src/backend/expungeservice/wsgi.py
• pick a base image, such as library/ubuntu to reference in src/backend/expungeservice/Dockerfile

Endpoint: registration & login

Write an endpoint to log the user into the internal website. Take the user's credentials, validate against information stored in the database, and return a token.

This endpoint should handle POST, with the body containing username and password in JSON:

{
    "username": "<USERNAME>",
    "passowrd": "<PASSWORD>"
}

See issue #42

Epic: Search if a person has eligible expungeable records

As a user
I would like to enter in a search name and/or date of birth.
so that I can see a list of eligible and non-eligible expungeable records.

Parent of #6, Parent of #7, Parent of #9, Parent of #10

Create A Footer Component

Build a footer component that provides a copyright and terms link.

Endpoint: /api/stats

Endpoint for reporting on system statistics such as number of searches done, number of records expunged, and anything else we think is interesting

Setup hosting environment for prototype

We'll need a dev environment for the prototype of this application.

This task is done when...

• identify the hosting environment (AWS, et. al.)
• Setup environment

type eligibility list interface

I am currently hard coding a list of crimes that are ineligible for expungement (List A, List B, Etc.), however I get the feeling that maybe this list should be stored in the database and have some sort of interface so the public defenders office can view it and/or modify it as necessary.

I imagine that these lists don't change very often, but i'd probably want this feature if i was them.

Refactor front end to Tachyon theme

Replace Bootstrap with Tachyon and implement an IA that provides modular and global scopes. Also use BEM naming conventions for class(Names).

statute should be normalized before persisting

Description

Currently the charge class is initialized with a statute with whatever string value is passed to it. The Oregon e-courts website displays statutes in a variety of representations. A statute may be 6, 7, or 8 characters long, and may or may not contain decimal points, parenthesis, and other separators.

Have the charge initializer call a method to normalize the statute so that only alphanumeric characters are being persisted.

This task is done when...

• A charge class is initialized with a statute containing only alphanumeric characters.

Implement remaining rules in analyzer

The analyzer has a few rules setup. Implement the rest.

Child of #9

Create an "exercise" task to help new folks get up to speed faster

In the apps I've worked on before, we always had an exercise task that newcomers would complete just after joining. This task was designed to make sure newcomers could "learn by doing" by making a set of minimal, full-stack changes to give them something to do while they get familiar with the codebase.

If we had an exercise task like this it would also help with triage when lots of new people join: we could immediately give them something to work on that would teach them about the codebase and help them be more useful when working on features or refactors later on.

This task is done when:

• There is an exercise task that meets the above requirements. This should probably be a page in the wiki that introduces the different parts of the app code and how the work together, and that walks someone through the process of making a simple, full-stack feature change or addition.

Add info on setting up the frontend to the README.md

The README.md includes info on setting up and running the backend in development. Here is an excerpt:

Running:

$ make run

will run the Flask app inside a Pipenv virtualenv.

This task is done when:

The README.md includes instructions on setting up the frontend in development (installing npm, npm install, etc.) and running the development server with npm run start

Build React app into webserver container

Build the webapp into a Docker volume used by the webserver container. Write a Make target to control that process

See issue #43

Usability testing milestone

Once we have a prototype, we'll need an entire sprint focused around usability testing. Without testing users on the experience, we'll not be sure how to iterate on the existing design and validate our assumptions.

This is a milestone (epic) issue that will include all of the tasks for usability testing.

Create crawler

The crawler will accept search terms and login credentials to login & navigate the site using the parsers to scrape payloads required to crawl the site and to extract data from the search results page and each associated case page.

Child of #8

CaseParser is now crashing when parsing case 74 for J.D.

Description

The Case Parser is now crashing on a case it had previously run through without issue.

Update case parser to check for probation information

The case parser also needs to collect information regarding probation violations. Update case parser to check if probation was revoked.
Use the test fixture in: https://github.com/codeforpdx/recordexpungPDX/blob/master/src/backend/tests/fixtures/case_details.py#L639

This task is done when...

• Tests exists for new functionality
• Case has a method Revoked that it responds to
• CaseParser collects probation information

Figure out user authentication