codeforpdx/recordexpungpdx

A project to automate the expungement of qualifying criminal records in Oregon. This project is done in conjunction with Qiu-Qiu Law.

recordexpungpdx's Issues

Update name of analysis table to "analyses" to match pluralization in other tables

Except for auth, the other tables in config/postgres/initdb/scripts/create-tables.sql are pluralized (users, clients, rules, etc.), so the analysis table probably should be analyses so it matches the other tables.

This task is done when:

• The "analysis" table is renamed to "analyses"

Update error methods in the crawler

The parsers have methods that will run when an error is encountered. These methods need to be implemented so that when an error does occur the user and the dev team can be notified appropriately.

Files to be updated:

• Node Parser https://github.com/codeforpdx/recordexpungPDX/blob/master/src/backend/expungeservice/crawler/parsers/node_parser.py#L16
• Param Parser https://github.com/codeforpdx/recordexpungPDX/blob/master/src/backend/expungeservice/crawler/parsers/param_parser.py#L34
• Record Parser https://github.com/codeforpdx/recordexpungPDX/blob/master/src/backend/expungeservice/crawler/parsers/record_parser.py#L53

This is done when...

• The methods are updated so that it logs the error
• And the methods raise the error so that the user can be notified of the issue

Create database objects

Search component integration

Create a component that allows you to search records and return results.

States:

• Empty search form
• Validations on search form
  • All search fields are required.
  • The date format must be MM/DD/YYYY.
  • Technical error, please try again in a few minutes.
• Loading
• No records found
• Search results
  • See mockup below for details

HTML to use: https://github.com/hmarcks/expunge-pdx-frontend/tree/master/search

Usability testing milestone

Once we have a prototype, we'll need an entire sprint focused around usability testing. Without testing users on the experience, we'll not be sure how to iterate on the existing design and validate our assumptions.

This is a milestone (epic) issue that will include all of the tasks for usability testing.

Configure integrated tests

Set up CI along with a testing framework and directory

Implement remaining rules in analyzer

The analyzer has a few rules setup. Implement the rest.

Child of #9

Containerize Flask app

Build a Docker container for our Flask application. This container will run the Python/Flask code.

This involves:

• adding a Docker file to src/backend/expungeservice
• adding a stanza to docker-compose.yml
• adding targets to Makefile for building the image, starting and stopping the container (see how the database container is done)
• flesh out src/backend/expungeservice/wsgi.py
• pick a base image, such as library/ubuntu to reference in src/backend/expungeservice/Dockerfile

Update case parser to check for probation information

The case parser also needs to collect information regarding probation violations. Update case parser to check if probation was revoked.
Use the test fixture in: https://github.com/codeforpdx/recordexpungPDX/blob/master/src/backend/tests/fixtures/case_details.py#L639

This task is done when...

• Tests exists for new functionality
• Case has a method Revoked that it responds to
• CaseParser collects probation information

Endpoint: /api/search

Endpoint for searching the remote site. The web app will POST to this endpoint with search params in the body, and the endpoint will make a request of the remote site.

Data format TBD.

Build project wiki on github

The project wiki on github provides external users with documentation about what our task is, what we're doing and ensures continuity when the project is handed off.

Describe the task here, including any supplemental information.
Tag yourself, or whoever is responsible for it.

This task is done when...

• build out the project wiki

Figure out user authentication

Sync parser and analyzer data classes

The parser has some data classes (e.g. Case) which are also duplicated in analyzer. Sync the two so that analyzer can understand objects generated by parser.

OR

Have the main app convert from one to another. The analyzer has predefined enums defined for some of the data types which basically stems from the expunge analysis flowchart while the parser objects are likely to have strings as types mostly.

Child of #9

Refactor front end to Tachyon theme

Replace Bootstrap with Tachyon and implement an IA that provides modular and global scopes. Also use BEM naming conventions for class(Names).

Build a project plan with milestones

Partner requests a project roadmap with timelines for deliverables.

Describe the task here, including any supplemental information.
Tag yourself, or whoever is responsible for it.

This task is done when...

• a project plan is developed and team syncs on deliverables
• plan is shared with partner agency

Add user view

Create view in HTML / CSS for adding a new user

Wireframes: https://marvelapp.com/644d2f9/screen/50214073
Design system: http://dev.huntermarcks.net/design-system

Will probably want to build in the prototype for now. You can fork this repo: https://github.com/hmarcks/expunge-pdx-frontend

Containerize Nginx

Make a Docker container for the webserver. The weberver will:

• - serve the React application
• - route requests to the Flask service
  Involves:
• - writing an Nginx config file config/nginx/nginx.conf with static route to the web app and routes to the Flask services
• - configuring Nginx to use HTTPS
• - adding a stanza for the web server to docker-compose.yml
• - adding a target to Makefile to build the web server image
• - adding a target to Makefile to build the web application as a docker volume that the web server will use to serve static html/css/images/etc

We also need to pick a Docker image to use for this task. Something like this: nginx - Docker Hub seems like a good place to start

Do not parse charges on Record parser

Context

The record parser extracts the cases from the search results page. It also extracts the charges associated to the case, but this is not necessary since the charges can be extracted from the case details page in more detail.

Tasks

• Update tests to check the parser skips the charge column.
• Remove the sections responsible for parsing the charge columns for each case.
• Ensure all existing functionality does not change.

Build Icon Component

Create a reusable icon component in ReactJS.

We are no longer using React, so we'd need a new way to insert inline svg.

This is low priority as we can use font icons for now, but inline svg is ideal.

Here's relevant article: https://css-tricks.com/pretty-good-svg-icon-system/

Endpoint: /api/stats

Endpoint for reporting on system statistics such as number of searches done, number of records expunged, and anything else we think is interesting

Create A Footer Component

Build a footer component that provides a copyright and terms link.

Account HTML template

Create view in HTML / CSS for the Account view.

This is very similar to the "Add user" view.

Wireframes: https://marvelapp.com/644d2f9/screen/50214083
Design system: http://dev.huntermarcks.net/design-system

Will probably want to build in the prototype for now. You can fork this repo: https://github.com/hmarcks/expunge-pdx-frontend

Epic: Search if a person has eligible expungeable records

As a user
I would like to enter in a search name and/or date of birth.
so that I can see a list of eligible and non-eligible expungeable records.

Parent of #6, Parent of #7, Parent of #9, Parent of #10

Fix errors in auth table

The auth table has an extra comma and is missing a semicolon.

refactor case parser

The case parser could be refactored to a state design pattern.

type eligibility list interface

I am currently hard coding a list of crimes that are ineligible for expungement (List A, List B, Etc.), however I get the feeling that maybe this list should be stored in the database and have some sort of interface so the public defenders office can view it and/or modify it as necessary.

I imagine that these lists don't change very often, but i'd probably want this feature if i was them.

Class for connecting to database

Write a Python class for connecting to a Postgres database. Constructor should take host, port, name, username, and password and should expose connection and cursor objects. Usage should be:

(in context of HTTP request, such as a GET or POST handler):

import database
g.database = database.Database(db_host, db_port, db_name, db_username, db_password)
g.cursor.execute("""SELECT username FROM users;""")
rows = g.cursor.fetchall()
print(rows)

Cannot install the itsdangerous dependency

Description

When running pipenv install the following error occurs.
Could not find a version that satisfies the requirement itsdangerous==1.0.0

Possible fix

Update to the latest version

Add main backend logic

This would probably implement the logic when user clicks "Search".

This should invoke the crawler/parser to get the data for given input (client name, dob), then call the analyzer, get the results and store it in database.

statute should be normalized before persisting

Description

Currently the charge class is initialized with a statute with whatever string value is passed to it. The Oregon e-courts website displays statutes in a variety of representations. A statute may be 6, 7, or 8 characters long, and may or may not contain decimal points, parenthesis, and other separators.

Have the charge initializer call a method to normalize the statute so that only alphanumeric characters are being persisted.

This task is done when...

• A charge class is initialized with a statute containing only alphanumeric characters.

Create parser to pull valid params from search page

There are a few params that are sent back in the response when visiting navigating to the search. Need to retrieve those params in order to perform a search.
child of #11

CaseParser is now crashing when parsing case 74 for J.D.

Description

The Case Parser is now crashing on a case it had previously run through without issue.

Setup hosting environment for prototype

We'll need a dev environment for the prototype of this application.

This task is done when...

• identify the hosting environment (AWS, et. al.)
• Setup environment

Add info on setting up the frontend to the README.md

The README.md includes info on setting up and running the backend in development. Here is an excerpt:

Running:

$ make run

will run the Flask app inside a Pipenv virtualenv.

This task is done when:

The README.md includes instructions on setting up the frontend in development (installing npm, npm install, etc.) and running the development server with npm run start

Automate navigation of the site

Child of #8

Endpoint: /api/users/

Endpoint for making and getting users. Endpoint should handle POST and GET requests.

POST /api/users/ to create a user:

{
    "username": "<USERNAME>",
    "email": "<EMAIL>",
    "admin": "<TRUE|FALSE">
}

admin is optional and will default to False

GET /api/users/USER_ID to get a user

The GET response is TBD but will include some set of columns in the users table. It could be:

{
    "user_id": "",
    "username": "",
    "email": "",
    "admin": "",
    "date_created": "",
    "date_modified": ""
}

Update README for installing Python 3.7

Need instructions for Mac and Linux

Automate expunge decision

Build a tool that will analyze each charge and decide if it can be expunged

test issue please ignore

testing

Update database schema w/suggestions from Tim

Tim made some suggestions Jan 2 about our users table. He recommends moving the columns related to authentication to a separate table.

Create a record parser

Build a parser that collects all the data from the search results page on OJCIN that builds a list of cases. Each row in the results page would be a case.

Create crawler

The crawler will accept search terms and login credentials to login & navigate the site using the parsers to scrape payloads required to crawl the site and to extract data from the search results page and each associated case page.

Child of #8

Create Login Form

Build react components that are reusable for building forms and provide form validation configurations.

implement views as react components

Create a case parser

Build a parser that returns a list of charges for a case from the case detail page.

Build React app into webserver container

Build the webapp into a Docker volume used by the webserver container. Write a Make target to control that process

See issue #43

Re-visit the system design doc

The design doc is out-of-sync wrt issues logged since and subsequent discussions. We should sync up:

• endpoint names

• query parameters: what filters are needed?

• POST bodies: what does the JSON sent to the server look like?

• responses: what does the JSON returned by the server look like?

• headers: what headers does the API require?

• status codes: what status codes do endpoints return?

Main login integration

Integrate the UI to log in to the app, reset password, and set password.

User flow:

Template w/ error messages: http://dev.huntermarcks.net/log-in
Design system: http://dev.huntermarcks.net/design-system

Create an "exercise" task to help new folks get up to speed faster

In the apps I've worked on before, we always had an exercise task that newcomers would complete just after joining. This task was designed to make sure newcomers could "learn by doing" by making a set of minimal, full-stack changes to give them something to do while they get familiar with the codebase.

If we had an exercise task like this it would also help with triage when lots of new people join: we could immediately give them something to work on that would teach them about the codebase and help them be more useful when working on features or refactors later on.

This task is done when:

• There is an exercise task that meets the above requirements. This should probably be a page in the wiki that introduces the different parts of the app code and how the work together, and that walks someone through the process of making a simple, full-stack feature change or addition.

Endpoint: registration & login

Write an endpoint to log the user into the internal website. Take the user's credentials, validate against information stored in the database, and return a token.

This endpoint should handle POST, with the body containing username and password in JSON:

{
    "username": "<USERNAME>",
    "passowrd": "<PASSWORD>"
}

See issue #42