anidata/ht-archive

Anidata 1.0: Frontend node service for data collected from palantiri

ht-archive's Issues

Visualize a single entity and related data

User story: As a prosecuting attorney, I should be able to see what data/webpages are related to the entity of interest, so I can explore potential pieces of evidence to help build my case file.

Tasks:

• Brainstorm visualizations for entity resolution data
• Create new page/view for visualization
• Add search functionality to search for entities to visualize

Pull DB details from config instead of command line argument?

Database host URLs are sometimes pretty long and complicated. I think it would make life easier if we stored DB details and other config in a config file in the project instead of requiring the user to pass those details every time they want to run app.js.

Create a REST API

Create a REST API for the database. The endpoints could essentially emulate what the Data Access Objects do

Relevant files:

• routes/index.rs
• Everything in routes/daos

Fix entity search results

Entity search result data seems to disagree with results from manually querying the SQL database.

Tasks:

• Identify source of the bug
• Make the fix

Fix search result display

Currently the "freestyle search" search results links link to the wrong BackPage post. For example, the search result list shows a post with the title "This is a ad", clicking on the link for the result display a post with the title "This is a different ad."

Tasks:

• Identify the source of the bug
• Make the code fix

Migrate wiki setup info to README.md

Copy the useful bits of setup information on the wiki page to README.md so setup instructions have more visibility.

Clean documentation/Markdown files

We need to update documentation and Markdown files (files ending in .md) and remove old information (e.g. references to GitLab instead of GitHub, old labeling systems, etc).

Tasks:

• Update README.md
• Update CONTRIBUTING.md
• Update LICENSE to BSD

Use the REST API for the client side JS

Remove the swig code in the current views directory and instead use Angular, React, or some other cool JS framework to call the API

Dependencies:

• #7

Create a mock test database for prototyping

Create a database that isn't nearly a quarter million rows for prototyping

Role dbadmin doesn't exist by default on Postgres server

Just a quick note about a lack of clarity in the docs: If a user installs and starts a postgres server themselves instead of using the Docker command that creates a user named dbadmin, then running the queries in the Wiki to create and populate the crawler DB as user dbadmin will result in a "role does not exist" error. If they try to execute the commands as user postgres, loading data from the SQL dump into the database throws errors, since crawler.sql specifies that the commands should be executed by a user named dbadmin. We should specify in the Wiki that users who don't use Docker will have to create a superuser named dbadmin for the commands in the wiki to work.

On a related note, to run the commands to create the database from the command line you have to run the commands as user postgres like so $ sudo -u postgres psql -c "CREATE DATABASE foo;". Trying to run the commands as they appear in the wiki psql --username postgres -c "CREATE DATABASE foo" fails because of a peer authentication error. See this SO answer for an explanation of why.

Remove special characters from phone number query

As a user, I should be allowed to enter a phone number in the entity resolution search as space separated (123 456 789), hyphen separated (123-456-789), or without any delimiter at all (123456789). Currently I can't do that, because the phone numbers are stored in the DB with hyphens as separators.

Instead of strictly enforcing the hyphen format, maybe we should strip out the hyphens from the existing phone numbers in the DB. The app can then strip out any non-numeric characters from the user's query on the backend, and the SQL query will still return matches. This will allow for more robust searching that doesn't break if a user doesn't use the xxx-xxx-xxxx phone number format.