devonwalshe/pd

pd's Issues

Application user flow

Review the document:
https://docs.google.com/document/d/1fn-PuLoznVRrOr3IhftslaNwFIJ7ZiQl0ec5QfV4Ndo/edit?usp=sharing
Mockups:
https://docs.google.com/presentation/d/1KmPtraMczloZVm8X3YFYGwrL2GpANjiq6V2j487CLdE/edit?usp=sharing

Key tasks the user needs to accomplish:

1. CRUD resources

• Pipelines
• RunMatches
• InspectionRuns
• FeatureMappings

2. Upload data files, each representing one inspection run, which belong to a pipeline

• upload two files at a time
• which creates new RawFile, InspectionRun and RunMatch records

3. Configure the datamatch with parameters

• feature mapping dropdown
• coordinates True / False
• short joint threshold numeric
• short joint variance numeric
• short joint lookahead numeric
• joint length difference numeric
• backtrack validation lookahead numeric
• feature match threshold float
• metal loss match threshold float`

4. Launch data match - it will take some time to complete - will need to add a status field to the run_match endpoint to indicate progress

5. Match results analysis page

• displaying:
• pipe section count
• matched features
• unmatched features
• revert the match and go back to try new parameters

6. Manual check

• navigate pipe sections
• match features
• mark pipe sections as checked

7. Data output

• Download a csv file for the run match

General notes

• Add a application header to the top left - 'AKD data matching tool'
• Navigation links to the side of the title:
  • Dashboard
  • Upload Datasets
  • Pipelines
  • Run Matches
  • InspectionRuns
  • Feature Mappings

Match runner overview

I've set up the endpoints that will allow the user to upload files and launch a match between the two files, and then output the results of the match to a CSV. I'm going to discuss the endpoints here so we can work together on tying it all up.

Before you read this - please review #2 to refresh yourself on the user flow (we are dropping item 5 for now).

I imagine you'll have questions so let me know and I'll update here and in Slack so we're on the same page.

Short version:

• POST to /pipeline/ - make a new pipeline
• POST to /raw_file/ - uploads files - do this 2 times with additional form data (discussed below)
• POST to /run_match/ - make a new run match with all of its related data
• POST to /matchrunner/<runmatch_id>/ to launch a data match on the files
• GET to /matchrunner/<runmatch_id>/export to return the data as CSV text, which you'll then need to package into a file for the user to download.

More detail:

1. Pipeline

   • This is the top of the food chain for the application, the resources InspectionRun and RunMatch both require a pipeline as their starting reference point.
   • POST to /pipeline/ with a name parameter, thats it.

2. RawFile

   • Input dataset. We need two of these to set up a run match, I think the page where we start a new run match should start with a form where you enter two files, and all the necessary data, but actually send two post requests (we can discuss this more)
   • A POST request to /raw_file/ now requires additional data attributes, sent as a multi-part form:
     • file - file upload,
     • source text input,
     • data_mapping_id - dropdown (endpoint /feature_maps/),
     • pipeline_id - dropdown,
     • run_date - text-input,
     • sheet_name - text input,
     • source- text input
   • You will get the corresponding InspectionRun id (gets created automatically) in the response from the post. I can make it available elsewhere if you need but you could also traverse backwards from GET /inspection_runs/ which lists raw_file_id's

3. InspectionRun

   • An Inspection Run is created automatically (this is important because the RunMatch references the inspection runs directly, not the raw files).

• you need two inspection run ids to create a run match

4. RunMatch

   • The parent for all our match data
   • After we upload 2 RawFiles, it will generate 2 corresponding InspectionRun's that we can use to generate a RunMatch.
   • A POST to RunMatch requires just a name, run_a (earlier InspectionRun), run_b (later InspectionRun), and a pipeline_id
   • A RunMatchConf record is created in the background with my standard defaults, but we should allow the user to tweak the launch of the matcher

5. MatchRunner

   • endpoint at POST /match_runner/<run_match_id> to launch the match for the run
   • I tried to make it run in the background but need more work, DB was causing connection issues.
   • This will be a long running request, matching the data in the background.
   • I've added some narrative info to the GET /run_match/1 endpoint

6. MatchExporter

   • endpoint at GET /matchrunner/<run_match_id>/export
   • We should have an export CSV button for this in two places - on the matching interface and on the run_match listings.
   • Another long running process, assembles all the data from the database for the runmatch and outputs a CSV as text.
   • The user should receive a file download prompt.
   • I figured there wasn't any point sending a file which you would have to handle on your end, so decided on text - let me know if that works

Application feedback 2020-08-17

With reference to: #2

Issues I've identified:

DONE - Matching stopped working entirely
DONE - Skipping sections bugs out often, doesn't update the section title
DONE - Loading spinner seems to always be going
DONE - Remove the upload datasets nav item
DONE - Run Match page should list run matches, with a button for new, which takes to you to the upload datasets page and follows the flow listed in #2
DONE - add comments field to the tooltip
DONE - all plotted items should have no icon - filter based on whether they have length / width rather than feature_category
DONE - Can we reduce the metal loss plot size normalization threshold to 1/2"? It seems like a lot are falling under that would otherwise show some dimensions.
DONE - 'Uncomplete' should read 'Incomplete' and it doesn't seem to save
DONE - Skipping to the end not working (maybe we don't need it with the skip to weld ID?)

• table headers wrapping (fixed set of headers for now)
• features are plotted in reverse to where they should be - they should be plot subtracted from the weld us_weld_dist, so if weld_a has a us_weld_dist at 45ft, and a feature on side A has a us_weld_dist of 3 ft, the feature should be plotted at 42ft on the pipe (no biggy this one is easy to miss, but thats what its like in reality moving left to right)
• double arrows should take you to next incomplete section, not vice versa

Client observations:

DONE - Waiting on rulers
DONE - differentiate the color between pipe section that has data for it vs the areas above the 0/360 margins (I would suggest just only having the pipeline matched and we'll put the rulers above the section with the non mapped features.
DONE - Remove the icon for the plotted Metal Loss / Mill Anomaly features, Anything from 0.01" to 1" size just set it to ~2px squared in the view.

To bump

• Features at the top - with a length but no width should be displayed with there horizontal scaling (I'll try to bump this as well)
• Table headers - need to do something about this - is it hard coded for the moment? It will need to handle arbitrary field sets - not all matches will have the same... I think I'll bump that to the next round of work though if we can just get it wrapping properly
• Need CRUD (Create, Read, Update, Destroy) for Pipelines, RawFiles (uploaded), FeatureMaps, InspectionRun (I'm going to bump this as well, will set up default data - too much interface right now)
  • so new, list, view, and edit on each of those

Manual pairing interface review #1

Hey, got some comments back from the clients - all expected stuff but just writing it down so we have it in mind, plus a couple of my own comments.

DONE 1. Table / Graph display items:

- add a small gutter between side A and side B on the table
- I've added a link icon, rather than an 'x' overlay for unlinked features, place the link over the top right corner?
- make outline box thinner - 1px
- datapoints (both table and on graph, omit the others): 

side (visualisation only)
feature_id - change to 'ID'
feature
feature_category - change to 'category'
orientation_deg - change to 'orientation'
us_weld_dist_wc_ft - change to 'US weld dist'
us_weld_dist_coord_m (I'm going to change this to ft later)
length_in - change to 'length'
width_in - change to 'width
depth_in - change to 'depth'

DONE - 2. Pairing flow:

- open 'pairing mode' button -> a shaded button 'match features' appears
- the first feature you click on (side A or side B) sets which side your matching and highlights it
- shade other features of the first side (can't pair features from the same side)
- click on the feature from the other side -> allow to click / unclick for active / inactive
- click button 'match features' which becomes active when two features are clicked

DONE - 3. Features with dimensions (length, width) [metal loss, mill anomalies, dents]

- scale features based on their dimensions (length / width)
- scale icon to smaller height or width if box is rectangular, keep it centered

DONE - 4. Features (mostly) without dimensions [valves, flanges, fittings, + the rest]

- place icon above the pipe graph
- draw a straight line down to the bottom of the pipe from the bottom centre of the icon

5. Rulers

• react-charts? https://react-charts.tanstack.com/docs/api#axes--scales - Looks pretty basic but might help
• X axis - ft and inches (is there a good axis drawing library in React?)
• Y axis - 0 - 360, intervals of 30

DONE - 6. Unlink features

• need to be able to unlink features, can indicate that with the link / unlink png's at the top and keep the same flow?

DONE - 7. Confirm pipe section has been manually checked

• Once the manual check of the pipe section is complete, add a button to complete that pipe section - update manually_checked to True
• Set up a label at the top - red bg when with 'manual check confirmed, green bg with 'manual check confirmed'

DONE - 8. Weld data section

• Section above the pipe section and below the filters
• Title: Weld Information
• Horizontal list [side: A, weld id: 12321, upstream weld distance: 26.2343, j joint length: 15.51, wall thickness: 0.1] (distinguish between side A and side b)

DONE - 9. Pipe section navigation

• remove run dropdown (the run will be selected in the previous screen) - replace with display run_id (adding a name to the model so we can display that)
• buttons for: next/previous unchecked section, next/previous section
• toggle (default on) for skip sections with no features
• remove pipe section dropdown, replace with 'jump to weld' - add a toggle for side A or side B
  • Jump to weld - enter the weld_id
  • added filters to the welds listing endpoint to filter by weld_id (from user input) and run_match (already there) http://localhost:5000/welds/?weld_id=110&run_match=1