open-data-rescue/climate-data-rescue

Climate Data Rescue is an archival data rescue platform using Ruby on Rails.

DRAW - Data Rescue: Archives and Weather

Contents

• Introduction
• Development Setup
  • Environment Variables
  • Starting and Stopping the application
• Attribution and Acknowlegements

Introduction

This application seeks to aid in the digitization of paper records through the crowd-sourced transcription of scanned image files, particularly documents ill-suited for Optical Character Recognition (OCR). It specializes in recovering data stored in a highly structured format. The app fully supports multilingual localization of the user interface and the data schema organization (field labels, help texts, etc).

Administrators are given an interface with which to define their data collection schemas, and users are provided a mechanism to transcribe data contained on the page and indicate its position.

Development Setup

Apart from GIT and whatever IDE/editor you prefer to use the minimum to run a dev env is docker desktop.

Environment Variables

You will need to create a .envrc file. The minimal content should have

export MYSQL_ROOT_PASSWORD=yourpassword
export MYSQL_PASSWORD=yourpassword
export SIDEKIQ_REDIS_URL=redis://localhost:6379/0
export SMTP_PORT=10025
export SMTP_SERVER=127.0.0.1
export HOSTNAME=localhost
export HOSTPORT=3000
export DEVISE_SECRET=
export SECRET_KEY_BASE=

For development set the MYSQL_ROOT_PASSWORD and MYSQL_PASSWORD to the same value will be used by the docker scripts to create an instance of the database and set up the initial database. Also by the Rails database.yml for the connection. In local dev their value is unimportant and can be used as-is.

DEVISE_SECRET, and SECRET_KEY_BASE will need values. You can generate values for them using the following commands:

docker-compose -p draw-dev -f docker-compose-dev.yml run draw bundle install
docker-compose -p draw-dev -f docker-compose-dev.yml run draw bundle exec rake secret

Starting and Stopping the application

The dev docker compose uses external volumes so that we can persist data between runs. These are created using the following:

docker volume create --name=draw-db-data
docker volume create --name=draw-redis-data
docker volume create --name=draw-node_modules
docker volume create --name=draw-node_modules_sidekiq

Then to start the dev docker instances use

docker-compose -p draw-dev -f docker-compose-dev.yml up

or

docker-compose -p draw-dev -f docker-compose-dev.yml up --build -d

The secound will detach the docker processes from your shell so it remains running.

You will be able to access the running server via http://localhost:3000 and a test user will available ([email protected] with password password)

Attribution and Acknowlegements

This application was created in 2015 by Robert Smith as an undergraduate independent study project in the Department of Geography at McGill University with Renee Sieber from the departments of Geography and Environmental Studies as the supervisor. It was developed in coordination with Victoria Slonosky in her efforts to recover data from thousands of pages of historical weather observations.

This project was inspired by the Scribe Framework.

climate-data-rescue's Issues

Ability to edit field options once they have been created

I know you are already planning on doing this, but I'm just adding it to the list so that it doesn't fall through the cracks.

vertical mirror window for time and date

Construct a vertical mirror window, similar to the header window, to keep the date and time in view

Transcription Process - Can draw infinite number of boxes

Issue: I can draw an infinite number of boxes
After transcribing the barometer group, I drew the first box. All good.
Then I reopened the transcription bar for the next annotation. When I tried to move the background image of the ledger sideways, it drew another box...again and again.

[DRAW-010] On ‘Help Guide’ page, there is a missing link where text says ‘click here’

Steps to reproduce:

1. Launch DRAW website.
2. Navigate to the ‘Help Guide’ section http://test.citsci.geog.mcgill.ca/help-guide.
3. Observe missing link where text says ‘click here’.

Actual result:
On ‘Help Guide’ page, there is a missing link where text says ‘click here’.

Expected result:
Text which says ‘click here’ is clickable.

Notes:
Issue occurs on:
Google Chrome Version 55.0.2883.87 m, Internet Explorer 11, Firefox version 37.0.1.
Windows 7 Professional x32, Intel Cote i5, ATI Radeon HD 4550.
Repro rate: 5/5.

Screenshots attached: DRAW-010.png

Tiffany Hirsch

Safari User?

Issue
The 'Safari User' text takes up a lot of space.

Potential Solution
We only put 'Safari User' as a hyperlink to the FAQ question where we explain this.

add timepicker for time field

Minor CSS Adjustments

Home Page
Before

After

Modifications
.body-bg-img { opacity: 1; } p.homepage-intro { padding: 0 20px; text-align: center; }
Note: change the opacity to 100% on all pages.

---

Content Pages
Before

After

Modifications
h1 { color:#fff; } .main-app .main-content-container.default {
background-color: rgba(253,253,253,0.8);
} .main-wrapper { background-color: rgba(0,0,0,0.3); }

---

Admin Pages
Before

After

Modifications
.main-app .main-content-container.default { background-color: rgba(253,253,253,1); min-height: 300px; }

Can scroll up indefinitely

Issue
I can move the header placeholder up infinitely. At some point I no longer see the image. It can get confusing for a beginner user.

Potential Solution
Prevent scrolling up beyond a certain point.

Transcription bar - Displays as 2 rows in certain circumstances

Issue
If the user clicks on one of the field groups that has fewer fields, then moves the transcription bar on the page, and then clicks on one of the field groups that has a greater number of fields, the fields can display on two columns instead of just one. This makes it a bit weird to transcribe.

This does not occur when the user clicks on the field group with more columns before moving the transcription bar.

[DRAW-001] Empty row appears in ‘delete this annotation’ pop-up

Steps to reproduce:

1. Start a new transcription.
2. Enter data for a single row in the transcription box and save.
3. Once data is saved and this row appears in a box, hover curser over the box to prompt edit options.
4. Select the ‘delete annotation’ button.
5. An empty row appears in the ‘delete this annotation’ pop-up.

Recommended result:
No empty rows appear in the ‘delete this annotation’ pop-up.

Actual result:
An empty row appears in ‘delete this annotation’ pop-up. This makes the pop-up look as though it is missing text within this row.

Notes:
Tested on:
Google Chrome Version 55.0.2883.87 m, Internet Explorer 11, Firefox version 37.0.1.
Windows 7 Professional x32, Intel Cote i5, ATI Radeon HD 4550.
Repro rate: 5/5.

Screenshot attached: DRAW-001.png

Tiffany Hirsch

[BROKEN!] 'New Annotation' not working

Issue
The transcription bar does not appear when I click on 'new annotation' on the transcription page.
Also, the page doesn't display the rectangles that I had previously drawn. It looks as though I have never been on that page despite the fact that I have entered 17 annotations (according to the annotation count).

[DRAW-009] Spelling mistake on ‘Help Guide’ page

Steps to reproduce:

1. Launch DRAW website.
2. Navigate to the ‘Help Guide’ section http://test.citsci.geog.mcgill.ca/help-guide.
3. Observe spelling mistake. The word ‘handwritten’ is misspelled as ‘hadnwritten’.

Actual result:
In the ‘HAVING TROUBLE READING’ section of the ‘Help Guide’ page, the word ‘handwritten’ is misspelled as ‘hadnwritten’.

Expected result:
The text ‘hadnwritten’ is spelled as ‘handwritten’.

Notes:
Issue occurs on:
Google Chrome Version 55.0.2883.87 m, Internet Explorer 11, Firefox version 37.0.1.
Windows 7 Professional x32, Intel Cote i5, ATI Radeon HD 4550.
Repro rate: 5/5.

Screenshots attached: DRAW-009.png

Tiffany Hirsch

'Draw a box' - Needs standardization: too much liberty for users

Issue
As it is now, the user has too much liberty with the 'draw a box' tool and could potentially do non-sensical things (as displayed in the attached image).

Potential Solution
Generate boxes that have a pre-determined width and height that the user simply need to drag to the relevant position on the screen.

[DRAW-007] Grammatical issues appear in the transcription pop-up tabs

Steps to reproduce:

1. Navigate to a new transcription.
2. Select ‘transcribe’ button to prompt transcription pop-up.
3. Observe each of the tabs (ex: 'Barometer', 'Clouds', etc.).

Recommended result:
Capitalization and punctuation should appear in the transcription pop-up tabs.

Actual result:
Grammatical issues appear in the transcription pop-up tabs.
For 'Temperature and humidity of the Air', the H in humidity should be capitalized. Also, there is a lack of capitalization on the descriptions presently available (under ‘Temperature and humidity of the Air’ and ‘Rain and Snow’ tabs). As such, these descriptions appear completely in lower case, with no punctuation.

Notes:
Tested on:
Google Chrome Version 55.0.2883.87 m, Internet Explorer 11, Firefox version 37.0.1.
Windows 7 Professional x32, Intel Cote i5, ATI Radeon HD 4550.
Repro rate: 5/5.

Screenshots attached: DRAW-007.png

Tiffany Hirsch

[DRAW-008] Scrollbar overlaps top right of navigation menu when using Internet Explorer

Steps to reproduce:

1. Launch DRAW website using Internet Explorer.
2. Observe the top right of the navigation menu overlapping with the scrollbar.

Actual result:
Scrollbar overlaps the top right of the navigation menu when using Internet Explorer.

Expected results:
The navigation menu is provided with enough room to not overlap the scrollbar.

Notes:
Issue occurs on: Internet Explorer 11.
Issue does not occur on: Google Chrome Version 55.0.2883.87 m, Firefox version 37.0.1.
Tested on: Windows 7 Professional x32, Intel Cote i5, ATI Radeon HD 4550.
Repro rate: 5/5.
Screenshots attached: DRAW-008.png

Tiffany Hirsch

Transcription bar - Flow

Issue
As a user, I find it slightly annoying to have to re-enter the date and time for every 'field group' as well as being interrupted every time I finish one to draw a box around the transcribed observations.
For a standard Type_A1 page, the user has to enter the 'observation time' & draw a box & re-click on the 'new annotation' button a total of 9 times to finish a single line.

Potential solution 1
When the user clicks submit for one of the 'field groups' she is taken to the field group directly to the right. The data is thus only submitted once the entire line has been transcribed. Then the user is asked to draw a box over the entire line.

Pros

• Fewer steps.

Cons

• The user would have to enter the entire line of data before it is saved.
• Complicated to implement from what I understood from Rob.

Potential solution 2
When the user finishes drawing the box, she is automatically taken back to the transcription bar (saves them from having to re-open it and drag it back into place).

Ensure that field option ids aren't being set for the hidden field when selecting default options

[DRAW-005] Data entered in transcription pop-up does not save after clicking and dragging box

Steps to reproduce:

1. Navigate to a new transcription.
2. Select ‘transcribe’ button to begin transcription process.
3. When data entry pop-up prompts, enter in values and select ‘save’.
4. Click and drag box around the data just transcribed.
5. Data entry pop-up prompts a second time. None of the previous data entered is saved. Instead, these sections appear as empty.

Recommended result:
Data should save after submitting the first time and should be appear with its respective box.

Actual result:
Data entered in transcription pop-up does not save after clicking and dragging box. The user is required to submit the data during the second time in order for it to save.

Notes:
Tested on:
Google Chrome Version 55.0.2883.87 m, Internet Explorer 11, Firefox version 37.0.1.
Windows 7 Professional x32, Intel Cote i5, ATI Radeon HD 4550.
Repro rate: 5/5.

Screenshot attached: DRAW-005.png

Tiffany Hirsch

[Layout Issue] Main content is not centered

Issue
The HTML injected in the

is not centered on the page. Moreover, the 'Actions' sidebar gets moved down to the bottom of the page when padding is added inside the "page_content" area.

[DRAW-003] Edit options are not functional when transcription box is too thin

Steps to reproduce:

1. Start a new transcription.
2. Enter row data, sizing the transcription box thinner than the length of the edit option buttons.
3. After the box is created, cover over it to prompt edit option buttons.
4. Click the edit option buttons.

Recommended result:
Edit option buttons should be function even if the length of the box is too thin to display the entire button.
Or
Transcription box sizing is required to be at least the length of the edit option buttons.

Actual result:
Edit options are not functional when transcription box is too thin.

Notes:
Tested on:
Google Chrome Version 55.0.2883.87 m, Internet Explorer 11, Firefox version 37.0.1.
Windows 7 Professional x32, Intel Cote i5, ATI Radeon HD 4550.
Repro rate: 5/5.

Screenshot attached: DRAW-003.png

Tiffany Hirsch

Transcription bar - Reinitialize Select2 fields

Issue
The Select2 fields retain their previous values when the transcription bar is reopened. This is somewhat confusing to the user - makes it seem like it didn't save properly.

Potential Solution
Reinitialize all fields except for date and time.

'Draw a box' - Being explicit about the area to transcribe

Issue
I transcribed a line of data. Pressed save. Then did other things for a while before coming back to the page. When I came back to 'Draw a box...' I no longer remembered what data I had last transcribed.

Potential Solution
Add the name of the 'field group' that was last transcribed in the 'Draw a box...' message.
e.g "Draw a box around the [field group] observations you just transcribed."

'Draw a box' - Being explicit about the area to transcribe

Issue
I transcribed a line of data. Pressed save. Then did other things for a while before coming back to the page. When I came back to 'Draw a box...' I no longer remembered what data I had last transcribed.

Potential Solution
Add the name of the 'field group' that was last transcribed in the 'Draw a box...' message.
e.g "Draw a box around the [field group] observations you just transcribed."

highlight the mirror windows

to better differentiate them from the background image

Cannot create enter values for regular select2 fields

Transcription bar - Can't see 'Observation time' because it gets cropped

Issue
The observation time does not display all at once because it is too big for the field.

Hence, part of it gets cropped, making it impossible to see the entire value. This can lead to more errors (e.g. can't see that I put 'pm' instead of 'am').
Moreover, I find that re-writing the date for every transcription is a pain in the neck.

Potential solution
Turn the 'observation time' field in to a Select2. This would also make it more user friendly.

'Draw a box' - Page behaving as an image when trying to draw the box

Issue
As you can see below, the ledger page behaves as an image when comes the time to draw a box around the transcribed area. I am clicking and dragging, but I am pulling the image instead of making a rectangle.

[Very minor issue] 'Editing Field' page: roll-over text never dissapears

Issue
The rollover text that when hovering on the green '+' signs or the red 'x' signs continuous to display after having clicked it. It can get in the way when the rollover text displays on top of the '+' sign, making it impossible to click it.

_url:_http://test.citsci.geog.mcgill.ca/fields/17/edit
time: ~4:45PM

'Instructions' on transcription page should link to popup tutorial

Issue
Changing pages is no fun. The tutorial should be displayed in a popup on the same page.
Let me know what you need for that Rob.
e.g.

[BROKEN!] 'New Transcription' not working

Issue
When I click on 'New Transcription' from the 'Actions' navigation menu, I end up here instead of on the new transcription page.

Ensure that you can't delete field groups that have annotations

If admin tries to delete a field group that has annotations, pop up a warning and confirm the action, telling them that it will delete all of those annotations if they continue.

Add dependent: :destroy to the relationship of field groups to annotations

[Minor issue] Transcription bar - Stacked field groups

Issue
When I click on the 'Wet Bulb' field, the transcription bar becomes narrower because there are fewer fields in that group. As a result, the field groups get squished together and the 'Rain' and 'Snow' field groups are moved below the first line.

error parsing time on create annotation in firefox (on staging)

22:22:16 INFO: Started POST "/annotations" for 67.70.24.212 at 2016-09-15 22:22:16 -0400
22:22:16 INFO: Processing by AnnotationsController#create as JS
22:22:16 INFO:   Parameters: {"utf8"=>"✓", "filename"=>"1491_A_1a_1874-10-15_1874-10-17_1.jpg", "annotation"=>{"meta"=>{"tran$
22:22:16 DEBUG:   ^[[1m^[[35m (0.3ms)^[[0m  BEGIN
22:22:16 DEBUG: meta: {"transcription_id"=>"67", "page_id"=>"112", "field_group_id"=>"8", "x_tl"=>"", "y_tl"=>"", "width"=>""$
22:22:16 DEBUG: data: {"wind_direction"=>{"page_id"=>"112", "user_id"=>"22", "field_id"=>"19", "data_type"=>"string", "select$
22:22:16 ERROR: undefined method `+' for nil:NilClass
22:22:16 ERROR: ["/opt/www/climate_test/releases/20160915211331/app/controllers/annotations_controller.rb:67:in `block in cre$
22:22:16 DEBUG:   ^[[1m^[[36m (0.3ms)^[[0m  ^[[1mCOMMIT^[[0m
22:22:16 INFO:   Rendered annotations/_annotation.json.jbuilder (1.6ms)
22:22:16 INFO:   Rendered annotations/create.json.jbuilder (2.5ms)
22:22:16 INFO: Completed 500 Internal Server Error in 11ms (ActiveRecord: 0.6ms)
22:22:16 FATAL:
ActionView::Template::Error (undefined method `split' for nil:NilClass):
    1:
    2:     json.id annotation.id
    3:     json.date annotation.date_time_id.split("_")[0]
    4:     json.time annotation.date_time_id.split("_")[1]
    5:     json.width annotation.width
    6:     json.height annotation.height
  app/views/annotations/_annotation.json.jbuilder:3:in `_app_views_annotations__annotation_json_jbuilder___141037657427533679$
  app/views/annotations/create.json.jbuilder:1:in `_app_views_annotations_create_json_jbuilder___668766090457861190_46414420'
  app/controllers/annotations_controller.rb:90:in `create'

'Draw a box' - "Annotation could not be saved | Bad request ! Error"

Issue
After submitting a transcription for the barometer group, I went straight to 'new annotation' without having drawn a box around the transcribed area. Then when I submitted the line of observations I got the following error.

Potential Solution
Could we disable the 'new annotation' button until the user has drawn the box?

[very minor issue] Field group colour automatically reverts to purple when editing

Issue
Whenever I edit a field group, it's display colour reverts back to purple by default. Hence, if we make changes to a field group but forget to change it's colour back to what it was, the transcription bar will end up all purple.

Example:
Try editing 'Wet Bulb'. Click save without changing anything. It's colour on the transcription bar will no longer be green; it'll be purple.

Can't delete content page

Issue
Cannot delete content page.
Time: 12:22PM
URL: http://test.citsci.geog.mcgill.ca/static_pages/8

ID returned null

see also screen capture

Type_A2 page appeared after clicking on 'New transcription'

I don't think we should be getting Type_A2 pages at the moment right?

scrolling across page with Chrome and mouse

scrolling across the page in a horizontal direction using a mouse and chrome on Windows. The header moves, not the background image.

adding help text to field group

Monday May 30 @6:47 pm. Tried to add help text to field group.

Field Options: move 'Illegible' and 'Empty' at the bottom of the list

It would be more intuitive to have 'Illegible' and 'Empty' at the bottom of the options list rather than at the very top.

'Draw a box' - Popup modal

Issue
The first time(s) that I used the transcription bar, I did not notice that I had to Draw a box before moving on to the next transcription - I did not notice the message at the top of the screen.

Potential solution
Have a popup modal show every time the data is submitted. Also, disable the 'new transcription' button until the box is drawn.
Something like this:

[DRAW-002] ‘Transcribe’ button should appear as ‘Continue Transcribing’ on ‘My Transcriptions’ page

Steps to reproduce:

1. Complete the transcription of a row and save.
2. Navigate to another section of the website.
3. Navigate to ‘My Transcriptions’ by selecting the ‘Transcribe’ tab, then ‘My Transcriptions’ on the drop-down.
4. Observe ‘Transcribe’ button.

Recommended result:
‘Transcribe’ button should appear as ‘Continue Transcribing’ on ‘My Transcriptions’ page.

Actual result:
The button to continue previous transcribes appears as ‘Transcribe’.

Notes:
Tested on:
Google Chrome Version 55.0.2883.87 m, Internet Explorer 11, Firefox version 37.0.1.
Windows 7 Professional x32, Intel Cote i5, ATI Radeon HD 4550.
Repro rate: 5/5.

Screenshot attached: DRAW-002.png

Tiffany Hirsch

[DRAW-012] User can input invalid time prior to submitting feedback

Steps to reproduce:

1. Launch DRAW website.
2. Navigate to the ‘User Feedback’ page http://test.citsci.geog.mcgill.ca/feedback.
3. Input an invalid time in the ‘What time did the issue occur?’ section (ex: 91:91 AM).

Actual result:
User can input invalid time. If the minutes entered is over 59 (ex: 91:91 AM) then this time is declined prior to submitting feedback and an invalid time error notification appears. However, if the minutes section is 59 or less (ex: 91:58 AM) it is able to be submitted, whether the hour section is valid or not.

Expected result:
Invalid time notification should appear immediately after inputting incorrect numbers, blocking the user from entering this data.

Notes:
Issue occurs on:
Google Chrome Version 55.0.2883.87 m, Internet Explorer 11, Firefox version 37.0.1.
Windows 7 Professional x32, Intel Cote i5, ATI Radeon HD 4550.
Repro rate: 5/5.
Screenshots attached: DRAW-012.png

Tiffany Hirsch

[DRAW-004] Buttons on the ‘Start a New Transcription’ page do not properly align

Steps to reproduce:

1. Navigate to the ‘Start a New Transcription’ page http://test.citsci.geog.mcgill.ca/transcriptions/new.
2. Observe buttons not aligning properly.

Recommended result:
Align the two buttons horizontally rather than stacking them vertically.
However, if wishing to keep them stacked vertically, align both buttons to the left side.

Actual result:
Buttons on the ‘Start a New Transcription’ page do not properly align.

Notes:
Tested on:
Google Chrome Version 55.0.2883.87 m, Internet Explorer 11, Firefox version 37.0.1.
Windows 7 Professional x32, Intel Cote i5, ATI Radeon HD 4550.
Repro rate: 5/5.

Tiffany Hirsch

Screenshot attached: DRAW-004.png

[DRAW-011] Replace background photo of Roddick Gates with McGill Observatory

Steps to reproduce:

1. Launch DRAW website http://test.citsci.geog.mcgill.ca/.
2. Observe background photo.

Actual result:
Website background is a photo of Roddick Gates.

Expected result:
Replace background photo of Roddick Gates with McGill Observatory.

Notes:
Issue occurs on:
Google Chrome Version 55.0.2883.87 m, Internet Explorer 11, Firefox version 37.0.1.
Windows 7 Professional x32, Intel Cote i5, ATI Radeon HD 4550.
Repro rate: 5/5.

Screenshots attached: DRAW-011.png

Tiffany Hirsch

[DRAW-006] A few tabs in the transcription entry pop-up are lacking descriptions

Steps to reproduce:

1. Navigate to a new transcription.
2. Select ‘transcribe’ button to prompt transcription pop-up.
3. Select all of the tabs (ex: 'Barometer', 'Clouds', etc.) to view their respective descriptions.

Recommended result:
All tabs should contain descriptions.

Actual result:
The tabs for ‘Barometer’, ‘Clouds’, and ‘Wind’ are all lacking descriptions.
However, descriptions are present under the tabs ‘Temperature and humidity of the Air’ and ‘Rain and Snow’. As such, the appearance of descriptions is inconsistent.

Notes:
Tested on:
Google Chrome Version 55.0.2883.87 m, Internet Explorer 11, Firefox version 37.0.1.
Windows 7 Professional x32, Intel Cote i5, ATI Radeon HD 4550.
Repro rate: 5/5.

Screenshots attached: DRAW-006_01.png and DRAW-006_02.png

Tiffany Hirsch

Add "Transcribe" button to Nav-Bar

Issue
Now that the side-bar is gone, it will be less straightforward for new users to start a new transcription.
Thus, it may be desirable to add a "Transcribe" button to the nav-bar, with two sub-menu items: "New transcription" and "My transcriptions".

'Draw a box' - Unable to edit the box after clicking.

Issue
I often make mistakes when drawing the box.
It would be great to have the ability to edit it.

Annotation box marked completed annotation doesn't allow tracking in web browser to bound the complete page area

When completing a transcription for an entire row go the LHS of the page, the row is larger than the browser window can display. When drawing the boundary box, the button can't go beyond the browser window and fails to draw the bounding box. Might need some interactivity in case of error and multiple attempts are need to draw the bounding box of the annotation.

from vicky