msfreach/msf-reach

Web platform for MSF-REACH

msf-reach

Data server and web content for CogniCity MSF, built with NodeJS to provide JSON APIs to Postgres objects and accompanying web pages.

Dependencies

• see package.json, or run

$ npm install

Local environment

For running locally, make sure you have a .env file with any overrides for src/config.js, and also update public/resources/js/events.js#L9 to your localhost (and port number as per config). Use of cognito for authentication is disabled by default to make local testing easier (but enabled for our hosted environment).

Environment variables

Login

If using Cognito: Set AWS_COGNITO_ALGORITHM to 'RS256', and AWS_COGNITO_PEM to the public key incl. line breaks—on AWS ElasticBeanstalk where line breaks are not supported in environment variables you need to replace new lines with commas when entering the environment variable, which will be converted back to line breaks in the code.

If using Active Directory: Set the AZURE_AD_* variables with values supplied by AD operator.

Either way, set AUTH=true to force users to login (otherwise if false it will bypass login for local test and development)

Other environment variables:

• ARCGIS_TOKEN: obtain from https://arcgis.cartong.org/arcgis/tokens/generateToken by using username, password, setting expiry for one year, and referrer https://(environment.)msf-reach.org/
• APP_NAME: leave this as default 'msf-reach'. Only used for logging.
• API_KEY: set this to a randomly generated string and then also set for chatbot to make its calls.
• VIZALYTICS_API_KEY: as provided by Vizalytics
• AWS_S3_REGION: s3 region for upload files bucket
• AWS_S3_BUCKETNAME: bucket name for upload files bucket
• BODY_LIMIT: defaults to 100kb for upload post/put body data (but not file uploads)
• CACHE: true / false to use an internal http cache
• COMPRESS: true / false for gzip compression
• CORS: leave this one out to default to true for CORS headers (required for https and neither here nor there for http)
• CORS_HEADERS: actual CORS headers (leave this one out to use default)
• DEFAULT_EVENT_SEARCH_DISTANCE: distance in m (default 1000000m) to look for events to check for duplicates on entry
• GOOGLE_API_KEY: set to server side Google API key for geocoding etc.
• PG*: Postgres connection details
• LOG*: Logging options. Best to leave this out to use defaults.
• PORT: TCP port to run on (omit to use default of 8001 for Elastic Beanstalk)
• BASE_URL: base URL eg. https://msf-reach.org/ , if ommited defaults to 'http://localhost:8001/',
• REDIRECT_HTTP: if running behind https proxy / load balancer, redirect http -> https for incoming requests to load balancer / proxy.
• SESSION_SECRET: set this to a random string.
• SMTP_USER/PASS: AWS SES SMTP credentials.
• TWILIO_AUTH_TOKEN: auth token for twilio for sms webhook (in testing)
• TWITTER_*: twitter app credentials.

Build and Run

Code is written in ES2015 and compiled with babel to a dist folder. To build and run locally:

npm run start

To build for deployment:

npm run build

Database

The database schema can be found in msf-reach-schema.

Testing

Integration tests run using unit.js and mocha. ESLint is used for formatting. For more information see doc/TESTING.md

To run tests, do:

$ npm test

Testing is run on travis-ci.org.

Data Formats

• GeoJSON
• TopoJSON

Documentation

Further documentation can be found in doc/ including:

• API.md - information on API
• TESTING.md - information on testing

msf-reach's Issues

map starts zoomed out on report cards

can you fix with leaflet calls please

make report card bigger on desktop

ideally make it dynamically adjust

cc @LucieGueuning

searchable maps (with google autocomplete)

@matthewberryman I can do a simple autocomplete (+geocoding) if we get a google API key ...

Client: event page should zoom to extents

• generate extents from current reports in event
• currently just zooming to event epicenter with fixed zoom level

Add integration tests for endpoint

Add CRUD support for events endpoint

VERY IMPORTANT - events creation

Natural disaster (earthquake/volcano/typhon/flood/tsunami/drought/others) / Armed Conflict / Disease outbreak (Cholera/Ebola/Dengue/Malaria/Measles/Meningococcal Meningitis/Yellow Fever/others) / Displacement / Malnutrition / Search and Rescue / others + specify

cc @LucieGueuning

new reports, contacts, events should open in new tab

No http -> https redirection

• Can re-use code from cognicity-server (v2)
• ATM have left port 80 off on the server. Will enable once this issue is fixed.

Express API cache is not functioning?

The endpoints /hazards relies on caching to reduce server load.

I have a hunch that the cache was not operating because when I run server locally I get:
[apicache] bypass detected, skipping cache.

I added a timestamp to this endpoint to verify. The timestamp updates with every call, which according to apicache docs it shouldn't

Possibly due to this:
https://github.com/kwhitley/apicache#user-content-client-side-bypass

Rename to MSF REACH

• not a CogniCity project, as agreed with @matthewberryman
  Will need to update
• repo name
• documentation
• testfile name
• schema database name

map on contacts card now too big with addition of nationality pickers

PDC icon?

If @anexact and @LucieGueuning can supply one I can put it in.

can't select some map styles on report card due to attribution

cc @LucieGueuning

Add CRUD reports endpoint

• nest under events

date and nationality pickers on contact card can't be seen - text is white

refer jquery-ui for datepicker css
refer https://github.com/usergroupcoop/country-select-js for nationality selector css

report card last page should have button for new report

first login with new user fails

Rather than a prompt to login and change password, I get
InvalidPasswordException: password does not conform to policy: Password not long enough
Yet the initial password I have set is long enough and otherwise conforms to the policy I've set (and I tried temporarily relaxing it). It's not an incorrect password (that correctly throws up an incorrect password error).

bring web content in

Run with a monolithic app (API + static content) for now

submit button broken on contacts page

sorry I broke this with updates, @mehrdadgit please fix.
cc @anexact

Log in Page

1. @LucieGueuning to supply high res image for use.
2. @mehrdadgit to replace login page image with that one, and set opacity of that image (try 70% to start with).
3. @LucieGueuning and team to then review and @mehrdadgit to tweak opacity if required.

Document which endpoints and methods are private and which are public

minor cosmetic bug on new report card

items shift as selection rectangle is added

need list of reports per event

split social media on contacts card onto new page

contacts and mission histories

With a view to keeping data returned by API and represented on map small, do this on landing page, and

1. @talltom to work on API (done in https://github.com/usergroupcoop/msf-reach/pull/34 ) incl. restrict data returned (for mission histories and contacts) by radius around event , and
2. @mehrdadgit to work on putting that data on map.

check report card wording

https://docs.google.com/spreadsheets/d/1fGuGeet2TKD6Pxy8xm1jHEn1ujxZMWPIqeIUSV7VmCQ/edit#gid=203260487

enforce only one tag / report

@mehrdadgit please wait until I have done a commit as I have done some of the work.

cc @LucieGueuning

events need a name

as far as I can see the backend is ready https://github.com/usergroupcoop/msf-reach-schema/blob/master/schema.sql#L22
just need to update UI

cc @LucieGueuning

reports on map need info displayed

cc @LucieGueuning

database has stale data

1. From local testing where I forgot to update WEB_HOST variable in https://github.com/usergroupcoop/msf-reach/blob/master/public/resources/js/events.js#L9
   which is then causing https://github.com/usergroupcoop/msf-reach/issues/16 (the "An error occurred" bug). cc @talltom @mehrdadgit for local testing. @talltom can we update the build script to change this? if so open another ticket for that. I will close this one once database is started with a fresh copy.
2. From change of event types, which is why old event types are showing up.

cc @LucieGueuning

PDC layer

@talltom I see you've just put in usergroupcoop@e21ff08

• if after testing you're happy then please close. Just adding this in to github for tracking.

reports details

What do we need here exactly?
@talltom to comment on what's required, @mehrdadgit to complete.

twitter feed search should also work if enter is pressed

ATM requires user to click on search button.

cc @LucieGueuning

report card submission throws random errors

Sometimes I get an image upload error (initially due to an IAM error, but I resolved that, and now sometimes randomly an image upload error)

Sometimes on submitting a report for an event:
error: Error: Invalid report_key for this event.
at Connection.parseE (/var/app/current/node_modules/pg/lib/connection.js:567:11)
at Connection.parseMessage (/var/app/current/node_modules/pg/lib/connection.js:391:17)
at TLSSocket. (/var/app/current/node_modules/pg/lib/connection.js:129:22)
at emitOne (events.js:115:13)
at TLSSocket.emit (events.js:210:7)
at addChunk (_stream_readable.js:252:12)
at readableAddChunk (_stream_readable.js:239:11)
at TLSSocket.Readable.push (_stream_readable.js:197:10)
at TLSWrap.onread (net.js:589:20)

pre-fill and pre-load Twitter search

take event name and summary words from events API and pre-fill Twitter search keywords on events page, and run search automatically on those, allowing user to modify and re-run search

unauthenticated path access returns error message

Should redirect to login page.

TypeError: Cannot read property 'split' of undefined
    at Object.fromHeader [as getToken] (/var/app/current/dist/lib/util.js:46:37)
    at middleware (/var/app/current/node_modules/express-jwt/lib/index.js:50:25)
    at Layer.handle [as handle_request] (/var/app/current/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/var/app/current/node_modules/express/lib/router/index.js:317:13)
    at /var/app/current/node_modules/express/lib/router/index.js:284:7
    at Function.process_params (/var/app/current/node_modules/express/lib/router/index.js:335:12)
    at next (/var/app/current/node_modules/express/lib/router/index.js:275:10)
    at jsonParser (/var/app/current/node_modules/body-parser/lib/types/json.js:103:7)
    at Layer.handle [as handle_request] (/var/app/current/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/var/app/current/node_modules/express/lib/router/index.js:317:13)
    at /var/app/current/node_modules/express/lib/router/index.js:284:7
    at Function.process_params (/var/app/current/node_modules/express/lib/router/index.js:335:12)
    at next (/var/app/current/node_modules/express/lib/router/index.js:275:10)
    at logger (/var/app/current/node_modules/morgan/index.js:144:5)
    at Layer.handle [as handle_request] (/var/app/current/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/var/app/current/node_modules/express/lib/router/index.js:317:13)

public key handling on Elastic Beanstalk

UnauthorizedError: PEM_read_bio_PUBKEY failed
    at /var/app/current/node_modules/express-jwt/lib/index.js:102:22
    at /var/app/current/node_modules/jsonwebtoken/verify.js:27:18
    at _combinedTickCallback (internal/process/next_tick.js:67:7)
    at process._tickCallback (internal/process/next_tick.js:98:9)

because rfc1421 mandates line breaks but Elastic Beanstalk environment variables don't support line breaks, see ThoughtWorksStudios/eb_deployer#64
The simplest way is probably to do a replace on ',' -> '\n' as suggested in ThoughtWorksStudios/eb_deployer#64 (comment) which means that if you run the code locally (with line breaks) it will still work, and avoids having to find the end of the dashes, etc.

select past missions and contacts for linking with events

contacts cards

1. Tom to extend contacts API to allow submission of data,
2. Mehrdad to work on contacts API.

Refer to existing contacts file I sent via email for fields for new submissions for now.

report card check for missing location

Check happens on last page and dumps the user back to a black report card page.
Should happen earlier on the location entry slide of the report card, and allow user to input it.

cc @talltom @LucieGueuning

Twitter search

Twitter Search function glitches in several ways:

1. no way to refresh the search, thus new information in Twitter, especially from peers, cannot be detected and thus cannot receive Reach-Out report card; if the search cannot be refreshed, it does not allow the operator to pick up latest info from peers who are requesting report cards;
2. seems to be a bit of trouble with the search finding relevant tweets, included those with precisely matching terms to the search bar; I'm not able to determine why or how it is missing them, but it finds some terms and not others—public API issue maybe;
3. when using refresh on the page in Chrome, it just takes you back to the Event Tab, but doesn't reload the tweets.

fields on event archive

Archive button should pop up a field for entry of further details on event completion
https://docs.google.com/spreadsheets/d/1fGuGeet2TKD6Pxy8xm1jHEn1ujxZMWPIqeIUSV7VmCQ/edit#gid=203260487

report cards require image

Should make this optional I guess

event icons based on event type, on landing page map

https://drive.google.com/drive/folders/0Bw27FPU-gFxYRkQ0aFZOaFAtVDg

Determine strategy to rotate event UUIDs

• need to log changes in a separate table

no option to select none for 2nd nationality

put name of event on report card

Change
https://github.com/usergroupcoop/msf-reach/blob/master/public/report/index.html#L36
to read "MSF would like to request any information you may have about an event, " + name_of_event_this_report_is_associated+ ", that may be happening in your area."

travis CI lint fails

https://travis-ci.org/usergroupcoop/msf-reach/builds/262485584?utm_source=email&utm_medium=notification

Not affecting function in #9 fix, just linter is being picky on my copy and paste.

display extra event fields and contact fields in tables, maps

1. @anexact please list for us what needs to be displayed for events and contacts on map and in tables on events page.
2. @mehrdadgit can you please have a look at this if you get time, otherwise @talltom to work on this tonight our time, or failing that I can have a quick look tomorrow.

cc @LucieGueuning