nbisweden/encam

Encyclopedia of Cancer Microenvironment

encam

Encyclopedia of Cancer Microenvironment http://encima.one

Local development

docker-compose up --build

This starts a flask server serving a static version of the webpage using nginx on http://localhost:8080 using the development server (proxied via nginx).

Building and deploying

Make a new access token on https://github.com/settings/tokens/new It needs these permissions:

• repo
• write:packages
• read:packages

Set up github credentials on docker using the token as password:

docker login docker.pkg.github.com --username danr

Run the deploy script which will build the docker image, upload it as a github package and then tell our running SNIC machine to pull it and restart:

VERSION=0.0.12 ./deploy.sh

The server needs a token to pull the image. The permissions need to be:

• read:packages

Login on the server as well:

docker login docker.pkg.github.com --username danr

Moving to a new server

1. On the old server: copy the config, certbot and snap directories in $HOME.

2. On the new server:

   1. Install docker.
   2. Login with github API keys as above to be able to pull the docker image.
   3. Create a volume and attach it to the running instance.
   4. Mount the volume to the config directory of the instance. On SNIC this looks like:

   sudo mkfs.ext4 /dev/vdb
   mkdir -p ~/config
   sudo mount /dev/vdb ~/config
   

   Detailed instructions about mounting a volume: https://github.com/naturalis/openstack-docs/wiki/Howto:-Creating-and-using-Volumes-on-a-Linux-instance

   5. Put the old server's directories config, certbot and snap in $HOME.
   6. Make sure permissions on config/content are liberal:

   chmod 777 config/content
   chmod 666 config/content/*
   

3. Locally: update the deploy.sh script to point to the new server IP address.

4. Locally: run the deploy.sh script.

5. Update the DNS on .one to point to the new IP address.

Running backend tests

To run the tests for the encam project

python test.py

in the api folder. The expected output is no assertion errors.

This can be run in a virtualenv or with

docker-compose run api python test.py

To get the code coverage, install the coverage package with pip and run

coverage run test.py

To get the report from the code coverage

coverage report -m

and the annotated HTML listings detailing missed lines

coverage html

Running frontend tests

This can be done in the frontend directory with

yarn install
yarn run test

This can also be executed in the container with:

docker-compose run -w /app encam-frontend-devel yarn run test

encam's Issues

Code review backend filtering

Also mentioned in #38

Form: Unfolding menu for specific characteristics

 Now, in made up dataset we have location and morph type specified only for COAD and READ, but in reality it will be for other cancers also… Can it be somehow condensed in some kind of unfolding menu?

Frontend testing

• Test backend filtering
• Frontend testing
• End-to-end/integration testing
• Code review backend filtering
• Code review of frontend. Possible reviewer: Martin

Frontend for filtering form and box plots

• Create form for filtering of 2nd page (box plots)
• Draw the box plots
• Enable having either cell types or tumor types facets on the x-axis

[Backend] Run tests as github action

Cell type list graphs - Advanced

As a user
I want to select two cell types from the list
So that I can see two informational graphs about them

• Bar for each tumor location, showing the cell types abundance
• Survival forest for each of the tumor locations (one line/row for each tumor location)

Epic: #8
Additional to #9 the graphs should contain data from two different cell types

About section

As a user
I want to have an about page
So that I can get information about the research group and the usage of the site

Cell type list graphs - Basic

As a user
I want to select a cell type from the list
So that I can see two informational graphs about it

• Bar for each tumor location, showing the cell type abundance
• Survival forest for each of the tumor locations (one line/row for each tumor location)

Epic: #8

Update data set

As an administrator
I want to be able to update the data set
So that I can have the most recent patients

Admin tasks

• Apply for project in SNIC cloud
• Set up servers

Epic: #16

CSS overhaul + code reviewing

Cell type list graphs - Expert

As a user
I want to select multiple cell types from the list
So that I can see two informational graphs about them

• Bar for each tumor location, showing the cell types abundance
• Survival forest for each of the tumor locations (one line/row for each tumor location)

Epic: #8
Additional to #9 and #10 the graphs should contain data from multiple different cell types

Initialize backend

current endpoints:

    /db.json
        Pre-calculated survival analysis for the first/front page
    
    /codes.json
        Tumor type codes -> Tumor type acronym expanded
        
new endpoints:

    /form_configuration.json
        The different fields that should be in the form +
        the specifics like COAD's anatomical locations and so on
        
    /filter_database + query describing the filter
        returns the rows matching the filter
        then the frontend calculates the CI + error bounds and draws the plot
        (if this is too inefficient calculate this in the backend)
        
        pass the query as json object in the body (POST)

Refactor plotting code

It appears to be viable to do all plots using vega-lite, this is being investigated in the branch vega-lite branch. This would remove the hairy d3 soup that makes the plots right now.

KMPlot: Cumulative expression plot with group size rulers

[Backend] Kaplan-Meier groups

Create the groups for Kaplan-Meier plots based on user's input

Tumor location from body pictures

Body and tumor location graphs

As a user
I want to select a tumor location
So that I can see informational graphs about the cell types

Body and tumor location graphs - Advanced

As a user
I want to select two or three tumor locations
So that I can see two informational graphs about the cell types.

Epic: #2
Additional to #3 the graphs should contain data from all different locations

Cell type list graphs

As a user
I want to select a cell type
So that I can see information and graphs about this cell type

Put cells (or tumors) vertically to be able to deselect them to make the graph narrower

Upload this for feedback and input

Set up Python environment

As a developer
I want to set up my Python environment
So that I can develop my project

Epic: #16

Create project and install work environment

• Apply for SNIC project
• Initiate servers
• Install docker and docker-compose

Related epic: #16

Forest plots exploration in R, python and javascript

Dimitrios & Dan:

• understand some of the statistics

Dimitrios:

• understand R code and
• port it to python

Dan

• use some example data and experiment with drawing the graphs in python instead
• see what data is inside the lifelines objects
• try to send that data to javascript
• see if similar/same plots can be done in javascript

(Enables assessing how much backend processing will needed)

Related epics #2 #8 #16

https://lifelines.readthedocs.io/en/latest/Survival%20Regression.html

Set up Javascript environment

As a developer
I want to set up my Javascript environment
So that I can develop my project

Epic: #16

Push-button deploy

A quick solution to put new version of the page live, eg via dockerhub or github packages.

System administration

As a developer
I want to run the system in the cloud
So that it is easy to maintain

Use new simulated database

From last sprint:

• Change data to the new simulated database
• Use abbreviated tumor names and with hover tooltip info

Body and tumor location graphs - Basic

As a user
I want to select a tumor location
So that I can see two informational graphs about the cell types.

• Bar for each cell type, showing its abundance
• Survival forest for each of the cells (one line/row for each cell type)

Epic: #2

Deploy script to build the frontend

Deploy script to build the frontend.

Idea: let flask be the server for the compiled static page (in addition to serving the dynamic api endpoints). It would be nice if the flask dockerfile can be reused for development and deployed serving.

Move form configuration tumors out of variant_values

It would make more sense to return this in an own field in the configuration (tumors):

https://github.com/NBISweden/encam/blob/feature/backend-init/api/server.py#L89

While we're at it we can change from this old:

interface Conf {
  variant_values: {
    column: string,
    values: string[],
  }[],
  tumor_specific_values: {
    column: string,
    tumor: string,
    values: string[],
  }[],
  cell_types_full: string[],
  cell_types: string[]
}

to this new:

interface Conf {
  variant_values: {
    column: string,
    values: string[],
  }[],
  tumor_specific_values: {
    column: string,
    tumor: string,
    values: string[],
  }[],
  tumors: string[],
  cells: string[], // No point in sending cell_types_full
}

Making an issue because this will need a sync between backend and frontend

Administration and work environment

[Frontend] Draft version of coherent page

Make the page coherent when viewing the splash page and the modules.

Placeholder: receive more module descriptions from Artur

Read and plan work when Artur sends the next module descriptions.

Graphs for cell types

As a user
I want all the cell type graphs to include two bars (stroma and tumor abundance)
So that I get all the available information

Related to: #8 and #2

Cell type list

As a User
I want to see the cell type list
So that I am able to filter based on them

(20 cell types initially)

Layout cell buttons

• Check position and layout to be ok
• Color of bars plots should match colors of drawing
  Idea: pick the colour from the cell images

Set up docker compose to start both backend and frontend

For development and should start the frontend in development mode.

Reference:
https://github.com/NBISweden/chemical-space-web-service/blob/master/docker-compose.yml

Layout human body picture

• Layout using pictures sent from Patrick with actual tumor positions
• Idea: use organs as icons, make body image smaller. Artur sends a sketch.

Clustering

Artur will provide this module's instructions after July.

Mobile version

As a smartphone user
I want to be able to have a mobile version
So that I can access the page quickly through my mobile

Fix .one and URL forwarding

Artur will check the .one settings

Calculate Kaplan-Meier plots and serve them

Backend: calculate the KM (Kaplan-Meier) graphs and serve them

incl. calculation of medianized groups, p-value using log rank test and Cox p-value

Second page analysis

This issue is mostly a placeholder for now: we're waiting for descriptions, code examples and suggestions for python packages for the further analysis that will be on the "second page". (This page might just be accessed by scrolling further down but we can call it "second page" for now.)

Draw Kaplan-Meier plots in the frontend

• adjust the threshold by changing the size of the groups
• Try with 2-4 groups say
  Backend issue: #47

Placeholder: meeting with Artur around May 7 or 8 about next module Box plots by groups

If possible have meeting about the next module for supporting two filters at the same time in the same plot, at the end of next week (May 7 or 8). Depends on Artur's schedule.

Body and tumor locations

As a user
I want to see two bodies
So that I can select a (tumor) location

Draw layout sketches

Add interactivity and make plots look as desired

• Make it possible to select cells and tumor locations
• Make sure that plots and overall page layout looks as desired:
  • Forest plot: color by cell type
  • Forest plot: crop at high values
  • Prepare for human body layout: scatter buttons and graphs in the middle area
  • Move plots that are not related to a body location to the right margin
  • Make vertical bar plots
  • Mock-up the 11 cohorts (and then Artur is sending new data in the beginning of next week)

Related epics #2 #8

Update UI based on new design documents

• Prepare for two column of tumor types with human body picture in the middle (Patrick draws)
• Use the immune cell drawings Patrick provided in pptx
• Change data to the new simulated database
• Use abbreviated tumor names and with hover tooltip info