codeforphilly/chime

COVID-19 Hospital Impact Model for Epidemics

CHIME

The COVID-19 Hospital Impact Model for Epidemics (penn-chime.phl.io)

Background

The CHIME (COVID-19 Hospital Impact Model for Epidemics) Application is designed to assist hospitals and public health officials with understanding hospital capacity needs as they relate to the COVID pandemic. CHIME enables capacity planning by providing estimates of total daily (i.e. new) and running totals of (i.e. census) inpatient hospitalizations, ICU admissions, and patients requiring ventilation. These estimates are generated using a SIR (Susceptible, Infected, Recovered) model, a standard epidemiological modeling technique. Our model has been validated by several epidemiologists including Michael Z. Levy, PhD, Associate Professor of Epidemiology, Department of Biostatistics, Epidemiology and Informatics at the Perelman School of Medicine.

Originally developed in github.com/pennsignals/chime, active development is now at github.com/CodeForPhilly/chime.

Documentation

• User Documentation
• Developer Documentation

Contributing

• Contributing: Overview

Join our Code For Philly project or our Slack workspace in the #covid19-chime-penn channel if you'd like to chat with us. We'd appreciate your feedback.

Thanks

• Created by Predictive Healthcare @ Penn Medicine
• Project management provided by Code for Philly
• Production hosting provided by Linode Kubernetes Engine

chime's Issues

Set up workflow to publish multi-page docs to gh-pages

Create a workflow for publishing a multi-page set of docs to GitHub pages

Create Observability Stack for Monitoring and Logging

Generating the observability stack serves the following purposes:

• Allows for monitoring of both the cluster and application.
• Identify resource contention and scale issues through metrics
• Allows developers to pinpoint errors and surface them for ops alerts

Components to generate:

• The data pipeline which will deliver logs from the environment and application
• The metrics visualization
• Data management layer either on the cluster, or provide the connectivity to another component which will surface.

Technologies and software to consider:

• Elasticsearch
• Grafana
• Prometheus
• Splunk

Normalize COVID-19 data from Seattle (or Washington)

Find the published source and normalize for https://docs.google.com/spreadsheets/d/1P1W6nLPYPoIqKmvKJWq1iUCkTsrPOu7DkOMTmWotJik/edit?usp=sharing

Investigate implementation of GA

Can we implement google analytics on top of streamlit? how can we get visitor data to app?
Does implementing GA tie the repo to one account?

Begin article: "Contributing: Data Analysis"

Edit docs/contributing/data-analysis.md and then open a PR against CodeForPhilly/chime#master

export dash as pdf

Allow users to export dash outputs (graphs and text) as a pdf report.

Hospitalization, ICU, Vent % allow floating points

Similar to hospital market share, these sliders should accept floating point inputs.

Add caching

streamlit supports caching... maybe this would be a good idea to speed things up?

Streamlit caching docs

Toggle linear/log scaling of y-axes on plots

@quinn-dougherty added a bool flag to functions that look like def some_chart(*args) -> alt.Chart:. if logscale is flagged, do something like .encode(y=alt.Y("col", scale=alt.Scale("log")
... read the docs for specific syntax.

Create Penn Medicine Branding

Insert white label mast on a webpage. https://www1.pennmedicine.org/includes_shared/penn-medicine-header.html

With Mast header "Penn Medicine - COVID-19 Hospital Impact Model for Epidemics"
Remove the website title "COVID-19 Hospital Impact Model for Epidemics" since it will appear in the header.

A quick fix can also be to include an image in the top of the sidebar.

Implement python workflow in github actions

Add workflow for python testing in actions

Allow user to specify R0 (in lieu of doubling time)

Calculation (extends "more info about this tool"):

We can compute the implicit $R_0$ from our doubling time inputs and our assumed duration of infectiousness $(1/\gamma)$. Since $R_0$ is the number of additional individuals infected by each infected individual when $I << S$, we can compute the growth rate $g$ as a function of $R_0$ and $\gamma$:
$$g = \gamma R_0 $$
From our previous equation for $g$, we can substitute and rearrange to find:
$$R_0 = (2^{1/T_d} - 1)/\gamma$$

So, for our $\gamma$ of 1/14 days, a doubling time of 4 days corresponds to an $R_0$ of 2.65.

k8s: enable ingress-caching of /static/ content

Like in https://medium.com/@vdboor/using-nginx-ingress-as-a-static-cache-91bc27be04a1

Tooltip in graphs should display y-value

Current tooltip only displays the key and x-value (days):

Check that tools used are open source

Implement scancode toolkit on repo to check licenses and stuff
*this makes it easier for open source teams to contribute

Assigned Kris cause he knows how to do this and what it actually is.

site title

gimme dat sweet custom title

@mdbecker @cjbayesian what do you want this called?

Suggestions regarding input variables

Copied from my input in the web pages feedback page this morning:

Dear Colleagues,

as a mathematician working at a Fraunhofer institute for digital
medicine in Bremen, Germany,

I take great interest in prognostic model available
http://penn-chime.phl.io/

I like to suggest the following:

[1) .. (suggested to make this open source, obdiously alredy done)]

2. Modify your web page such that critical data such as the doubling
   time and ICU % / Ventilation % might be entered as float values,

as changing these value from say 1 to 2 % results in vastly different
predictions and the by now still unknown true values
are likely to be non integer bus small numbers.

For example my current estimate form positive tested case in the 3
Germany's independent city-states (Berlin+Hamburg+Bremen)

for the doubling time for example is 2.2 days, for the whole of Germany
it its 2.9 (considering the official data published by the
Robert-Koch-Institute)

May estimates for these doubling times is based on 7* log(cases from
last publishes day) / cases(7 days before) / log(2).

We are at a time were social distancing was just started, thus these
numbers might be close to the natural spread rate, but they are strongly linked to testing, availability, usage and rules applied to who will be tested.

It is to be hoped strongly, that the doubling times will get well above your probably significantly
too high default value of 6 days due to the measures started a few days ago.

I would think 3 day at most 5 day ought to be the default for the doubling time for your web page.
Looking a longer scale worldwide non china data from Johns Hopking on a log scale I see a factor 100 increase in diagnosed cases in little more than 30 day, related to a doubling tome of about 4.7.

3. One might also consider entering these two numbers (last days #
   positive tested case / #positively tested cased 7 days before or some other
   time delay) instead of a not well possible direct guess for the doubling time stated
   as integer input in your model.

4. At a later time instead of # positive tested cases the number of
   hospital admission related to COVID 19 at two days a few days apart would probably provide more realistic values, but these are still very few and not well documented publicly in Germany to use the at the current stage.

Best regards

Volker Dicken

Detailed description of each input parameter

In a hidden-by-default section at the bottom, provide a list containing details of each input parameter and how the default value is estimated. This is in lieu of the ability to add this information as a tooltip in streamlit.

Also, LOS should display as Length of Stay for accessibility

document devops structure

What has the ops team built? How does it help the app?

Make documentation for app stakeholders to show what/why we built what we built

unable to connect heroku app to frontend router via new toml files.

Problem: When launching the heroku app without the setup.sh, H11's appear (router backlog) because the app is not being connected on the backend.

Solution: Re add the setup.sh allowing the $PORT to determine the port for the .toml files for streamlit.

admitted patients (census) chart is weekly granular, should be daily

as per discussion in thread pennsignals/chime#58 ,

it used to look like this

now it looks like this

the granularity of the chart should be daily, not weekly, this detail has to be recovered from the matplotlib version.

Automate pushing new container to dockerhub on tag

Add a new GitHub Action workflow that pushes a new container image tagged with :latest and :${version} to DockerHub on tag

Convert 'days from today' into calendar dates

Use the current date as day 0 and plot projections with x-axis as actual date.

Begin article: "Operations: Deploy to Your Own Cluster"

Edit docs/operations/byok8s.md and then open a PR against CodeForPhilly/chime#master

Currently Known Regional Infections parameter confusion

A user asked

When I adjust the "currently known regional infections" parameter, the charts remain unchanged. Is this intended? If so, could someone explain why changing that number wouldn't affect projections?

To which @cjbayesian replied

It's not a bug but rather a consequence of how we're estimating the actual number of infected individuals. Specifically, we're using the number of hospitalizations and hospitalization rate to compute the number likely infected. We only use the number of confirmed cases in the region to infer the detection rate.
You're not the only one confused about this.

User responds:

Here's the issue: currently known regional infections is all we have, which means that our position on the timeline is <0 (well, its <0 for the model that assumes 1 hospitalization)
Acknowledging that the information value of known regional infections is dangerously unreliable as an input, the issue I'm facing is a way to keep the feature in the tool for its palliative effects without diminishing the predictive validity of the output
My (stats-literate) colleague suggested offering a way to use currently known regional infections as a (really bad) estimate of the true proportion of infected people in the population if it could be combined with estimates of the screening rate
Perhaps people need to check a box to use it that gives a popup warning them "do not use this. Its really bad, you're liable to mislead yourself" so that people fixated on using it can play with it

Support Interventions

Given that many governments are enacting policies which effectively change R0, it might be useful to support an "intervention" - a sudden change in R0.

Begin article: "Contributing: Operations Support"

Edit docs/contributing/operations-support.md and then open a PR against CodeForPhilly/chime#master

Implement modules from app.py

Depends on #76 - script breakdown from that documentation implemented in repo

• update any unit tests on functions in github actions

Automate deploying new container to LKE cluster on tag

Following #47, automate updating the instance on our LKE cluster via the same GitHub Action workflow

The tool should not crash when the confirmed cases=0 and/or hospitalization=0

There should be either instruction to use "1" instead of "0", or update the code to use 1 when someone enters 0.

currently the tool crashes.

Dashboard documentation page

• static page for app documentation
• includes description of overall analysis method & individual model parameters

Done: documentation made in markdown format

Investigate modules from app.py

how can we break app.py into modules?

• requirement: put custom set variables in separate scripts so partners can maintain that module themselves while updating the rest of the app from the repo

Done: app.py modules understood and documented, plan for re-factoring exists
Quinn & Sam - wanna make a braintrust?

The data table is un-readable for large numbers

The table should resize to display all numbers OR be exportable.

However, others weren't able to recreate this:

I wasn't able to recreate this issue, even with bigger numbers. I tried it in Chrome on a Mac and Edge on a Windows machine and the columns adjusted to larger numbers appropriately.

`pip install chime`

Related to https://github.com/pennsignals/chime/issues/6

Per this slack message,

i was able to compare the model to the Portugal reality (black datapoints: still not enough data but ok). Im still working on it. What I had to do first was to decouple the code from the web-app integration (it has a few dependencies). maybe the core can transformed into a module ( #chime-app maybe?)

I am going to guess at some desired functionality / API design, please discuss / correct me in the comments:

• Should be able to pip install chime or something similar
• Should be able to from chime import sim_sir and model sir in code (? Not sure)
• should be able to ...

Heroku "staging" instance

Set master branch up against heroku as a POC for orgs that want to make their own instance and to ensure tool still works on heroku

Add model tests to pytest

Implement manual checks done by Penn (@mdbecker will add here) to github actions

Dependent on #59

Should `N` be removed from `sir`?

This was a comment from @luminoso on PR #55 but it is really its own issue...

is there any reason to include N parameter in def sir(y, beta, gamma, N): function @ models.py? Because app.py invokes sim_sir() and it defaults N to N = S + R + I. when sir() is invoked you just compute always that scale = 1

IMO, N should be removed from sir() for three reasons:

1. it doesn't belong to the original formula on the wikipedia
2. in the current implementation it is always computing scale = 1
3. scale can be computed outside sir() for whichever function that needs it in the future

CC @cjbayesian @mdbecker

Scale Ingress Controller

The cluster is currently running only 1 ingress controller, which could become a major bottleneck when production traffic is turned on. We need to pick a higher number and scale up to it.

implement the Scripts To Rule Them All interface

Implement the Scripts To Rule Them All interface for developers:

script/server — Brings a fully working environment up at localhost:[port]
script/setup — Run after script/server to initialize local database with seed data
script/update — Run after changing code or switching branches to refresh server

An example of this implementation can be found at CfP's Prevention Point project

@garysieling has a WIP here- will finish testing later: https://github.com/garysieling/chime/tree/feature/36-scripts-to-rule-them-all

Fix linter errors

When I run this command:

        flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics

I get a bunch of linter errors:

15 E122 continuation line missing indentation or outdented
1 E203 whitespace before ','
1 E225 missing whitespace around operator
5 E261 at least two spaces before inline comment
8 E302 expected 2 blank lines, found 1
3 E303 too many blank lines (2)
6 E305 expected 2 blank lines after class or function definition, found 1
17 E501 line too long (155 > 127 characters)
4 F401 'functools.reduce' imported but unused
107 Q000 Remove bad quotes
1 W391 blank line at end of file
168

The linter doesn't appear to displaying this in the build output right now, so that I consider finding out why it's not failing the build at the same time if possible.

Update viz package

• Replace matplotlib with interactive viz package (altair, plotly, etc) to speed up app delivery and improve interactivity
• replace visualizations with new package code
• Another factor to consider is the amount of space each input (e.g. Currently Known Regional Infections, Currently Hospitalized COVID-19 Patients) takes up. It would be nice to be able to use the screen space more efficiently to minimize scrolling and fit other informative text/data. The current fix is to hide additional text/data behind checkbox toggles.

See pennsignals/chime#58

CC @quinn-dougherty

implement app monitoring framework

Allow developers and product owners to see app activity and usage

Done:

K8 observability implemented on product

• Start a personal Kubernetes cluster - dev-completed don't need
• Add a private Docker registry along with a dashboard - portability
• Add a monitoring tool stack with Prometheus, Alertmanager and Grafana
• Start the 3 microservices on Kubernetes
• Monitor the microservices metrics through Prometheus and Grafana

Add python tests

Make sure python code works and isn't broken by changes

Dependent on #59

Not an expert but here's some tips on writing python tests

Add CI tool for testing

Potential goals, from slack conversation:

1. Goal - make sure the Penn version of this is easy to deploy
2. Goal - make sure this is easy for new places to adopt

Some potential next steps (pulled from slack discussion):

1. Hook up Github actions
2. Document workflow for tagging releases
3. Document workflow to prove site has deployed
4. Script & document this workflow: fork the repo, spin up a commodity k8s cluster anywhere, drop a kubeconfig into GH Actions Secrets, and have an automated deploy-on-tag flow going

Create method for Clinical owner to collect known cases in region

1. A clinical owner of CHIME needs to quickly grab reliable figures to reset the model for a new day zero
2. A clinical owner of the Forecast tool to quickly grab the numbers to assess forecasting accuracy.
   Known cases in Delaware + Chester + Montgomery+Bucks+Philly

Data resides at:
https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_19-covid-Confirmed.csv

External monitoring

It might be valuable to have some external monitoring, e.g. have pingdom notify slack if the site goes down (though perhaps low priority given the attention this already has).

Natural language summary of outputs

In addition to the output plots, it would help to have an automatically takeaway in words. For example:

Peak admissions projected in XX days.
ICU admissions projected to surpass XX in YY days.
...

Create static k8s manifests for scaled deployment

For our first pass at providing an alternative to Heroku, we've spun up a dedicated chime-live Kubernetes cluster on Linode. Currently it has 3 nodes with 6 CPUs each

To close this issue, add a set of static manifests to this repository for a deployment that takes the current Streamlit-based Chime implementation, runs as many workers as possible on that 3x6 cluster, and includes deploying an in-cluster ingress controller that will load balance between all those workers and serve out of penn-chime.phl.io

Skip any templating or automation at this phase, we'll manually deploy the manifests and manually edit them as needed. After we learn more, we'll create follow-up issues around templating or automation

Configure test DNS domain

Domain lke.penn-chime.phl.io has been pointed to the cluster to enable testing. App manifest will need to be updated to include this testing domain.

Set up automated UI testing

Split this off #60 -

I like this suggestion. I think verifying the models and data aren't broken is most important. However, even after the refactor, there is some coupling of logic and presentation (the framework we are using, Streamlit, makes it pretty hard to fully decouple the two). So, if we want to encode the manual testing procedure, I think that sounds like a puppeteer/selenium thing to me. Or even kasaya which seems like it might be a way to get QA help from non-coders.

Something like Cypress/Selenium can potentially serve three separate purposes:

1. Testing areas in streamlit that are hard to test through the provided APIs
2. Regular UI testing (though maybe this is something this project is delegating to streamlit)?
3. Unlocks the potential for a tool that that smoke tests that a deployed app is up and running