reustle/covid19japan

covid19japan's Issues

Add Prefecture Breakdown of Active and Recovered Cases to Table

@reustle

As time progresses the number of total cases versus active cases will likely progressively diverge, making the number of active cases a much more relevant metric for how day to day behavior should change. A similar argument could be made for recovered cases.

Right now with the given setup the page only displays total active and recovered cases. However, cases by prefecture would be much more helpful information.

The page currently has a table which could easily be extended to include this information.

If pointed in the direction of where the page is currently getting its data, I could probably add this in pretty quickly. This is assuming we also have the relevant data at a prefectural level.

New Design

Trying to improve the design while keep the coding part simple.

Changes:

• Re-designed header part
• Re-designed the site icon (a bit branding)
• Added daily increased number in KPI
• Moved graph/map keys to the right side, so it's easier to connect with the graph/map content
• Changed "Prefecture Data" section to "Case Analysis" section
• Re-designed "Helpful Links" section for better readability
• Added footer

@reustle Let me know your feedback

Show Tests Performed Metric

https://www.mhlw.go.jp/stf/newpage_09724.html
https://www.mhlw.go.jp/stf/newpage_09754.html

It would be helpful to include the number of people tested daily, to compare that against how many were positive and negative.

Unsure of the best way to graph it yet, any suggestions?

Add 'Delta' dataset to help visualize rates of change

#23

I wanted to see what the first and second derivatives looked like to help me decide when to stockpile food and start watching '28 Days Later' for survival tips.

Currently not working

Loading the page at https://covid19japan.com/ results in no data displayed and the following error in console.

main.js:81 Uncaught (in promise) TypeError: Cannot read property 'length' of undefined
    at pullLatestSumAndDiff (main.js:81)
    at calculateTotals (main.js:90)
    at (index):225
    at main.js:56

Show Prefecture Case Count On Hover

When you hover over a prefecture on the map, show the number of cases

Fix Prefecture Sorting

When we load prefecture information from the sheet, we currently assume that it is sorted by the number of cases already. We should sort it on the front end as well.

Django Backend

@olingern moving our convo here

At some point, we will outgrow Google Sheets. Doing some quick data modeling in Django would be easy, and we'll get a free admin interface w/ user accounts, but the problem lies with the data entry side. Google Sheets makes it easy to see who is editing what, and not overwrite each other. What minimal UI could be pulled together with off-the-shelf libraries to help achieve this?

One thought is, a simple table of each of the patients where you click 'edit' and a modal opens. As soon as a modal opens, we put that users icon on the table saying that the person is currently viewing the form for that patient. That would automatically be reflected (sockets?) on the table for everyone. Ideally, nobody opens a form and leaves for 30 min.

Open to all suggestions. Thanks 🙏

Brainstorming data model:

let patient = {
  id: 'number', // our internal patientId. check source reference to see foreign ID
  dateAnnounced: 'timestamp',
  dateAdded: 'timestamp',
  createdBy: 'REF user_id',
  modifiedBy: 'REF user_id', // last edited by
  ageBracket: '0,10,20, etc',
  gender: 'M,F,null',
  detectedLocation: 'REF location',
  residentLocation: 'REF location',
  status: 'REF patientStatus',
  sources: 'REF 1-to-many patientSource',
  relatedPatients: '1,14',
  occupation: 'taxi driver 3',
  notes: 'freeform notes field',
  origin: 'REF origin',
}

let origin = [
  { label: 'charterFlight' },
  { label: 'diamondPrincess' },
  { label: 'japan' },
]

let location = {
  city: 'string',
  prefecture: 'REF prefecture',
  country: 'jp',
  lat: '',
  lng: ''
}

let prefecture = [
  { label: 'Tokyo'}
]

let patientSource = {
  label: 'i.e. MHLW',
  patientId: 'number ref to original patient ID',
  sourceId: 124, // ID on the source such as MHLW
  url: 'url here',
}

let patientStatus = [
  { label: 'hospitalized' },
  { label: 'Discharged' },
  { label: 'Recovered' },
  { label: 'Deceased' },
  { label: 'Unspecified' },
]

Travel Bans

1. Add a list of countries banned from travel into Japan (or need quarantine)
2. Add a list of countries which people coming from Japan are banned from entering

i.e. https://twitter.com/motokorich/status/1240248532377858050

parts of Spain, Switzerland, and all of Iceland, plus mandating 14-day quarantines for travelers from 38 countries, inducing all Europe, UK, Ireland and Egypt.

Update: V1 is now launched on the site, we just need to update the data

---

Case Count In Title

Show the total number of cases in the page title

Embed Javascript Widget

We currently have https://covid19japan.com/embed which allows news sites to embed the map directly into their page. Since iframes are difficult, the content is either cut off or too wide. There are many resources on how to create responsive iframes. I feel the widget should work as follows

Provide some kind of javascript snippet such as

<script src="https://covid19japan.com/embed.js"></script>

Which contains roughly the following code

document.write('<div id="covid19japan-embed"><iframe src="https://covid19japan.com/embed"/></div>')
document.getElementById('covid19japan-embed').style = // TODO

Basically we create a wrapper element, load the iframe within it, and set the wrapper style to handle the responsiveness properly.

User front-end framework to facilitate development of aforementioned feature requests

@reustle
I would love to contribute, I am familiar with Vue.js framework, but can also pickup React.js if everyone else is more comfortable.

Automate Data Update Checks

Crawl the government release pages and notify somehow (via email?) that a new update has been posted.

Sources listed in the readme here: https://github.com/reustle/covid19japan#data-sources

Similar issue: #17

Bundle JS Libraries

We are importing a lot of libraries at this point. Let's pull them all into a single file and save them within the repo, and load them as one file. Probably good to use webpack.

Draw Prefecture Borders

Need to fix the rendering order so that prefecture borders are drawn on top of prefectures

Related discussion #64 (comment)

data.json looks from a different time zone

Right now (2020/3/3 22:34 JST), https://covid19japan.s3.ap-northeast-1.amazonaws.com/data.json ends with

{"date":"2020-03-01","confirmed":"257","recovered":"41","deceased":"6","critical":"20","tested":"2339"},{"date":"2020-03-02","confirmed":"283","recovered":"42","deceased":"6","critical":"23","tested":"2517"}],"updated":[{"lastupdated":"Mar 3 2020, 21:12 JST"}]}

the live patient database contains 283 confirmed cases, but 6 were announced on 2020/3/3.
Consequently, the total confirmed cases on 2020/3/2 is 277, and the difference should be 6
(but the website lists 283 confirmed cases on 2020/3/2 and the difference is 26)

A possible explanation is data.json was generated on a different timezone (and I do not know which one). I would assume most users are based in Japan and would rather have data.json in JST instead of an other (and unknown to me) timezone.

Please keep up the good work, this is a very valuable resource to me.

Split Data Pulling and Rendering

We currently render data as soon as it is loaded (same functions).

Split the loading of data into its own functions, which then triggers separate drawing functions. This will help with #10 and #11

Backend Data Validity Check

Like #11 but on the new backend cache code

Simple example 6bd933b#diff-45a12245c90c3f724f1644a02264341eR29

NaN appearing kpi-active and kpi-confirmed

@reustle Looks like JS is doing its thing 😅

Thanks for the service 🙏

Visual guide

Color

Map
#4C080A #8E0000 #BA000D
#F44336 #FB9B7F #FDEACB
#FFFFFF

KPI
#F44336 #BA000D
#1976D2 #004BA0
#37474F #000A12

Primary
#FFFFFF #EAEFF1 #A3ACBF #000A12
#1976D2

Typography & Font Size

Open Sans ExtraBold 48px
Open Sans Bold 34px
Open Sans Regular 21px
Open Sans Regular 16px
Open Sans Regular 12px

Concept

High level goal: Make coronavirus official information available in English

Map

A map outlining each prefecture in Japan, showing a count of the # of cases in each.

Data Table

Below the map, a table with high level info for each case

Helpful Resources

A list of helpful links, resources, and data sources for the page above.

Styles to SCSS

The styles are currently written in raw CSS.

We could use GitHub's built in Jekyll support to auto-generate CSS from SCSS files, which would allow us to clean the styles up greatly (with nesting).

Add Thank Yous

We need to thank Mapbox and Sentry for their generous service donations

Footer Explanation of "Active"

Explain somewhere how we calculate "active"

Add World Outbreak Data Table

Add a simple outbreak condition of the world at the end of the site.
So users can know how severe the situation is in Japan, compare to other countries.

Sheet Data Cache

Google sheets is starting to throw some 500 errors now that we're well past 100 concurrent users at any time. Let's create a script to pull the sheet values every 5 minutes and save a json file somewhere. Github Gist? S3?

Add City-Level Data

Do we have the ability to move from prefecture-level to city-level? It seems like many published cases include this information.

How would they be visualized?

Implement Babel

I noticed that the page doesn't load on Internet Explorer 11 (which still has a fairly large marketshare in Japan). Not sure what the issue is, but I wonder if Polyfill.io could help fix?
<script crossorigin="anonymous" src="https://polyfill.io/v3/polyfill.min.js"></script>

Auto Reload Data

Automatically reload the data source after 5 minutes. Once reloaded, if there are changes, redraw the elements on the page and the page title.

We don't need to re-draw the map, just the data layer.

How/where is data from the spreadsheet converted into the data.json?

Looking at the code in the repository, I do not find code that generates the data.json from the Google Doc spreadsheet. Am I missing it?

Add a LICENSE file

Which license do you suggest?

Age / Gender Data

Display age / gender information of the confirmed cases. This needs to be done with the patient data, and calculating it on the frontend.

Simple bar chart (stacked) https://c3js.org/samples/chart_bar_stacked.html

1. Include the 'patient data' sheet from the source spreadsheet into the cached json file that is generated every few minutes
2. Create a function to calculate the number of cases by gender / age bracket. The output will likely look something like this

[{
  ageBracket: 0, // Age 0-9
  maleCases: 13,
  femaleCases: 11
},{
  ageBracket: 10, // Age 10-19
  maleCases: 4,
  femaleCases: 7
},{
  ageBracket: 20, // Age 20-29
  maleCases: 27,
  femaleCases: 25
}]

3. Feed this new data into a newly created function drawAgeDistributionGraph which renders the stacked bar chart (Male and Female bars stacked per age bracket).

Include time of "last updated."

With how fast news about the virus changes, I think it would be useful to see not only the date of the last update, but also the time.

Japanese translation (first phase)

@reustle I've quickly translate index.html to Japanese and save it as index_ja.html in my repository forked from your original. I'd like to contribute the site as a Japanese person.

I'll send you the pull request after a few minutes. Since the map doesn't work in my local environment, Could you please modify to work well if you would accept the pull request.

I want to translate prefectures' name in the "Prefecture Data" table, however, it's difficult to translate quickly for now. It's my challenge.

Improve KPI with tested numbers and critical condition patients

To give users a thorough understanding of the current condition, add tested patients numbers and critical condition patients numbers in KPI.

Apply Mapbox Geographic Bounds for interactive map

Hey @reustle, just a minor design suggestion. The interactive Mapbox map has a good minimum zoom, but you could also apply a maxBounds to the map, so it's impossible to accidentally scroll the map far away from the general region of Japan itself when zooming or scrolling around.

Move Prefecture Calculations Into Code

We currently calculate metrics by prefecture within a sheet. This should be handled by the frontend on the raw patient data. The idea is to eventually move all calculations into Javascript, because dealing with multi-sheet spreadsheets as a data source is pretty fragile.

Remove Template Literals

Remove the use of template laterals as IE doesn't support them.

Sentry Issue: COVID19JAPAN-27

Error: Invalid character
  at None (/static/main.js:234:24)

"Active" explanation in Japanese

We've added the explanation of Active as a tooltip, but it's a bit tricky to swap the languages since it is done using an attribute. I think we'll either have to update the element's tippy attribute directly on language switch, or use a tippy API endpoint to manually set the text on language switch.

TypeError: this.paint is undefined

Sentry Issue: COVID19JAPAN-9X

TypeError: this.paint is undefined
  at queryRadius (../src/style/style_layer/fill_style_layer.js:42:9)
  at loadVectorData (../src/source/tile.js:189:88)
  at callback (../src/source/geojson_source.js:306:18)
  at vu.prototype.receive (../src/util/actor.js:94:24)

@olingern seems to have started around the deploy of auto reloading? Not sure if its related, but seems yes. Maybe fixed in our recent push, though.

Mapbox Boundaries

Mapbox has offered to give us access to their boundaries feature. Would this be helpful with regards to prefecture data? Ward boundaries? etc?

Add data and information for Diamond Princess

@reustle - Thanks for making this, it's been really helpful for me personally and I'm sure many others

The site should also add numbers for Diamond Princess. Additionally, site visitors should know that the Japanese government let people off without properly testing, and that there are still more people on the ship who are sick who will be let off later.

To avoid creating multiple issues I've listed other ideas below:

1. Add to the Helpful Links section basic info like "the best method of prevention is hand washing, this is how to properly wash your hands," "does wearing a mask actually help," "what is the mild/severe/critical case ratio," etc. with links embedded in the information.
2. Describe not only the officially published numbers but the actual situation, e.g. that the Japanese government is under-testing compared to Italy/SK and that's why the numbers are lower, that even if you get sick there's no treatment, and to go to the hospital if your fever goes over 37.5C or you have difficulty breathing. All of this should be sourced and linked to as well.

I'm happy to make pull requests as well, please let me know.

Call To Action: Data Updates

Provide a clear call to action to draw people into the google spreadsheet, incase they can help with verification / addition of missing cases.

I made auto scraper for tracking Japan numbers

@reustle
Hello,
I'm a DevOps Engineer and I live in Tokyo, Japan.
For my personal usage, I started a basic python scraper to automate collecting data about Japan Corona Virus cases.
http://wasfy.ddns.net/corona_virus_tracker.html

I like to share it with you and to help if needed.

Best regards,
Hatem Wasfy

Add FETCH Polyfill

fetch needs a polyfill on older browsers

Frontend Data Validity Check

When loading a data sheet, ensure the data (keys) look correct before trying to load the data. There was an issue where the sheets were reorganized and broke the site.

Workflow spam in fork

After updating my fork to the latest version, I got hundreds of emails from Github, because the "Fetch data" workflow threw an error on my fork (because the secrets are not there).
I'd change the script to just stop running when the necessary environment variables are not there instead of throwing an error.

Or is there a way to only run in certain repos?

I know that I could turn off actions in my fork. But every contributor would have to do that manually, after getting spammed.

Static Map Image

Instead of loading mapbox on every page load, let's generate an image of the map and prefectures once every 5 minutes (like we do with the json cache) and store it on s3.

Once the user clicks the image, it will then load the real mapbox which they can interact with. I would bet that >75% of users don't touch the map, so loading mapbox isn't necessary.

Embed Page

Create a simplified version of the site that can be embedded in other web pages.

So far, it has been requested that the page includes only the title and the map, with a link to the full site.

Tests to Trend Line

Add the number of tests performed to the trend chart, or its own chart