reustle/covid19japan-data

covid19japan-data's Issues

Can't find Iwate prefecture data

There is Iwate prefecture in googleSheet COVID19 Japan Patient Database (Google Spreadsheet) but it isn't contained in Daily summary and Per-prefecture summary. Why isn't it contained?

Github Workflow has error when trying to add new data files.

Looks like the current action I'm using (add-and-commit) is not really work very well because it tries to do some branching magic before commiting which seems to break new file adding.

Need to find something to replace.

At midnight every night the github action fails until I manually push the data.

Integrate scrapers from prefecture stopcovid19 teams to automate initial data collection

This is a placeholder. I'll add details as I learn more

Motivation

• Case count is increasing rapidly. We're likely to become overwhelmed if we only have manual data entry
• Several other teams working on prefecture-level covid19 projects have automated scraping and ingestion of the prefecture level reports. If we can integrate these, we don't have to get drowned in figuring out how to scrape every single prefecture

Approach

Just the gist of the approach

• Integrate prefectures individually, starting with the prefectures with the highest daily case counts
• Add as a baseline for data entry?

Known scrapers / data sources

WIP

• MHLW from https://www.stopcovid19.jp/
  • scraping, api: https://github.com/code4sabae/app_sabae_cc
  • all data: https://github.com/code4sabae/covid19/blob/master/data/covid19japan-all.json
  • Bonus: Collection of potential primary sources by prefecture https://docs.google.com/spreadsheets/d/1SPqnO0yLn8ubax96sDJZVDcjAH8QT1suLCIgroPGVHY/edit#gid=0
• Tokyo:
  • Imports directly from Excel. Probably with this: https://github.com/tokyo-metropolitan-gov/covid19/tree/development/tool
  • Direct link to data: https://raw.githubusercontent.com/tokyo-metropolitan-gov/covid19/master/data/data.json
• Saitama:
  • Open Data: https://opendata.pref.saitama.lg.jp/
  • unfortunately, the opendata link is not frequently updated, and PDF seems to be the main location. Yet to be re-automated?
  • PDF Source: https://www.pref.saitama.lg.jp/a0701/covid19/documents/youseisyaichirannzeroyonn.pdf
• Hyogo:
  • blog article: https://qiita.com/y-chan/items/87919fd712d1560c5e95
  • Scraping source: https://github.com/stop-covid19-hyogo/covid19-scraping

Is there a list of known cluster-sparking situations in your data sources?

Hi @reustle & thanks for starting this project :-)

Seeing that Japan has been warning about the 3Cs all along, I was wondering whether any of the data sources you list contain a list of situations, that are known to have started outbreaks? Ideally, it would be continuously updated, i.e. via literature reviews of case studies.

Some are kinda obvious, and probably the same as for influenza and the like:

offices, schools, hospital waiting rooms, dining rooms, party rooms, karaoke boxes etc.

But for a dedicated, evolving list (i.e. as proposed for corona-warn-app/cwa-website#346), do you know of any such a specialized list?

Thanks for any hints!

7-day drop in new cases highlights nature of data sources

Every Monday the daily new cases drops dramatically, and peaks again the next day. It means 2 things, actually 3: [1] what we get to see is already over a day old, and [2] on Sundays many cases aren’t reported since [3] the data is collected manually.

There is mounting sentiment that not the whole picture is being told, as the magazine The Economist claimed excess covid-19 deaths can be 2x the officially reported deaths in some countries. Could that be the case for daily new cases in Japan?

Are there plans to fix this gap between what is told, and what is really happening?

There is a broken link in README.md

The link of the example of a daily report of the NHK is broken.

"Example: Daily Report (新型コロナウイルス 国内感染者)" is the link.

Confuse of data showed on 22 may 5:20 japan time

Today the site showed 16364 adding 41 from yesterday. But yesterday it showed 16276. So it suppose to be 88 not 41. But when i manually add up from prefecture data, it is correct 41 cases. So which one is correct?

Wrong date in the first element of daily JSON list

The date of the first element of daily property of the latest summary is wrong since it is "date": "10.22",

{
  "confirmed": 0,
  "confirmedCumulative": 0,
  "deceased": 1,
  "deceasedCumulative": 1,
  "reportedDeceased": 0,
  "reportedDeceasedCumulative": 0,
  "recovered": 0,
  "recoveredCumulative": 0,
  "critical": 0,
  "criticalCumulative": 0,
  "tested": 0,
  "testedCumulative": 0,
  "active": 0,
  "activeCumulative": 0,
  "cruiseConfirmedCumulative": 0,
  "cruiseDeceasedCumulative": 0,
  "cruiseRecoveredCumulative": 0,
  "cruiseTestedCumulative": 0,
  "cruiseCriticalCumulative": 0,
  "date": "10.22",
  "confirmedAvg3d": 0,
  "confirmedAvg7d": 0,
  "confirmedCumulativeAvg3d": 0,
  "confirmedCumulativeAvg7d": 0,
  "deceasedAvg3d": 0,
  "deceasedAvg7d": 0,
  "deceasedCumulativeAvg3d": 0,
  "deceasedCumulativeAvg7d": 0,
  "reportedDeceasedAvg3d": 0,
  "reportedDeceasedAvg7d": 0,
  "reportedDeceasedCumulativeAvg3d": 0,
  "reportedDeceasedCumulativeAvg7d": 0,
  "recoveredAvg3d": 0,
  "recoveredAvg7d": 0,
  "recoveredCumulativeAvg3d": 0,
  "recoveredCumulativeAvg7d": 0
}

Data not updating since 9/15

Looks like the GitHub actions are firing but latest numbers are of 9/15 (3 days ago).

Aomori Prefecture Recoveries Sep 3rd

As of September 3rd Aomori Prefecture only have 2 active cases as opposed to the 5 active cases shown. These three extra may be the ones connected to the American Military Base.

Osaka active cases

Once again thanks for all the hard work in maintaining this tracker!

Osaka numbers seem very out of whack compared to the official site https://covid19-osaka.info

we just hit the 5 million cells limit !!!

We cannot add more cells otherwise it would be more than 5 million cells, and this is not supported.

I guess the easiest way to move forward is to create a new spreadsheet and have generate.js and friends merge both files.

Right now, we're stuck :-(

Possible problems with the number of active cases at the prefectural level

May I just have a note on some pieces of data that have appeared there over the past week or so. Despite decreasing numbers of active cases, it seems as if the number of cases, when broken down to prefectures , would be increasing. E.g. for Kumamoto prefecture it appears as if there haven't been any recoveries over the past week.

In addition, when you add up regional data, the sum (ca. 41,000) is much higher than the cumulative one reported at the top of the page (currently ca. 24,000).

From the above, I suspect that the numbers of daily recoveries have not been deducted from the actual prefectural numbers of cases for some time now -- could this possibly be the case?

Active cases in Osaka

This seems similar to another issue already posted on here, but I was asked to open an issue. Every day I religiously check the covid19japan site for updates on active cases in Osaka. The past week or so, it's been doing nothing but going up. If I take the previous day's active cases, add "Today's" new cases, I get "current" active cases, which doesn't account for recoveries. It seems to conflict with the Osaka-specific site here, which paints a much better picture: https://covid19-osaka.info/

Thank you!

Protect against Prefecture Data not loading

When we fail to fetch Prefecture Data sheet, we should abort and fail to avoid creating data without any other of the KPIs.

857e8dc#diff-75a2ac3cb4607efaf03c2cd7dad96714

Data for Osaka is missing

The data has shown Osaka at zero for the last few days. I'm fairly sure this is incorrect.

Says "Zero Active Cases" for my prefecture, but there are plenty of active cases!

The Japan Covid 19 tracker is great. Thank you all so much for all your labor there. However, I live in Mie-ken and it keeps saying we have zero active cases and shows a congratulatory champagne bottle. Yet, it also indicates a number of new cases were reported each day (more than 22 in one day, just a few days ago), which would, of course, contradict there being no active cases. I check the Japanese language site of Mie regularly, and we actually have quite a few active cases, at least 60 in the hospital alone, and 4 are critical. I'm afraid we don't deserve the champagne yet. This "Zero Active" has been going on for a week or more. I wonder why it just recently started happening?

How can I obtain historic region level data?

The website has charts for each region (Kansai, Kanto, etc.). Is there an endpoint where I could obtain the raw data/ or alternatively find a source that gives me the same chart but with an x-axis that is labelled with the dates?

Great thanks for the awesome dashboard!

Incomplete data in "Sum By Days" on the Google spreadsheet

I left two comments in the last two weeks on https://docs.google.com/spreadsheets/d/1zdMR6BhAU00Ol_4WfYmZRmjUYZp_RkwOZMeRGUX4zBY/view#gid=1690900878, but apparently nobody looks at that form anymore. So I'm opening an issue here.

The sheet "Sum by Day" is missing data on the "confirmed" column. It has been the case every time a new sheet has been opened for a prefecture and data moved from the "Patient data" sheet to that new one. Interestingly, the data on the "Decesased (auto)" column does include all prefecture sheets.

Consequently, the total number of cases is missing about 3000 cases. That makes this sheet unusable (until two weeks ago I was copying the data manually into my own data sheet on my PC).

Should be a 10s fix, unfortunately I don't have write access to the Google sheet.

json is no more automatically updated

The number of cases on 6/29 is reported at 1050, though my node generate.js invokation returns the correct number (1381).

It seems the script (cron?) did not run for some time.

Death Data Discrepancy on January 19th 2021

Hello!
In regards to death data on January 19th 2021.

Based on the data reported in docs/summary/latest.json, it looks like the cumulative deaths in the morning of January 19th was reported at 4366 and later in the day was reported at 4680. However, the daily deaths only shows an increase of 104 deaths.

Would you mind explaining this discrepancy? Thank you!

Kochi City confused with Kochi Prefecture

We found a small bug in the Japanese data. Somehow Kochi City was being provided, despite us looking only at prefectures. Was the city and prefecture confused in the data set? Noticed this about Sept 14.

Metadata Update

Hello,
Your dataset was added to CoronaWhy (https://www.coronawhy.org/) Data Lake on Dataverse as a piece of common COVID-19 data frame http://datasets.coronawhy.org/dataset.xhtml?persistentId=doi:10.5072/FK2/LJFRRK

Would you be willing to help with the maintenance of your dataset in Dataverse, e.g. adding the relevant metadata and keeping the dataset up-to-date? That will help to make the dataset findable and accessible for the medical science community.

Problem with the Ibaraki case numbers?

There seems to be a problem with the Ibaraki case numbers. it currently reads 50,946 - with daily new cases just 423.
Whereas Tokyo has about 70,000 total cases and approx 7000, new cases; and Kanagawa 32,000 total cases and 3000 new cases. Both are in the order of (daily cases x10)=(Total cases). But in the case of Ibaraki it is closer to (daily cases x100)=(Total cases) - I don't think this is correct.
What is the source of the data? I imagine that the data in recent weeks has only included new cases and not people who have recovered.

On the local site - I think the total is listed as - 8,461 cases (https://www.pref.ibaraki.jp/1saigai/2019-ncov/index.html) - I could be wrong though; but I would have thought that is closer to the correct number.
The local government website seems to have changed how it presents the data - could that be causing the issue with the data entering the website? (if you are using an automated data collection system).
If you could take a look - it would be useful. Thanks

No data for Tokyo last two days?

Seems the Tokyo numbers are not updating for 12th & 13th July. Wonder if there is an issue as that has a material impact on total numbers.

Any plans to start capturing vaccination data?

There are now some official sources (e.g. https://www.kantei.go.jp/jp/headline/kansensho/vaccine.html ) with counts of vaccinations. Are there any plans to start including these as well?

Tochigi Prefecture Recoveries - Aug 12

Hi. I believe the recoveries for Tochigi Prefecture are incorrect for August 12th. They are currently listed at 1808, but there are only 261 cases in the prefecture. According to the linked dashboard (https://covid19-tochigi.netlify.app/en), the number should be closer to 182.

Problem adding new files in the Github Action workflow

For the past two days, the github action workflow fails when we cross the day boundary because it seems to fail to add new files that were created.

Recovered of Tokyo seems not to be updated for a while

Forgive me if this topic is not for here.
I'm not sure where I can report.

As the title stated, I suspect the number of recovered of Tokyo seems not to be changed for a while.
The webpage below said almost thousands "Discharged", while covid19japan.com says 59 "Recovered".
https://stopcovid19.metro.tokyo.lg.jp/en

This might intentional... But just I want to let you know in case you're not aware of...

First data date in daily field is wrong

The date of the first element of daily property of the latest summary is wrong since it's 2002-05-10: 18 years ahead. Also the month and day are wrong, it should be something like 2020-01-08, so I suppose there is a problem with the original data.

"daily": [
    {
      "confirmed": 1,
      "confirmedCumulative": 1,
      "deceased": 1,
      "deceasedCumulative": 1,
      "recovered": 0,
      "recoveredCumulative": 0,
      "critical": 0,
      "criticalCumulative": 0,
      "tested": 0,
      "testedCumulative": 0,
      "active": 0,
      "activeCumulative": 0,
      "cruiseConfirmedCumulative": 0,
      "cruiseDeceasedCumulative": 0,
      "cruiseRecoveredCumulative": 0,
      "cruiseTestedCumulative": 0,
      "cruiseCriticalCumulative": 0,
      "date": "2002-05-10",
      "confirmedAvg3d": 0,
      "confirmedCumulativeAvg3d": 0,
      "confirmedAvg7d": 0,
      "confirmedCumulativeAvg7d": 0
    },

Pull data in from governments using CKAN

Several governments are using the stopcovid19 dashboard which is powered by CKAN. They have some of their data in CSV format that might be fetchable programmatically

e.g. Fukuoka
https://fukuoka.stopcovid19.jp/
https://ckan.open-governmentdata.org/dataset/401000_pref_fukuoka_covid19_patients

Note, their data is still usually more than 24 hours delayed, but good for cross checking old patients.

Two records with negative tested values.

@reustle
There are two records where the tested value is negative.

[
  {
    "confirmed": 36,
    "confirmedCumulative": 927,
    "deceased": 2,
    "deceasedCumulative": 32,
    "recovered": 0,
    "recoveredCumulative": 144,
    "critical": 0,
    "criticalCumulative": 46,
    "tested": -1130,
    "testedCumulative": 14525,
    "active": 34,
    "activeCumulative": 751,
    "cruiseConfirmedCumulative": 712,
    "cruiseDeceasedCumulative": 7,
    "cruiseRecoveredCumulative": 458,
    "cruiseTestedCumulative": 4234,
    "cruiseCriticalCumulative": 15,
    "date": "2020-03-18",
    "confirmedAvg3d": 34,
    "confirmedCumulativeAvg3d": 870,
    "confirmedAvg7d": 41,
    "confirmedCumulativeAvg7d": 795,
    "deaths": 2
  },
  {
    "confirmed": 97,
    "confirmedCumulative": 1311,
    "deceased": 3,
    "deceasedCumulative": 45,
    "recovered": 21,
    "recoveredCumulative": 272,
    "critical": 2,
    "criticalCumulative": 57,
    "tested": -909,
    "testedCumulative": 23521,
    "active": 73,
    "activeCumulative": 994,
    "cruiseConfirmedCumulative": 712,
    "cruiseDeceasedCumulative": 10,
    "cruiseRecoveredCumulative": 597,
    "cruiseTestedCumulative": 4234,
    "cruiseCriticalCumulative": 11,
    "date": "2020-03-25",
    "confirmedAvg3d": 69,
    "confirmedCumulativeAvg3d": 1204,
    "confirmedAvg7d": 54,
    "confirmedCumulativeAvg7d": 1094,
    "deaths": 3
  }
]