Giter Site home page Giter Site logo

emibcn / covid Goto Github PK

View Code? Open in Web Editor NEW
9.0 3.0 20.0 7.15 MB

Progressive Web Application that displays extracted data from the official web https://dadescovid.cat

Home Page: https://emibcn.github.io/covid/

License: GNU General Public License v3.0

HTML 0.72% JavaScript 98.80% SCSS 0.37% CSS 0.11%
covid frontend catalunya widgets reactjs create-react-app github-page workflow javascript webapp visual-widgets covid-data hacktoberfest climate-change covid-19 web-scraping

covid's Introduction

Node.js CI Coverage Contributor Covenant DeepSource Download maps and charts from backend and publish with GitHub Pages

Covid Data refactored (Web Application)

This applications displays public information about Covid19 Pandemy in Catalonia, extracted from the official webs https://dadescovid.cat and Seguiment Covid19 BCN.

The aim of this application is to display the data in different ways than the originals, improving performance and adding more value to it, trying to help the users to better understand the situation and evolution of the Covid-19 pandemic in the region of Catalunya (Catalonia).

App GIF example

Contributing

If you have an idea about:

  • Improve app performance, design, usability, testing or translations
  • Add data sources
  • Add visualization widgets
  • Add regions
  • Have funds or can look for them to speed up all the development processes
  • Something else?

Please, open an issue and let me know. If you're a developer, designer or translator (or just a user) and want to help develop, design, translate or test this app, let me know by opening an issue or -still better!- opening a pull request.

There are some already opened simple issues. If you feel you can afford one of them, just comment on it (so nobody starts duplicating work in parallel) and commit a PR when you think you are all done with it. If you don't know what a PR is, you can read these open source guidelines.

This is a Work in progress in its very early stages. All contributions are welcome, including writing a good Contributing page.

All contributors will be rewarded 🏆 by thanking them here (unless you prefer not to)! If funds are found, they will be fairly distributed across all contributors.

Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms.

🏆 Thanks to

How it works

The application has been split into 2 parts: the frontend (this one, the application you see deployed here) and the backend (the one containing the data used in the frontend, updated daily with official data).

The frontend

This applications has been created using Create React App. The applications is built and deployed to GitHub Pages using it's workflow.

The frontend is a React web application. It's intended to be used directly from the browser, or installed as a Progressive Web Application. It should work in all devices and browsers (if not, open an issue and I will do my best). Some of the code in this app is inspired from the code in the official app. Specifically, how it handles the data sources against the SVG maps.

Features

This application does not use any tracking system by itself, only the ones that could use GitHub Pages (they own the servers, they know the client IP). This means I have no feedback of the use of the app, so let a star or a follow, an issue or a pull request. They all are welcome! (Specially the last one :P ). It also means there is no EU cookie disclaimer/acceptance button, there is no one spying here, there are no ads.

The application lets you add more visual widgets into the grid/dashboard. The configuration of those widgets is saved in the location of the web browser (the hash), so you can undo or go back easily with native browser back/forward control. It's also useful to share a dashboard or a widget, as its URL contains its configuration. Furthermore, the app saves the same configuration into the localStorage system of your browser. If you come back to the app without your preferred configuration in the URL, it will automatically load it from the localStorage. Each time a parameter is modified, it is saved and all needed data is downloaded.

Each widget have a menu button where you can find its actions: Edit, Remove and Legend. Those actions open a dialog to show information to you or let you change some parameters.

There is a Slider with a Play/Pause button to handle the day the data is shown for. The Play/Pause button enables/disables the automatic day increase and restart, helping you to see time-based correlations.

The application automatically detects when the used data has been updated and need to be redownloaded, and when the application itself has been updated, and notifies the user to apply the updates.

The application have a Menu with an About section. An Update! section is visible when an app update is available. There are more to come!

The backend

The backend (the real one) are servers from institutions like the Generalitat de Catalunya, the Ajuntament de Barcelona or others that may come in the future. Those servers might have strict security restrictions (like CORS), or might not perform as desired, or the data offered there has already been processed, or because the data is updated just once a day (and we don't need to bomb the servers).

Anyway, those backends are scrapped once a day by a workflow on a sibling project: covid-data. That project consists in a small shell script that downloads the dadescovid.cat Maps data -and processes it a bit-, a full JS project to download dadescovid.cat Charts data and another full JS project to download data from Barcelona's RStudio/Shiny server. The workflow calls those parts sequentially and, finally, deploys all collected data to its own GitHub Pages. That deploy is not intended to be used directly from your browser, but through this application, which downloads the data (JSON and SVG files) to fill in the app widgets (or other apps that may come to re-use the same data/infrastructure).

So, in fact, this is a serverless application, where the cloud is provided by GitHub, GitHub Workflow and GitHub Pages (all with free tiers), and the original 3rd party backends, which are not part of this project (for the moment).

TODO list

  • Add languages.
  • Add more sections: Help, Settings, ...
  • Change branding icons.
  • Add tutorial for beginners.
  • Add more data sources.
  • Add more data visualization tools.
  • Add more regions.
  • Take a look at Google Maps iEPG new layer service.
  • Add tests.

Disclaimer

This app and the code are released as-is. The app may fail because there is a problem with GitHub or its servers (probably at Azure), or because the app itself (the code) has a bug, or because its deploy process has failed in some step. If your work or the live of someone relies on this app, please, install your own stack and pay someone to ensure it does not fails (and let me know! I could be that paid someone).

Licenses

The application, scripts and documentation in this project are released under the GNU General Public License v3.0.

The scripts and documentation published in covid-data are also released under the GNU General Public License v3.0.

The data scrapped and published in covid-data is licensed by their owners:

The Contributor Covenant is released under the Creative Commons Attribution 4.0 International Public License.

covid's People

Contributors

abhijith126 avatar ashish979 avatar carbarsoft avatar deepsource-autofix[bot] avatar deepsourcebot avatar dependabot[bot] avatar emibcn avatar gislinielsen avatar iamvevaar avatar magdalenapaluch avatar mandy6720 avatar mblaul avatar miguelmj avatar ppinfy avatar sahanamarsha avatar sebas-alarconr avatar wikiyu avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar

Forkers

abhijith126 reachtokish toyamarodrigo sahanamarsha mandy6720 ppinfy mblaul ashish979 gislinielsen meghalbisht magdalenapaluch carbarsoft siriwatknp raunak-bardia iamvevaar nyariki ets-reactnative2 nandor-magyar f8aar npaulkr

covid's Issues

Add Cypress e2e to test and automatically generate the README's example GIF

As this is going to a fast paced project (sic), manually generating the GIF and the end-to-end testing will eventually be an stopper.

Proposal

  • Use Cypress with free tier: only 1 worker in the GitHub Workflow, no Cypress SaaS needed.
  • Add a dedicated directory to store all Cypress test files.
  • Do unit tests (some hard to test components from non e2e tests) and/or component compositions. Save as disposable artifacts.
  • Do example/tutorial/README integration test. Save as GIF into the badges branch.
  • Allow developers to execute e2e in development environment (as a package.json script?).

Note: This issue can be handled by more than one person and split into sub-issues:

  • Create Cypress environment: system and NPM dependencies (docker&docker-compose?), generate cypress config/ini files
  • Create the unit/composite tests
  • Create the tutorial test
  • Convert the tutorial test to GIF
  • Add tests and GIF creation to GitHub Workflow
  • Ease development environment execution

If you want to do just a sub-issue, create it by yourself and link it here.

Add Language selector

Let the user select his/her preferred language:

  • Language selector component could be placed directly into the menu
  • It already has a data source at App's state passed all way down to the Menu
  • If changed, all app is already translated into the selected language

Add a Snackbar for each backend data update

Each backend (currently, Maps and Charts) automatically tries to update the data -at least- once a day, when the official publication and covid-data Worflow finishes.

All of that is done without user intervention, neither giving any feedback.

Proposal

Add a Material/UI Snackbar to each backend update callback/subscription success execution.

  • Currently, backends only allow reaction to a successful update.
  • Probably -in the near future-, we'll have more callbacks available to better handle data updates statuses:
    • Schedule time for next update.
    • Start data update check.
    • Update found/not found.
    • Updating data.
    • Error updating data.
    • Success updating data (the only one currently exists).

Data updates callbacks:

(JS-0466) Prefer React components to have a `shouldComponentUpdate` method

Description

We recommended using the shouldComponentUpdate method in React components because it allows your Component to exit the update lifecycle if there is no reason to re-render the Component again. If this method returns true, react will re-render the Component every time there is an update. By default, it returns true. …

Occurrences

There is 1 occurrence of this issue in the repository.

See all occurrences on DeepSource → deepsource.io/gh/emibcn/covid/issue/JS-0466/occurrences/

Note: Translate to functional component

Add widget to consume BCN's timeline

The idea is to add a widget:

  • This is a Timeline data, which could match with a Timeline visualization like these ones, or some other (like Twitter/Facebook?). The original one is at Seguiment Covid19 BCN, section Cronograma/Timeline.
  • Use react-window or similar to prevent mounting too much components
  • Configurable tag list selector:
    • The tags may need to be manually/auto-post-processed grouped/structured (some are redundant, some may be included into others). Or look for another solution, like searching user inputs (including spaces) into each tag string.
    • The user may select multiple tags in a single widget
    • The widget shows a list of the messages matching the selected tags
    • Editable (saved) Positive/negative text filter? (in addition to tags selection/filter)
  • Show some fast controls to the user, like:
    • Auto-scrolling on/off on date change
    • Zoom?
    • Positive/negative text filter?
  • The widget will change depending on the currently selected date:
    • Center the visualization into the first message of the day
    • Animated scroll up/down on date change (controllable)

Data shape

BCN Timeline is a JSON containing an array similar to this:

[
  {
    date: "YYYY-MM-DD",
    title: "Some short text",
    tag: "Some non-cleaned up tag system",
  },
  ...
]

This data is scraped at Covid Data - BCN. There, some data post-processing may be added. For example, an index of {date: [matching messages]}, or some tags/grouping/fixing.

Original visualization

image

Data loading

It is loaded with the backend manager:

const cancelUpdates = this.BcnData.data('timeline', onLoadCallback);

Similar to BCN charts.

Some tags examples

  • OMS declaracions emergència
  • Internacional actuacions
  • Govern d'Espanya Actuacions
  • Primer cas a Espanya
  • Cancel·lacions esdeveniments
  • Primera mesura de confinament massiu
  • Concentracions massives als carrers
  • Espanya actuacions
    -...

Add unit tests with `jest`

Add unit tests for components:

  • src/
    • App
    • About
    • AppThemeProvider
    • asyncComponent
    • Dashboard
    • ErrorCatcher
    • index
    • Language
    • Loading
    • MenuItems
    • Menu
    • ModalRouter
    • ModalRouterWithRoutes
    • Slider
    • Throtle
    • withDocumentVisibility
  • src/i18n/
  • src/Backend/
    • IndexesHandler
    • Provider
    • src/Backend/Base/
      • Cache
      • Common
      • ContextCreator
      • GHPages
      • withHandlerGenerator
    • src/Backend/Bcn/
    • src/Backend/Charts/
    • src/Backend/Maps/
  • src/Widget/

Current coverage

Coverage

Add data source "Evolució dels casos, altes hospitalàries i defuncions per Covid-19 a Catalunya"

Evolució dels casos, altes hospitalàries i defuncions per Covid-19 a Catalunya

http://governobert.gencat.cat/ca/dades_obertes/dades-obertes-covid-19/

https://app.powerbi.com/view?r=eyJrIjoiMjg2NjBkYjQtNWMyZS00YWZlLWIxZWMtM2UyMDAyNDZiYTI2IiwidCI6IjNiOTQyN2RjLWQzMGUtNDNiYy04YzA2LWZmNzI1MzY3NmZlYyIsImMiOjh9

It can be added as a new Backend (plus new widget type) or as an extension to current Maps and Charts (complementing it's JSON files it is really the same data source, adapting to its data shapes, re-using its widget visualization system).

As extension of current Maps and/or Charts (preferred, good first issue)

As new Backend (not good first issue)

Move Backends logic to a Data Context Provider/Consumer logic (allow easy backend adding)

Currently, backends are instantiated and central handled directly from Widget/List component, and then passed each backend index to all widgets (kind of...).

Proposal

  • Create a Context Provider/Consumer pair for each backend
  • Compose all providers into a single BackendsProvider, and add it to Widget/List parent (App)
  • Add backend consumers to each component/widget that needs it

Handle backend updates with correct timezone

Currently, the data auto update mechanism only works if the user has the same timezone than the official at Catalonia: CEST (with timelight.

https://github.com/emibcn/covid/blob/master/app/src/Backend/Maps/index.js#L132

  // Calculates haw many milliseconds until next schedulled update (today's or tomorrow)
  // TODO: Take care of timezones: Official date is in CEST/GMT+0200 (with daylight saving modifications), Date uses user's timezone and returns UTC
  //       Now, it only works if user timezone is CEST
  //       Probably, the best would be to translate both dates into UTC and, only then, compare them
  millisToNextUpdate = () => {
    const now = new Date();
    const todayDataSchedule = new Date(now.getFullYear(), now.getMonth(), now.getDate(), ...this.officialUpdateTime, 0, 0);
    const millisTillSchedulle = todayDataSchedule - now;

    return millisTillSchedulle <= 0
      ? millisTillSchedulle + 86_400_000 // it's on or after today's schedule, try next schedule tomorrow.
      : millisTillSchedulle
  }

Add CONTRIBUTING.md

To help new comers contribute.

Features:

  • Use plain English: contributors will normally be non-native English speakers.
  • Be welcoming.
  • Simple: short and concise better than long. If needed, long texts will go into sub pages (for example, setting up development environment).
  • Refer to code of conduct.
  • Add some coding standards: indenting, preferred choices (for example: const copied = [...original] instead of const copied = original.slice()), etc
  • More?

Note: There is some text already written in the README.md file.

Allow dragging widgets to reorder

Add dragging capability to widgets.

Features:

  • Consider unique line order (current) or multiple row/columns.
  • Consider adding a resize function, to allow widgets with different sizes.
  • On drag end: re-order widgets in URL hash and localStorage, or add a position property to each widget and handle it on add/remove/re-order.
  • I have done something similar using Dragact. Might be useful here, or maybe we can use Material/UI own draggable, or something else.
  • Drag handle: Use widget' title (like current app dialogs on non-small devices) or dedicated handle on it, like GitHub profile:
    image

This should go into a new component (or multiple components wrapped up into just one), used in Widget/List to wrap the list of widgets.

Add speed-o-meter

The slider has a Play/Pause button, which enables moving the current day forward automatically. The speed may be too fast for some kind of observations. It would be great to have a component which helps with that.

Proposal

Add a button next to the play/Pause button (maybe with something like this). When clicked, a popover (anchored menu, like this one) appears, with a vertical slider in it. Higher values makes the Play/Pause effect go faster, lower ones slower.

Add some lateral margin to Slider on non-small devices

On non-small devices, dates are shown and space is "cheaper". Add some margin between the slider and the buttons it has around, for devices grater than md.

Files:

Source Slider docs: https://material-ui.com/api/slider/#css

Fix fullscreen dialog for small devices

I have been reported about a bug in some small devices, where the dialog' Close button falls out of the view. The user needs to scroll down a bit to see the button. Looks like it's when the app is viewed directly from browser, while it is showing the navigation bar.

Initial view (before scrolling):

image

Final view (after scrolling down):

image

Fle: https://github.com/emibcn/covid/blob/master/app/src/Widget/Actions.jsx#L53

Some extra info: https://stackoverflow.com/questions/37395561/how-to-hide-a-mobile-browsers-address-bar
Possible solution: https://chanind.github.io/javascript/2019/09/28/avoid-100vh-on-mobile-web.html

Add visual mark on App Menu opener when app update is available

On small screens, there is no visual mark about an app update (unless the user manually opens the menu and sees the Update! link).

Proposal

On small devices (lower than md, when useMediaQuery(theme.breakpoints.down('md'));), add a MaterialUI Badge (with something like new on it) when there is an update available (props.newServiceWorkerDetected === true).

Where to add the Badge: https://github.com/emibcn/covid/blob/master/app/src/Dashboard.jsx#L168

Improve backends context provider/consumers

Things to improve:

  • Stop passing backend indexes from Widget/List to the actual widgets: they'll get them from consumer. Ensure they all do it cleanly (sorry about differences between widgets, this project is growing fast).
  • Include in the HOC a condition to show a <Loading/> instead of wrapped component while index is not loaded (and subscribe to index download, passing it to wrapped component once loaded). Backend/Map uses a static index: might be needed to create an auto-resolved Promise to homogenize interfaces.
  • Include in the HOC the possibility to subscribe to data by props. For example, in Widgets/Bcn/Widget.jsx, we will be able to remove almost all class methods from ChartDataHandler class (hence simplifying new widgets creation and maintenance ). Here, the consumer will need to detect if this.prop.dataset is defined and, if yes, use it to download data and show a <Loading/> while loading that.

Originally posted by @emibcn in #33 (comment)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.