Giter Site home page Giter Site logo

istat-methodology / terra Goto Github PK

View Code? Open in Web Editor NEW
4.0 2.0 0.0 12.8 MB

Istat Experimental Statistic: TERRA

Dockerfile 0.13% Batchfile 0.08% JavaScript 15.82% Python 23.97% HTML 0.06% Vue 27.94% SCSS 0.98% PowerShell 0.07% Shell 0.38% Jupyter Notebook 30.58%
comext data-visualization official-statistics cloud data-analysis experimental-statistic

terra's Introduction

TERRA - imporT ExpoRt netwoRk Analysis

TERRA has been implemented by Istat's Team in the context of the European Big Data Hackathon 2021 organized by Eurostat.

TERRA is an open source dashboard implemented using modern javascript frameworks (Vue.js) and microservice architecture for the server components (currently we use Docker Containers). The implemented architecture is scalable and allows to integrate Python and R languages in a data processing pipeline.

TERRA was awarded first place at the European Big Data Hackathon 2021. Istat's team pitching is available on vimeo

Core functionalities

TERRA offers a set of functionalities that allow to analyse international trade relations both at macro and micro level. At macro level using social networks analysis techniques (implemented in Python) we investigated both the effect of shocks in transportation and the effects of relation disruptions. At micro level analysing google mobility data we provide an indicator (implemented in R) representing the restriction imposed by COVID pandemic. This indicator has been used to perform an analysis to:

  • evaluate the effects of pandemic in import and export;
  • nowcast and forecast new scenario derived by mobility policies.

TERRA provides the following functionalities

  • Interactive map: with a focus to the recent pandemic the Team provides for each country a set of macro – economic indicators together with the total value of import and export. Moreover to focus the analysis on the trade market for each of the 27 EU countries TERRA provides: i) the percentage of total import and export represented by the three main good exchanged (in import and export); ii) the percentage with respect to the total (of the import and export with respect to the three main partners). Moreover the interactive map displays the percentage variation in import and export with respect to November 2020. The analysis is based on COMEXT and Eurostat data.
  • Mobility policy analysis: this function given the mobility data available from google provides two different results: i) descriptive analysis of the main mobility index for the selected country: Retail, Grocery and Pharmacy, Parks, Transit Station, Workplaces, Residential; ii) provide a daily and a monthly mobility policy indicator whose value rage is 0–1 representing the level of restriction imposed by the selected country government.
  • Covid-19 impact evaluation on international trade: given a Country and Partner TERRA offers a set of descriptive statistics representing the trade situation between the two countries in the period before and after a specific reference date (suggested by the user). This functionality allows to analyse graphically and quantitatively the Covid-19 monthly impact on import and export time series in value. The user can choose a country and select a single BEC at 1-digit level or totals. Moreover it is possible to require a nowcasting to the actual date, or a forecasting of the series connected to possible scenario of mobility restriction introduced directly by the user. The Team used the interrupted time series analysis to estimate the effect of mobility restriction imposed by each government to the import and the export with respect to a country and detailed by broad economic categories (bec). The analysis is based on COMEXT and Google Mobility Data
  • Graph analysis: the Team used Monthly COMEXT data by Means of Transport to build the graph of international trade relations and use standard graph measures to characterize the relations structures. The Team built an interactive page to visualize all the measures and functionalities derived from the graph analysis. A detailded description of the function is available here.
  • Trade: Member State monthly imports and exports have been decomposed in terms of traded products in order to give a picture of trade trends under the Covid-19 pandemic in a short-term perspective. Despite the global drop of international trade during the pandemic, a strong heterogeneity across traded products is observed by computing year-over-year monthly percentage changes for each considered product class. Indeed, for each Member State, the critical pandemic months show an increase of share in the export of non-durable consumer goods, such as pharmaceutical products, chemicals or floods and beverage, and at the same time a drop in export of durable consumer goods or investment goods, such as motor vehicles, trailers and semi-trailers. On the other hand, the import side shows an increase of share for products needed to fight against coronavirus, such as textiles (facial masks). Since for most Member State the heterogeneity is strong localized during the pandemic waves, the indicators provide a clear picture of national deficit or surplus of specific products most needed at crisis time. The input data source consists in Comext monthly dataset by products, where Member State trade classified according to CPA2.1 at 2 digits classification level was considered. Comext data were appropriately transformed in order to get, for each Member State, shares of CPA product divisions on total export and import, and finally compute year-over-year (YOY) changes of such shares during 2020.

terra's People

Contributors

defausti avatar dependabot[bot] avatar francescoamato avatar giulio-massacci avatar lgiuliani80 avatar liminalfever avatar mauroistat avatar papizzo avatar

Stargazers

 avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

terra's Issues

[terra-update-batch] cambiare endpoint trasporto classe 07

COMEXT TR:

NSTR CLASS:

TODO

  • scaricare tutti i dati (intra ed extra). portare su terra extra e processare intra per iniziare ad analizzare

    • proposta: due output, uno completo e uno tenendo conto solo extra
  • in verifica la classificazione 2007 sia corretta

    • una volta corretta verificare nei dati la completezza
  • rifare la classificazione italiana (di github). fare traduzione. esiste una ufficiale ISTAT, da ricevere

  • verificare sulla classificazione ricalcola il totale

  • nel nuovo dato non è più presente il TOT. da verificare se a livello di codice viene calcolato o preso dai dati. nel secondo caso, da inserire la logica di calcolo a livello di codice

[terra-update-batch] cambiare gli endpoint dei fili di trasporto, prodotto e classificazione

A seguito di una nota Eurostat, sono rilasciati nuovi endpoint. Di seguito la nota:

Important note: New Bulk URL in October
Users of the Comext Bulk download facility should take note that the Bulk URL will change in October. Here is the tentative time schedule:
02-06/10 (first week of October): the new URL https://ec.europa.eu/eurostat/api/dissemination/files will be activated;
02-31/10: Transition period; both URL endpoints will be available and distribute the same content without changes;
31/10: (approximate date): deactivation of the old Bulk URL;
01/11: (approximate date): the new Bulk URL will be the only active way to access the Comext data.

[terra-update-batch] ricostruire il file metadata.json

mantenere solamente l'header e il map.
Qui un esempio:

{
"processingDay": "31-01-2024, 11:31:44",
"annualCurrentYear": 2022,
"annualPreviousYear": 2021,
"lastLoadedData": "10, 2023",
"appVersion": "1.0.0",
"map": {
"timeSelected": "202310",
"timeStart": {
"year": 2018,
"month": 11
},
"timeEnd": {
"year": 2023,
"month": 10
}
}

[terra-backend/python server] cambiare i nomi delle chiavi payload

da
base_payload = {
"tg_perc": "80",
"listaMezzi": [0, 1, 2, 3, 4, 5, 7, 8, 9], # means of transport
"product": "TOT",
"flow": 1, #import = 1, #export = 2
"weight_flag": True,
"pos": "None",
"selezioneMezziEdges": "None"
}

a

base_payload = {
"percentage": "80",
"transports": [0, 1, 2, 3, 4, 5, 7, 8, 9], # means of transport
"product": "TOT",
"flow": 1, #import = 1, #export = 2
"weight": True,
"position": "None",
"edges": "None"
}

[terra-backend/python-server] costruire endpoint per generare date di start e end per i grafi

fare un solo endpoint che restituisce tutte le info:
Esempio:

info = [{
data: "extra"
meta: {
count: 1343434343
date_start: 202201
date_end: 202212
},
{
data: "extra_trim"
meta: {
count: 1343434343
date_start: 202201
date_end: 202212
},
{
data: "intra"
meta: {
count: 1343434343
date_start: .202201
date_end: 202212
},
{
data: "intra_trim"
meta: {
count: 1343434343
date_start: 202201
date_end: 202212
}]

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.