Giter Site home page Giter Site logo

medialab / ricardo Goto Github PK

View Code? Open in Web Editor NEW
13.0 13.0 4.0 48.36 MB

RICardo Project, Historical Trade Database

Home Page: http://ricardo.medialab.sciences-po.fr/#/

License: GNU Affero General Public License v3.0

Python 5.09% CSS 16.38% HTML 12.56% JavaScript 58.79% Dockerfile 0.37% Shell 0.17% SCSS 6.65%

ricardo's Introduction

RICardo

RICardo Project, Historical Trade Database

This project contains 2 applications:

  • ricardo_api': a Flask application exposing a REST API serving data from the sqlite database in JSON format.
  • client: an HTML5 angular.js application displaying data vizualisations from the ricardo API.

The corresponding data-set is hosted at https://github.com/medialab/ricardo_data

Ricardo - Web Application (client)

Installation

If you want to run your instance of RICardo locally on your machine, be sure you have the following requirements installed.

Requirements

Installation steps

Clone RICardo from the command line:

$ git clone https://github.com/medialab/ricardo.git

Browse to RICardo root folder:

$ cd ricardo

Browse to RICardo client root folder:

$ cd ricardo/client

Install client-side dependencies:

$ npm install

Edit configuration file

$ cd src/js
$ cp config.sample.js config.js
$ vi config.js
# edit config.js and add the correct root to API
'use strict';

angular.module('ricardo')
  .constant('BASE_API_URL', 'http://localhost:5000')

You can now run RICardo with the following command :

$ npm start

Once this is running, go to http://localhost:8080/

You can build the RICardo application with the following command :

$ npm run build

The build will be available in the dist folder.

Angular app structure

Controllers are located under the src/js/controllers folder with one file per controller, and you should reference them in the file src/js/controllers/index.js.

Directives are located under the src/js/directives folder with one file per directive, and you should reference them in the file src/js/directives/index.js. Every chart component should have a directive.

Angular partials are located under the folder src/public/partials.

Api calls are done in the service src/js/services.js.

All files under the src/public folder are static.

CSS files must be referenced in the file src/js/style.js. The build process generates one CSS file from this JS file, and include it (with a style tag) in the index.html.

The same philosophy is applied to js dependencies, with the file src/js/external_dependencies.

Ricardo - API

Installation

If you want to run the RICardo API locally on your machine, be sure you have the following requirements installed.

Requirements

Database

The dataset needed to run this application is available in the ricardo data repository. You can follow the readme of the project to know how to build the database. It should produce a file called RICardo_viz.sqlite You must copy RICardo_viz.sqlite in the folderapi/ricardo_api.

You can change the path of the database by editing the api/ricardo_api/config.py if needed.

Installation steps

Go into the Api folder of the project

cd api

Install the server dependencies

pip install -r requirements.txt

Launch the server

python runserver.py

API structure (flask framework)

+api/
|__+ricardo_api
	|__ __init.py__ 					-> connection to database
	|__config.py 						-> information element of database
	|__models.py 						-> methods to get and transform datas from DB to JSON api responses
	|__views.py 						-> routes to serve datas
	|__RICardo_visualisation.sqlite

API responses

# RICentities
List all entities from the database

# reporting_entities
List entities which has reported Imp/Exp flows.

# flows
The main raw data API. flows will provide the exp/imp flows between countries.

# continent_flows
Flows aggregated by continent

# flows_sources
Provide a list of all sources used in a flow API call.
Used to get them all at once without repeating them in flow.

Docker

Run the stack with docker-compose

At the root of this project, there is a folder docker that contains a https://docs.docker.com/compose/ stack for the development.

You can run it by following those steps :

cd docker
docker-compose up

It will start for you the project stack with :

  • An nginx that serves the client on port 80
  • the API

In DEV mode (check the file .env), the auto-reload of the code is enabled.

Build the project images

The docker images of the project (ie. API & client) are built automatically by a gitlab process (see .gitlab-ci.yml). Those images are built with the Dockerfiles client/Dockerfile & api/Dockerfile.

To test those images on local before pushing to prod you can use the dedicated docker-compose which set the data version to use.

cd docker
docker-compose -f docker-compose.test-prod.yml -p ricardo-test-prod up --build

ricardo's People

Contributors

boogheta avatar diegantobass avatar gtible avatar jacomyal avatar jacomyma avatar jri-sp avatar mydu avatar paulgirard avatar pipojojo avatar sim51 avatar uf0 avatar yomguithereal avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

ricardo's Issues

World view - add continent as reporting

Add a continent mode where the user can visualise the continet trade to the rest of the world (World partner) : one curve by continent

We need #9 to be completed before implementing this.

Continent view

Add a continent view based on country view but with continent as reporting entities.
We need the new api wait for #9 to be completed

Add point to curve

add a data point on which one can rollover in curve to show the actual data point used to make the curves.
I think that might help but you can argue against I am not sure of that one.

Wolrd Entity and the various Total trade estimations

The total trade of one reporting entity to the World can be defined in different ways:

  • SUM : estimate trade tot he world as the sum of trade with the partners
  • TOTAL_REPORTING : use the total trade stastistics one can find in entities archives
  • TOTAL_PARTOF_REPORTING : same as TOTAL_REPORTING for politically dependent entities
  • TOTAL_ESTIMATED : use the total trade stastistics one can find in recent works (Mitchell, SAFC_US)

In RICardo we propose to use a mix-method to define the Total Trade.
In Country view, we will use the TOTAL_REPORTING or if not present TOTAL_PARTOF_REPORTING
In World view, we will use the TOTAL_ESTIMATED or if not present TOTAL_REPORTING

The SUM method is not used.

Deprecated information :

nouvelle colonne : total_type
valeurs possibles : null, reporting_total, estimated_total
Dans country view, on affiche le montant vers le partner World de type reporting_total
Dans World view, on affiche le montant vers le partner World de type estimated_total si il existe sinon on prend le reporting_total

"- Pour la country view (tous les partners + total des partners), on retient par défaut le code 0 (vide) dans 'Total trade Estimation' ou 1 s'il n'y a pas de données pour le code 0 (ce qui peut être souvent le cas). "

In country view we want to compare the total amounts obtained by summing all the partners imp/exp with the total declared in the database.

This total is in the database as flows to World Partner.
We should take the flow where 'Total trade Estimation' is null if exists otherwise where 'Total trade Estimation'=1

country view - add curves for a set of partner

After the user has choosen one reporting and had visualized the complete list of partners (as a treemap), he can switch to timeline subview where he can select a subset of partners using 4 different selectors :

  • country selector
  • colonial area selector
  • geographical area selector
  • continent selector

With those selector we can pick up to 5-10 (limit to decide) partner entities.

For eash entity in this selection one curve is represented (as in world view)

Country view - treemap for all partners flows

replace the list of partner (for the seleted reporting) in the country view by a treemap build in this hierarchy:
continent / entity / exp-imp flows

As done already the treempa should update if we change the time period

cosmetic World view

Main title

"Total trade of {selected country list comma separated} (aggregated), {smallest_date}-{biggest_date}, in £"

Curves title

"Total trade of {selected country list comma separated} (by country), {range_start_date}-{range_end_date}, in £"

labels

add a label on top of the 4 selectors : "select up to 5 reporting entity(ies)"

cosmetic changes for all views

Scale of curves

remove the unit abreviation letter (k,M,G) and add on top of scale the unit in full word (thousand, million, billion) plus the currency (£)

legend of curves

IMP = imports
EXP = exports

in table header:

replace country id by country name (I may do that in API if needed, to be discussed)
replace IMP, EXP as in curve legend
replace type by "partner type"

Order of columns :
reporting, partner,("partner type") year, imports, exports, total, currency

Ordered by

imports exports (with final s)

500 error

/ricardo_api/flows?partner_ids=926&reporting_ids=885

interface performance

Publish the interface commentting data tables and CSV export buttons features
this part might slow down the interface

[sprint-viz] - Roadmap

  • Merge with master
  • Drop second line of mirror flows in country view
  • Fix data table and download csv button
  • Make country select refresh the data in bilateral view
  • Fix cases without data in bilateral view

cosmetic on Continent view

~~ ## curve title~~
"Total Trade of {continent}, {smallest_date}-{biggest_date}"
Don't change dates when a smaller time range is selected. The curve stay in the total period so the title should always keep the total time range.

Title of bar chart viz

"Trade of {continent} by partner, {range_start_date}-{range_end_date}, in £"

before left-side country select add : "choose the reporting continent"

Add a scale to the barchart

Multi-curve viz title

"Trade of {continent} with {selected partner list comma separated}, {range_start_date}-{range_end_date}, in £"
(beware at some point he £ might change to national currency)

labels

add a label on top of the 4 selectors : "select up to 5 partner(s)"

nouvelle homepage

  • ajouter le logo dans le menu sur le bouton Ricardo (retour homepage)
  • on enlève le bandeau
  • on garde le titre "Ricardo project
    Trade between Nations from c. 1800 to 1938" auquel on ajoute le logo
  • on enlève le footer
  • on ajoute 4 blocs de présentation des pages viz
  • on garde le paragraphe How to cite
  • on ajoute un paragraphe Open Science avec code source et lien github data
  • ajouter des mentions légales
  • passer le vocabulaire dans une page dédiée
  • passer les deux longs paragraphes de description dans la page ABOUT
  • Ajout d'une liste des titres + excerpt des 5 derniers billets du blog récupéré par RSS

API - add a flow_grouped_by_continent api root

for the continent view :
Get all the partners for a continent aggregrating flows by continent and removing partners from this continent (remove internal flows).

/continent_flows?continents=Europe

for the world view :
Get all exchanges between one continent and the World partner by aggregating all reporting by continent.

/continent_flows?continents=Europe&partner_ids=1382

add comments on the metadata graphics

(or more generally on all the graphics, but I have suggestions only for the metadata)

For instance, it is puzzling (to me, who do not know the period well) why so few registers exist during the 1920s. I would have imagined that states would have stopped recording exchanges during the Great War, but is does not seem to be the case. Why did most of them stop just afterwards?

image

Is this just an artifact? If yes, it would be nice to have it mentioned. If no, it would be nice to know what the current knowledge is about such striking features. I would think of a floating hand-written-like arrow-accompanied comment like this (source):
image

It would make the graph more interesting to read, more thought-provoking, and would encourage the reader to explore. Some specific views could contain specific comments. On the long term, you could allow your readers to suggest additional comments by mail.

add dependency tree for reporting entities in metadata view

On this view, it is not clear what is a country, what is a town, what is a state inside a country...

image

It looks like some dependency tree / comments would help to understand the link between the entities. For instance, is Algeria is included into the French statistics until the independence? Is Hamburg included in Germany?

I am not sure what is the best way to visualize it... a color scheme? (a color for an independent state, an other for dependent region, and so on until the last level of the hierarchy is reached) But this does not help to know which entity a given sub-entity belongs to. A tree scheme? But the problem is that some countries may have been dependent at some point and have become independent (or the reverse). Icons? That could help to tell appart towns from provinces from countries... but what about a state-city?

Do you have any suggestions? I have no doubts these are important meta-data one has to represent in some way.

Filter by presence of data

In :
partner selector in bilateral view
partner selector bottom of country view

it would be nice to only display entities in the select form that actually have data to display.

Ordered by name

in country view, in partner list on the right side, add a "ordered by name" option to the existing "ordered by" selectro

Table data

in table header:

replace country id by country name (I may do that in API if needed, to be discussed)
replace IMP, EXP as in curve legend
replace type by "partner type"

Order of columns :
reporting, partner,("partner type") year, imports, exports, total, currency,source

The data table should be accessible by user on demand only.
In a tab ?

API should return null flows

API should return null flows even when both imp and exp are null to maintain a coplete set of data for a time frame.
It's ok to drop continuous null values at the beginning or at the end of a time frame.

Sources table

Add an API routes which outputs a csv file which lists all sources :

  • merge sources and sourceTypes tables
  • remove slug and acronym
  • add a bibliographic reference :
    source.name source.country source.dates. source.author. source.edition_date. source.volume. source.pages. sourceType.URL
    author/name/country/vol_number/vol_date/edition_date/author_editor/pages/source_category/type

country view

curve title

"Total Trade of {country1}, {smallest_date}-{biggest_date}"
Don't change dates when a smaller time range is selected. The curve stay in the total period so the title should always keep the total time range.

Title of bar chart viz

"Trade of {country1} by partner, {range_start_date}-{range_end_date}, in £"
(beware at some point he £ might change to national currency)

before left-side country select add : "choose the reporting country"

Add a scale to the barchart

Multi-curve viz title

"Trade of {country1} with {selected partner list comma separated}, {range_start_date}-{range_end_date}, in £"
(beware at some point he £ might change to national currency)

labels

add a label on top of the 4 selectors : "select up to 5 partner(s)"

menu/pages

  • enlever "view" des noms de pages existants

  • ajout d'une page À propos/About :

    • présentation plus longue du projet (contenus existant dans la home)

    • description de l'équipe

    • description du comité scientifique

    • ajouter le logo ANR avec la référence " Ce travail a bénéficié d’une aide de l’Agence National de la Recherche portant la référence RICARDO ANR-06-BLAN-0332"
      " This work ahs been supported by l’Agence National de la Recherche under the reference RICARDO ANR-06-BLAN-0332"

    • ajouter les logo médialab et centre d'histoire

  • page glossaire

    • le glossaire déjà présent en home
  • une page dataset/corpus

    • entités commerciales : paragraphe de présentation + lien de téléchargement d'un fichier tableur
    • sources : paragraphe de présentation + lien de téléchargement de la table
    • taux de change : paragraphe de présentation + lien de téléchargement de la table
    • base de données : paragraphe de présentation + lien vers github.com

World view - timeline curve bug

The global timeline should be the total trade of all selected reporting countries and not the first one entered.
When adding new reporting countries the global timeline should be updated.

cosmetic on bilateral

curve title

"Trade Flows between {country1} and {country2}, 1857-1938, reported by {country1}"
Don't change dates when a smaller time range is selected. The curve stay in the total period so the title should always keep the total time range.

title of bilateral schema

"Mirror flows between {country1} and {country2}, {range_start_date}-{range_end_date}, in £"

labels

label to add on top of left-side flow chart : "reported by {country1}"
label to add on top of right-side flow chart : "reported by {country2}"

before left-side country select add : "choose the reporting country"
before right-end country select add : "choose a partner country"

rest of the world partner

for every year for every reporting we could attribute the differences between the total flow as reported and the sum of flow with partners to a virtual rest of the world partner.

country view - barchart enhancement

make entities name with white background to avoid import/export total amount to be surimpressed with entity name when entity name too big.

when filtering, remove the entity filtered out (don't grey it).

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.