openspending/babbage.ui

Bindings for Angular applications to use Cubes Slicer API

OpenSpending

OpenSpending is a project to make government finances easier to explore and understand. It started out as "Where does my money go", a platform to visualize the United Kingdom's state finance, but has been renamed and restructured to allow arbitrary financial data to be loaded and displayed.

The main use for the software is the site openspending.org which aims to track government finance around the world.

OpenSpending's code is licensed under the GNU Affero Licence except where otherwise indicated. A copy of this licence is available in the file LICENSE.txt.

OpenSpending is a microservices platform made up of a number of separate apps, each maintained in their own git repository. This repository contains docker-compose files that can be used run an instance of Openspending for development, or as the basis for a production deployment. This repository also acts as a central hub for managing issues for the entire platform.

What are these files?

Most applications that make up the OpenSpending platform are maintained in their own repositories, with their own Dockerfiles, built and pushed to the OpenSpending organisation on Docker Hub:

• os-api
• os-conductor
• os-viewer
• os-explorer
• os-admin
• os-packager
• os-fdp-adapters
• os-data-importers

This repository maintains docker-compose files used to help get you started with the platform.

docker-compose.base.yml: This is the main docker-compose file for OpenSpending specific services. All installations will use this as the basis for running the platform.

docker-compose.dev-services.yml: This defines backing services used by the platform, such as Redis, ElasticSearch, and PostgreSQL. This file also includes fake-s3 in place of AWS S3, so you don't have to set up an S3 bucket for development. It is not recommended to use this for production.

docker-compose.data-importers.yml: This defines the services used for the separate os-data-importers application. They depend on services defined in docker-compose.dev-services.yml. Unless you are working on the data-importers or its associated source-spec files, it's not necessary to run this file.

docker-compose.local.yml: Create this file to add additional services, or overrides for the base configuration. It is ignored by git.

Dockerfiles/*: Most services are maintained in their own repositories, but a few small custom services used by the platform are maintained here. os-nginx-frontend is a basic frontend nginx server and configuration files to define resource locations for the platform. This will be build and run directly by docker-compose.base.yml.

I'm a developer, how can I start working on OpenSpending?

1. Define the environmental variables that applications in the platform need. The easiest way to do this is to create a .env file (use .env.example as a template).

2. Use docker-compose up to start the platform from the base, dev-services, and optionally local compose files:

$ docker-compose -f docker-compose.base.yml -f docker-compose.dev-services.yml [-f docker-compose.local.yml] up

3. Open localhost:8080 in your browser.

I'm a developer, how can I work on a specific OpenSpending application? Show me an example!

You can use volumes to map local files from the host to application files in the docker containers. For example, say you're working on OS-Conductor, you'll add an override service to docker-compose.local.yml (create this file if necessary).

1. Checkout the os-conductor code from https://github.com/openspending/os-conductor into ~/src/dockerfiles/os-conductor on your local machine.
2. Add the following to docker-compose.local.yml:

version: "3.4"

services:
  os-conductor:
    environment:
      # Force python not to use cached bytecode
      PYTHONDONTWRITEBYTECODE:
    # Override CMD and send `--reload` flag for os-conductor's gunicorn server
    command: /startup.sh --reload
    # Map local os-conductor app files to /app in container
    volumes:
      - ~/src/dockerfiles/os-conductor:/app

3. Start up the platform with base, dev-services, and your local compose file:

$ docker-compose -f docker-compose.base.yml -f docker-compose.dev-services.yml -f docker-compose.local.yml up

Now you can start working on os-conductor application files in ~/src/dockerfiles/os-conductor and changes will reload the server in the Docker container.

I want to work on the data-importers application. Show me how!

In Openspending, the os-data-importers application provides a way to import data and create fiscal datapackages from source-spec files. You can either work on the app independently, by following the README in the os-data-importers repository, or within the context of an Openspending instance, by using the included docker-compose.data-importers.yml file, and starting Openspending with:

$ docker-compose -f docker-compose.base.yml -f docker-compose.dev-services.yml -f docker-compose.data-importers.yml up

This will start Openspending locally as usual on port :8080, and the pipelines dashboard will be available on port :5000: http://localhost:5000.

I have my own backing service I want to use for development

That's fine, just add the relevant resource locator to the .env file. E.g., you're using a third-party ElasticSearch server:

OS_ELASTICSEARCH_ADDRESS=https://my-elasticsearch-provider.com/my-es-instance:9200

I want to run my own instance of OpenSpending in production

Great! There are many ways to orchestrate Docker containers in a network. E.g. for openspending.org we use Kubernetes. Use the docker-compose.base.yml file as a guide for networking the applications together, with their appropriate environment variables, and add resource locators pointing to your backing services for Postgres, ElasticSearch, Redis, memcached, AWS S3 etc. See the .env.example file for the required env vars you'll need to set up.

You'll also need to set up OAuth credentials for OS-Conductor (see https://github.com/openspending/os-conductor#oauth-credentials), and AWS S3 bucket details.

What happened to the old version of OpenSpending?

You can find the old OpenSpending v2, and the complete history for the codebase to that point, in the openspending-monolith branch.

babbage.ui's Issues

CORS error in Chrome

http://cosmopolitan.openspending.org/v1/polygons/country:MD?format=json. The 'Access-Control-Allow-Origin' header has a value 'http://localhost:63342' that is not equal to the supplied origin. Origin 'http://os-viewer.herokuapp.com' is therefore not allowed access. Have the server send the header with a valid value, or, if an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

Sample API results link in README.md gives an error

Deal with crosstab pagination properly

The crosstab isn't paginated, but it is limited to 10000 cells by the server API. If that limit is exceeded, the results will not just stop, but actively become inconsistent, depending on the sort. This should be clearly indicated by a warning box on the page when there are more than 10k results available.

filter by key but present label

Why does it filter by the label of a dimension when the key_attribute is defined? I would have figured that the idea is to present the label to the user but use the associated key to do the filter.

Number formatting based on locale

It seems that currently numbers are formatted in the US/English format with d3.format:

babbage.ui/src/components/utils.js

Line 4 in 36fa6e2

export var numberFormat = d3.format('0,000');

In German (and probably other languages) comma and dot are used differently, i.e. the thousand separator is the dot . and comma , is used to separate the decimal place:

EN: 1,000,000.23
DE: 1.000.000,23

d3.format has locale functions, but I have not used it myself and I am not familiar with babbage.ui's code https://github.com/d3/d3-format#formatLocale

Naive solution ideas/questions:

• Would the line in utils.js be a good starting point for implementing this?
• Does it need to be added to the translations in os-viewer and applied dynamically in some way? (or just use browser language defaults?)

Babbage UI as modular components

Description

We want to guide the great work done in Babbage UI to date towards a more modular component architecture.

Ideally, consumers of Babbage UI should be able to include components that they want into their code, such as, the Babbage Treemap, without any additional baggage.

Of course, Consumers should also be able to simply bring the whole kitchen sink as a UI/Vis toolkit for apps that require multiple views on data provided by an instance of a Babbage API.

Additionally, modular Javascript often (usually) means writing with CommonJS or ES6 for a proper module implementation, and building distributions for use directly in the browser, or as importable libs in other CommonJS apps.

Babbage is currently written as a set of Angular 1. x directives, and is not currently usable as a modular "component library". This leads to a final reason why a move to modular components can be desirable: It moves towards conceptual alignment with the majority of modern Javascript frameworks, including Angular 2, and it is hoped that this will make a future port to Angular 2 less painful as we evolve the Babbage ecosystem.

Desired output for this work:

1. Use Webpack for building distributions
2. Publish the library as CommonJS and Browser distribution to npm, as well as the browser distribution to Bower
3. Each Visualisation, or View, as a distinct, self-contained component
4. Ability to import / require a single component. This would look something like the React-D3 bindings used here, here
5. Optional, but strongly preferred by @pwalsh - Write src in ES6, which is transpiled to lib as ES5 - following the common pattern used these days for distributing ES6 code.

In any event, this can be done in stages, and the first, immediate goal would be the most minimal refactoring required to the existing code to be able to export the current directives for each visualisation as components.

Additionally, babbage.ui currently has some code that is not related to using babbage.ui as library of components, such as the S3 config in Grunt. My suggestion would be that we remove anything from babbage.ui that is not strictly required for creating the various visualisation widgets, and that this removed config and code could be pushed to a higher-level codebase that itself depends on babbage.ui as a component library.

References for the refactor:

• http://teropa.info/blog/2015/10/18/refactoring-angular-apps-to-components.html
• https://www.airpair.com/angularjs/posts/component-based-angularjs-directives

Work to be done in the feature/modern repository.

Tasks

• TBD
• Basic test coverage

Babbage UI page are zero-based, but the babbage API uses 1-based

I've got a patch for this to align with the Babbage API, coming up shortly.

Implement bar charts and line charts

I would propose to use angular-nvd3 for that purpose, so that we don't have to re-invent the wheel wrt. axis labelling and stuff like that.

Sankey labels are gone

For some reason, the labels in the Sankey visualisation aren't shown any more.

@holgerd77 you seemed interested to hack on this a bit. This would seem like a really neat way to get started. Any interest?

Firefox does not cache geo-json requests

browser sends the same request every time. So it increase loading time.

sometimes, Map view does not show paint regions by selected measure

Implement a sankey view.

I quite like @Yannael's sankey demo based on TED data: http://yleborgne.net/opented.

In cubes, this could be accomplished by doing a drilldown on both axes (the sender and the recipient) and then using cells as edges. Should be easy and the demo proves it can actually raise interesting questions wrt. spend data :)

Should we manage state independently of the URL?

At the moment, each change to the filter/query state will be written to the URL as a query string. This, in turn, will trigger angular to reload the route and thus apply the state. While this makes sure that URL state is always synched and that unused directives are disposed of fully, it means a full reload in many cases where that could have happend async, with a less shaky UX.

Thoughts?

Make sure views dispose fully

Right now, each view will register a callback with the $rootScope, which means they might have a pending reference even if they are no longer present in the DOM. We need to make sure that this doesn't mean they stick around in some sneaky way. This would become particularly important with #3.

Add sorting API for charts

Add the ability to change the sorting of the data the chart is using. Create an API to pass dimension/measure names and 'desc' / 'asc' to indicate the direction of the sort.

getJson and get function in api component needs to be more robust in handling errors

When hitting error (500) to OSNext API; the BabbageUI api component does not handle it gracefully:

index.js?25f5:17 GET http://next.openspending.org/api/3/cubes/23cdc48554ae8648deff7837c025d8c0:jpm2015/aggregate?drilldown=administrative_classification_3.Level_2%7Cadministrative_classification_3.Level_2_Label&pagesize=30&cut=administrative_classification_2.Level_1%3A%22%221%22%22&order=Amount.sum%3Adesc 
500 (INTERNAL SERVER ERROR)

One possible implementation for a fix:

diff --git a/src/components/BabbageUI/babbage.ui/src/api/index.js b/src/components/BabbageUI/babbage.ui/src/api/index.js
--- a/src/components/BabbageUI/babbage.ui/src/api/index.js
+++ b/src/components/BabbageUI/babbage.ui/src/api/index.js
@@ -15,15 +15,22 @@ export class Api {
       return Promise.resolve(this.cache[url]);
     } else {
       return fetch(url).then(function(response) {
-        return response.text()
+        if (response.status >= 400) {
+          return Promise.reject(new Error(`
+            code: ${response.status},
+            message: ${response.statusText}
+          `));
+        } else {
+          return response.text()
+        }
       }).then(function(text) {
         return that.cache[url] = text
-      });
+      }).catch(Promise.reject);
     }
   }

   getJson(url) {
-    return this.get(url).then(JSON.parse);
+    return this.get(url).then(JSON.parse).catch(function(err) { console.error("getJson:", err)});
   }

   flush() {

Intended as frontend for Babbage AE only, or for Cubes as well?

Since this project was apparently renamed at some point from "angular-cubes", I would like to know whether it is still intended to support Cubes as a backend, or if it is now for Babbage Analytical Engine only.

String cut values should be quoted with double quotes

See openspending/babbage#11

Gitter for chat?

Hey @pudo

Would you mind if we add the new OpenSpending Chat room on Gitter to the README here? It is a test to see if Gitter works for us

Implement better cuts and filter dialoge

Currently, filters can only filter on precise values, not on text searches, or simple math formulae like "greater than", "smaller than". I would like to support such filtering, but the backend doesn't have such query flags, since it's focussed on allowing pre-aggregation. We would probably need some support for this upstream, even if via a plugin to cubes.

Imagined operators:

• cut=loan.age:>40
• supplier.name:~Booz Allen*

Any thoughts, @Stiivi?