hasadna/open_pension

:chart_with_upwards_trend: Open Pension is a "Hasadna" project, that aimed to revealing the secrets behind the Israeli pension market.

Open Pension

Open pension is a project which aims to provide an easy way for people to explore where the money from the pension funds invested is invested. The project belong to the "Hasadna" project - an NPO which thrives for clreance and opening the data.

Revealing where the money is invested and by whom helps to find out which hidden motives play roles in managing the money.

Project structure

The project uses a microservice's architecture:

Front

This is the site which allow the users to watch where the money is invested, graphs to present data, links to the articles and more. The service uses NextJS which help us to use static pages when needed and SSR when need to display live content (like graphs and more).

Monthly

The monthly process XML files and process them to data which we can process later on. The data is processed on a MySQL server. We're using NodeJS, GraphQL and Prisma.io for that.

Storage

Since we don't know where we're going to host the service (AWS, google cloud, azure...), the service receives files, or listen to kafka events for download files from other services, and store them on a physical drive. Each file get a storage ID which helps us to track where the file was stored or what data which processed is relates to. The service is written in go and use MySQL for store the data.

Backoffice

The service help us to orchestrate the other services: upload new files for processing, create users, manage content and more. The service is using react for the content, mongo for storing users and other data and allow us to see how a file was processed.

Backoffice backend

This is the backend for the backoffice. Written in NodeJS, GraphQL, MongoDB.

Pre requirements:

You'll need to install:

• Docker
• nvm - Most of the servies works with node version of 14.17.3
• Go lang

Start working with the services:

The docker machine are already ready for development. You can have two options:

Turn on all the services

This will fire up all the services. This one is a good options if you need to run them all:

docker-compose up -d```  
  
### Turn on a specific service:  
There are a couple of services that we are handling: `monthly`, `storage`, `monthly`, `backoffice`, `backoffice_bakend`  
  
For that all you need to do:  
```bash  
docker-compose up -d monthly # But you can go with the other service which mentioned above.

Developing the front

For now, the front has a couple of technical issues to be developed under a docker instance. So you need to go to the front folder and install. It's basically goes like this:

cd front
npm inpm run dev

Data seed

After everything is all set up, you need to have two thing: a user to admin and some dummy data in the monthly DB.

Creating user

login to the backoffice backend:

docker-compose up exec backoffice_backend bashnpm run cli:createUser

You would then go through questions for the credentials of the user. Remember, this must be a very strong password.

Seeding the monthly DB

Unlike the admin user, the seeding data is good for playing around with the monthly service. It's not a must but good
when we need to develop the project:

docker-compose up exec monthly bashnpm run dummySeed

Accessing the services:

• backoffice - http://backoffice.localhost/
• backoffice_backend - http://backoffice-api.localhost/
• monthly - http://monthly.localhost/
• storage - http://storage.localhost

Cheatsheets:

Here are some couple of command lines snippets which can help you work with the services:

Kafka

./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --whitelist '.*'./bin/kafka-topics.sh --bootstrap-server localhost:9092 --list

Debugging

If you need to debug, you can run your local backend with the next mongo address:
mongodb://localhost:27017/open_pesion_dashboard

open_pension's Issues

Fix the router.

• Upgrade to RC3.
• Fix the router.
• Add a child route for managing-bodies component.

Create Models

Integrate Webpack

And move to Angular 2.0 RC

Pull data and display it.

Pull the data from django db, create endpoint, and show it in angular view.

Add privacy policy link

Add a link for that in the footer.
The "privacy policy" text should be like in http://www.openpension.org.il/#/privacy

Finish Docker for frontend

Handling CORS

Can't do a get request from angular, get a cross-origin error.

Create hero header component.

Show search component inside it.

Chart skeleton with d3.js

http://bl.ocks.org/kerryrodden/7090426

https://www.jasondavies.com/coffee-wheel/

Add SASS loader to webpack.

Reference: https://github.com/jtangelder/sass-loader

Review models.

@ranmoshe Review the models, check that there are represent the entities that moshe describe here: https://drive.google.com/drive/u/0/folders/0B7RkqHKoREPqRnZiRnZoWFF1ek0

Child router.

The router of /managingbody/:id should be a child route in managing-body component. Now he is at app.ts.

Reference: http://a2-first-look.azurewebsites.net/examples/router-child/plnkr.demo.html?bust=1463687660843
Look at app/app.component.ts file, you'll see the main path for characters (for example) and then look at app/characters/characters.component.ts file, you'll see the child router for the characters path.

API version

We need a versioning for our api endpoints.
Reference: http://www.django-rest-framework.org/api-guide/versioning/

The client will pass the version number in the header.
Reference: http://www.django-rest-framework.org/api-guide/versioning/#acceptheaderversioning

Create managing-body component

Adding semantic ui front end.

https://cdnjs.cloudflare.com/ajax/libs/semantic-ui/2.1.8/semantic.min.css
https://cdnjs.cloudflare.com/ajax/libs/semantic-ui/2.1.8/semantic.min.js

Research css architecture options

in this tutorial they include the component's styling in a file that they import in the component:

I like the idea, it makes the idea of reusable components stronger
Do we want to work like this too?

they also use something like shadow css (i think.... or what is the :host selector)
would be nice if someone can research this option

Client task runner (gulp)

Edit humans and cleanup requirements.

Fix managing body, Home, About components.

Create a simple test for angular.

• Create a unit test with jasmine.
• Create an e2e test with protractor.

Semantic-ui check

Create a pycodestyle test.

Reference: https://pypi.python.org/pypi/pycodestyle/1.8.0.dev0

Refactor ManagingBody model.

• Move the ManagingBody model to a separate file (from ManagingBody.service).
• Change it from export interface.. to export class...

Finish create Docker.

• Container for the database - PostgreSQL.
• Container for the back end - Django.
• Container for the front end - Angular 2.
• Container for the server - Nginx.

Refactor to use Angular-cli

Reference: https://github.com/angular/angular-cli

Tests.

Write a test to check the index.html page of Django (that's just for the sake of start small, we'll write proper tests in the future).

Reference: https://docs.djangoproject.com/en/1.9/topics/testing/

Import dummy data

.js and map.js files in seperate /build folder

Create op-search-result component

Reference: link

Managing Body summary endpoint for the landing page

Refactor ManagingBody.

• Move managing-body-list, managing-body, managing-body-detail, to one directory called managing-bodies.

Handle translation on the managing body endpoint

Since the label field is now translated on the Managing Body model, we should return all the available translations on the endpoint.

@nirgn975
Not sure if we should return all translations or just the relevant one dependent on the user request ...

Create op-search-result-list component

Reference: link

Create managing-body-list Component

Create a wiki

Write a wiki with Moses, and docs for the models.

Miscellaneous

• In every component, delete the element selectors we don't use.
• Delete the provider from managing-body-list and add it to the app.ts.
• move the managing-body-list component from the component directory to the app directory (delete the components directory).
• Add a space between the import of the components (like from {Observable} to { Observable }) In link
• Make sure every component selector start with op-.

Models translation support

We should support translations for models with textual fields.

Angular.js 2 basic components

Create a basic Angular 2.x app

Create issue and PR templates.

Add text for the about page.

http://www.openpension.org.il/#/about

Add the LICENSE file with appropriate copyright

Just the MIT file with the appropriate year, name and email

pi managing body endpoint.

https://github.com/d3/d3/wiki/Partition-Layout

Get all the data for the managing body endpoint.

• Total amount in a pension.
• The percentage from the total amount in all the pensions.
• The percentage changed from the last 4 months.

Add i18n_patterns.

Add i18n for Angular.

Reference: https://github.com/angular/i18n

Create managing-body-detail component

Create a "Send Feedback" link.

Create a "send feedback" Popup (/Modal dialog) with a name, email and text inputs, for people send us email with feedback about the website.
Don't forget to add there a "Send" button to send it to us.

Just create the dialog component and the logic to pop it up, don't worry about sending an email, we'll handle it later (in another issue).

Refactor to use moduleId

• Use moduleId: module.id, as the first selector in each component.
• Remove the app/... from the templateUrl, etc..

i18n infrastructure

We need to make sure the infrastructure for i18n is good to go.

Integrate Travis-CI