yarosj/prestige-of-districts

:mag_right: This application parses sites and retrieves data associated with failures of public services to display districts' prestige

Navigation:

• Dependencies
• How to use
• Configuration
• Documentation
• About

Dependencies:

This application requires docker version 18.06.3-ce or higher and docker-compose version 1.24.0 or higher:

• How to install Docker read there.
• How to install Docker Compose read there.

If you want to edit client app you need in npm version 5.5.1 or higher, Node.js version 8.11.2 or higher and Webpack version 3.2.1 or higher. Then you can install dependencies. To do this, you need to go to the directory /view and enter the following command in the terminal:

$ npm i

How to use:

To start app you need in running Docker.

Before starting you need to build app:

$ docker-compose build

Then start up your application:

$ docker-compose up

Important! If you run docker with sudo you should run docker-compose with sudo too.

Enter http://localhost:5000 in a browser to see the application running.

If you want to explore GraphQL API with GraphiQL enter http://localhost:5000/graphql.

If you want to watch interactive graph of GraphQL API enter http://localhost:5000/voyager.

If you have trouble with ports you can change them in docker-compose.yml.

---

If you want to edit client you can run client app with webpack-dev-server:

$ npm start

Or if you want to build client app you should enter:

$ npm run build

Configuration

Config variables can be passed in an docker-compose.yml.

Documentation

API documentation you can find here.

About

This application is written on the Node.js platform (Express.js + ApolloServer). Learn more about Apollo here.

MongoDB is used as a database. Redis is used for caching on server.

For client-server communication uses GraphQL - query language with open source software, developed by Facebook.

For “web scraping” uses the Puppeteer - “Headless Google Chrome”.

A multi-protocol message broker - RabbitMQ is used to communicate between the Node.js and Python processes.

For processing, classifying and extracting entities from the received text, SpaCy library in python is used. This library includes convolutional neural network models for tagging parts of speech, analyzing dependencies, and recognizing named entities (NER). SpaCy package structure is displayed on image 1. Learn more about SpaCy architecture you can there.

Image 1 - SpaCy package structure.

Since the Ukrainian and Russian languages ​​are not supported, the model was trained on the data of municipal services of the city of Kramatorsk.

The result of the text analysis (extraction of entities) is transmitted through the message broker to the geocoder where it finds the coordinates of the entities (locations)

Results are displayed on the heatmap (Image 2), pointmap (Image 3) and 3D heatmap on the client.

Image 2 - Pointmap example.

Image 3 - Heatmap example.

The client application is written in ReactJS, using ApolloGQL to communicate with the server. Failures are displayed on maps (point map heat map and 3D heatmap). For maps, the MapboxGl and DeckGl libraries are used. As the UI component Library is used SemanticUI.

License

MIT License. Copyright 2019 Yaroslaw Zhuk

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the 'Software'), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.