MobyDQ is a tool for data engineering teams to automate data quality checks on their data pipeline, capture data quality issues and trigger alerts in case of anomaly, regardless of the data sources they use.
This tool has been inspired by an internal project developed at Ubisoft Entertainment in order to measure and improve the data quality of its Enterprise Data Platform. However, this open source version has been reworked to improve its design, simplify it and remove technical dependencies with commercial software.
Skip the bla bla and run your data quality indicators by following the Getting Started page. The complete documentation is also available on Github Pages: https://mobydq.github.io.
You can run all tests locally using the following commands:
$ cd mobydq
$ # Backend
$ test/run-tests.sh
$ # Frontend
$ app/run-container.sh npm run test
Depending on the used editor, eslint and pylint can be integrated. You can run all linters locally using the following commands:
$ cd mobydq
$ # Backend
$ test/run-linter.sh
$ # Frontend
$ app/run-container.sh npm run lint
The containers run by docker-compose
have dependencies with the following Docker images:
- postgres (tag: 11.0-alpine)
- graphile/postgraphile (tag: latest)
- python (tag: 3.6.6-alpine3.8)
- python (tag: 3.6.6-slim-stretch)
- nginx (tag: latest)
- docker (3.5.0)
- flask (1.0.2)
- flask_cors (3.0.6)
- flask_restplus (0.11.0)
- graphql_py (0.7.1)
- jinja2 (2.10.1)
- jwt (0.5.4)
- numpy (1.14.0)
- pandas (0.23.0)
- pyodbc (4.0.23)
- requests (2.20.0)
- requests_oauthlib(1.0.0)