guinsoolab/darkseal

A Single place to Discover, Collaborate, and Get your data right

Darkseal

Unlock the value of data assets with an end-to-end metadata platform that includes data discovery, governance, data quality, observability, and people collaboration.

Quickstart

wget https://raw.githubusercontent.com/GuinsooLab/darkseal/master/docker-compose.yml
docker-compose up -d

For more information, please referer to here.

Main Features

• Data Trust
• Data Documentation
• Data Governance
• Discovery & Collaboration
• Teams & Users with Roles and Policies
• Data Glossaries
• Data Insights
• Data Lineage
• Security & Compliance
• Alerts & Notifications

Core Components

Darkseal includes the following:

• Metadata schemas - Defines core abstractions and vocabulary for metadata with schemas for Types, Entities, Relationships between entities. This is the foundation of the Open Metadata Standard.
• Metadata store - Stores metadata graph that connects data assets, user, and tool generated metadata.
• Metadata APIs - For producing and consuming metadata built on schemas for User Interfaces and Integration of tools, systems, and services.
• Ingestion framework - A pluggable framework for integrating tools and ingesting metadata to the metadata store. Ingestion framework already supports well know data warehouses - Google BigQuery, Snowflake, Amazon Redshift, and Apache Hive, and databases - MySQL, Postgres, Oracle, MSSQL, and Guinsoo.
• Metadata User Interface - One single place for users to discover, and collaborate on all data.

Snapshot & Gifs

Database Service

Workflow DAG

Data Tables

Table Columns

Stream Lineage

Documentation & Supports

• Introduction 😋
• Overview
  • Tutorial
  • Features
  • Releases
  • Roadmap
• Deployment
  • Docker Deployment
  • Bare Metal Deployment
• Service Integrations
  • Database
  • Dashboard
  • Messaging
  • Pipeline
  • ML Model
  • Metadata
  • Ingestion
• Developers
  • Architecture
  • Contribute
  • Web Hooks
• Appendix
  • FAQs

Contributors

We ❤️ all contributions, big and small! Check out our CONTRIBUTING guide to get started and let us know how we can help.

Don't want to miss anything? Give the project a ⭐ 🚀

License

Darkseal is released under Apache License, Version 2.0

darkseal's Issues

Arrange `Darkseal` document v1.0.0

Execution default of goal org.codehaus.mojo:buildnumber-maven-plugin:1.4:create-metadata failed.: NullPointerException

[ERROR] Failed to execute goal org.codehaus.mojo:buildnumber-maven-plugin:1.4:create-metadata (default) on project catalog-rest-service: Execution default of goal org.codehaus.mojo:buildnumber-maven-plugin:1.4:create-metadata failed.: NullPointerException -> [Help 1]

Failed to find Darkseal - Managed Airflow APIs

To set up metadata extraction through UI, you first need to configure and connect to Airflow. For more details visit

Airflow Exception [Failed to test connection.] due to [Failed to get access_token. Please check AirflowConfiguration username, password].