bluetab/td-dd

td-dd is a back-end service developed as part of /truedat project that supports the generation of a Data Dictionary

Truedat Data Dictionary

td-dd is a back-end service developed as part of Truedat project that provides API's for the following functionality:

• Data Catalog
• Data Lineage
• Connector Management
• Data Quality

Getting Started

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project on a live system.

Prerequisites

Install dependencies with mix deps.get

To start your Phoenix server:

Installing

• Create and migrate your database with mix ecto.create && mix ecto.migrate
• Start Phoenix endpoint with mix phx.server
• The td-dd API is published on localhost:4005
• The td-cx API is published on localhost:4008

Running the tests

Run all aplication tests with mix test

Environment variables

SSL conection

• DB_SSL: boolean value, to enable TSL config, by default is false.
• DB_SSL_CACERTFILE: path of the certification authority cert file "/path/to/ca.crt", required when DB_SSL is true.
• DB_SSL_VERSION: available versions are tlsv1.2, tlsv1.3 by default is tlsv1.2.

Elastic bulk page size configuration

• BULK_PAGE_SIZE_GRANTS: default 500
• BULK_PAGE_SIZE_IMPLEMENTATIONS: default 100
• BULK_PAGE_SIZE_JOBS: default 100
• BULK_PAGE_SIZE_RULES: default 100
• BULK_PAGE_SIZE_STRUCTURES: default 1000
• BULK_PAGE_SIZE_GRANT_REQUESTS: default 500

Store chunk size

• GRANT_STORE_CHUNK_SIZE: default 1000
• GRANT_REQUEST_STORE_CHUNK_SIZE: default 1000
• STRUCTURE_STORE_CHUNK_SIZE: default 1000
• DSV_STORE_CHUNK_SIZE: default 1000

If the variable is set to false, it will not be deleted in the case that there is no index in the hot swap process.

• DELETE_EXISTING_INDEX: detault true

Elastic Configuracion

The bulk_wait_interval variable defines the time interval between batches of bulk operations in Elasticsearch.

BULK_WAIT_INTERVAL_GRANTS: default 0

Elastic aggregations

• The aggregation variables are defined as follows: AGG_<AGGREGATION_NAME>_SIZE

Deployment

Ready to run in production? Please check our deployment guides.

Built With

• phoenix - A productive web framework
• ecto - Elixir toolkit for database integration
• postgrex - Elixir PostgreSQL driver
• gettext - Internationalization and localization support
• cowboy - An HTTP server for Erlang/OTP
• httpoison - An HTTP client
• credo - Static code analysis
• guardian - Authentication library
• bodyguard - Authorization library
• ex_machina - A factory library for test data
• cors_plug - Plug for CORS support
• phoenix_swagger - Swagger integration for Phoenix framework
• ex_json_schema - JSON Schema validator
• elasticsearch - Client for Elasticsearch
• vaultex - Client for HashiCorp Vault

Authors

• Bluetab Solutions Group, SL - Initial work - Bluetab

See also the list of contributors who participated in this project.

License

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see https://www.gnu.org/licenses/.

In order to use this software, it is necessary that, depending on the type of functionality that you want to obtain, it is assembled with other software whose license may be governed by other terms different than the GNU General Public License version 3 or later. In that case, it will be absolutely necessary that, in order to make a correct use of the software to be assembled, you give compliance with the rules of the concrete license (of Free Software or Open Source Software) of use in each case, as well as, where appropriate, obtaining of the permits that are necessary for these appropriate purposes.

td-dd's Issues

compile error

Compiling 401 files (.ex)

== Compilation error in file lib/td_dd_web/schema.ex ==
** (Absinthe.Schema.Error) Compilation failed:

Locations

/truedat/lib/td_dd_web/schema/structures.ex:28

The field "name" is not unique in type "DataStructureVersion".

The field must have a unique name within that Object type; no two fields may share the same name.

Locations

/truedat/lib/td_dd_web/schema/structures.ex:31

The field "name" is not unique in type "DataStructureVersion".

The field must have a unique name within that Object type; no two fields may share the same name.

(absinthe 1.7.0) lib/absinthe/schema.ex:366: Absinthe.Schema.__after_compile__/2
(stdlib 3.17.1) lists.erl:1267: :lists.foldl/3
(elixir 1.13.3) lib/kernel/parallel_compiler.ex:346: anonymous fn/5 in Kernel.ParallelCompiler.spawn_workers/7

Unknown failure: :nxdomain when running application.

Hello, I try to run the application and I am getting all the time the same error:

2020-02-07T10:45:46.290Z [error] pid=<0.1870.0> module=DBConnection.Connection Bolt.Sips.Protocol (#PID<0.1870.0>) failed to connect: ** (Bolt.Sips.Internals.Error) connect: Unknown failure: :nxdomain

The application seems that is running but in the console I can only see that error,

thanks in advance.

how to exprience truedat

Hello. Where are the install docs for /truedat? I found that user docs at https://docs.truedat.io/first-steps but there are no install docs that I could find.

I would like to try the open source version.

Truedat install.

Hello, could someone tell me if truedat data dictionary is an opensource or SaaS, if it is open source where you can download and install
Thanks.

https://www.truedat.io/

Docs?

Hello. Where are the install docs for /truedat? I found that user docs at https://docs.truedat.io/first-steps but there are no install docs that I could find.

I would like to try the open source version.