aws-samples/data-lineage-for-data-lake-example

Data Lineage for Data Lakes Example

This repository contains an example project for building Data Lineage for data lakes using AWS Glue, Amazon Neptune and Spline Agent.

Remarks: This setup works with AWS Glue Data Permissions Model and does not support Lake Formation Permission Model.

Stack

• API: HTTP API powered by an API Gateway and lambda functions
• Frontend: Vue.js application

Build & Deployment

To deploy the solution to AWS Cloud with terraform, export your AWS Credentials to terraform (AWS Profile or environment variables)

brew install terraform
git clone https://github.com/aws-samples/data-lineage-for-data-lake-example.git
cd data-lineage-for-data-lake-example

# download spline agent jar
wget https://repo1.maven.org/maven2/za/co/absa/spline/agent/spark/spark-3.1-spline-agent-bundle_2.12/0.6.1/spark-3.1-spline-agent-bundle_2.12-0.6.1.jar -O ./asset/lib/spark-3.1-spline-agent-bundle_2.12-0.6.1.jar

terraform init

terraform apply

To build and test the lineage visual application locally:

• update the lineage backend address in src/lineage-visual/src/main.js

axios.defaults.baseURL = "https://xxx.execute-api.<aws-region>.amazonaws.com/dev";

cd src/lineage-visual
npm install
npm run serve

Getting started

• Run the first Glue Job

aws glue start-job-run --job-name "RawToCurated_employee_optimize"

• Open the lineage visual application
• Run the second Glue Job

aws glue start-job-run --job-name "CuratedToAggregated_employee"

• Refresh the lineage visual application

Security

See CONTRIBUTING for more information.

License

This library is licensed under the MIT-0 License. See the LICENSE file.

data-lineage-for-data-lake-example's Issues

lineage visual application not showing graph

I have encountered a couple of problems.

First, I got the "Invalid Host Header" error.
I followed instructions here to fix it.
gitpod-io/gitpod#26

Then, it just displays a blank page.

No graph is shown when I click any of the links at the top of the page.

I verified the API call is working fine.

For some reason, the page is just not rendering the API responses.
What did I miss?

Alternative to Docker for deploying the demo

For user who doesn't want to or can't use Docker for deploying the demo, finch can be used instead by replacing 'docker' by 'finch' in the lineage_api.tf at line 21.

The replacing block should look like this :

resource "null_resource" "ecr_image" {
  triggers = {
    src_hash = data.archive_file.lambda.output_sha
  }
  provisioner "local-exec" {
    command = <<EOF
      aws ecr get-login-password --region ${local.region} \
        | finch login --username AWS --password-stdin ${local.account_id}.dkr.ecr.${local.region}.amazonaws.com
      cd ${path.module}/src/lambda
      finch build -t ${aws_ecr_repository.repo.repository_url}:${local.ecr_image_tag} --platform=linux/amd64 .
      finch push ${aws_ecr_repository.repo.repository_url}:${local.ecr_image_tag}
    EOF
  }
}

The Finch service has to be launched first by the following command :
finch vm init

Error: error creating Neptune Cluster: DBClusterParameterGroupNotFound: DBClusterParam

Using Terraform version 1.3.5, I receive the following error when attempting to run terraform apply: