View Code? Open in Web Editor
NEW
A complete development environment for Azure DataBricks workflows. This solution includes a development container with all the required tooling to connect, test, and deploy python based DataBrick workflows.
License: MIT License
Dockerfile 37.03%
Batchfile 1.07%
Shell 1.45%
Makefile 4.79%
Python 41.66%
Roff 14.00%
masonry's People
Contributors
Watchers
masonry's Issues
The codebase should have an initial README file with basic information, any code check badges should be added.
The codebase should have linter set up before the first code pushes
Add documentation about workflow:
Currently, we do not have a path for connection to clusters behind private VNETs or VPNs.
This issue should be closed when the following is completed.
Optional tasks to be completed based on the findings above:
Add a simple example to demonstrate the workflow/usage of Masonry
Need a new python library creation steps or script that creates a new library "project" that is connected with make file and other requirements.
Add the existing code base to the repo, stripping away any project specifics
The stub project(s) are currently implemented via a Git clone call. As a backup, a local (included) archive could be added.
To close this issue, the following must be completed:
We need a way to populate the cluster variables without manually setting them in the configuration file.
To bring the repo up to date and make it easier to navigate, the following needs to be completed:
A simple script to create or if possible, retrieve information about Azure Databricks cluster(s)
To round out the end-to-end testing scenario, Nutter should be included as part of the base, and an example of usage should be supplied.
To close this issue, the following must be completed: