rafaelpierre/pyjaws

PyJaws: A Pythonic Way to Define Databricks Jobs and Workflows

pyjaws's Issues

project documentation

Support for Databricks SDK

Expected Behavior

Currently, we use Databricks Jobs and Workflows REST API. We should replace this by Databricks Python SDK. Benefits:

• Not having to adapt to potential changes in the API
• Removing dependency on Databricks CLI (which doesn't have 100% coverage at the moment)
• Removing the complexity associated with Jinja templating, which will help increase the coverage of PyJaws as well

Current Behavior

PyJaws integrates with Databricks REST API

Steps to Reproduce (for bugs)

Context

Your Environment

• pyjaws version used:

Add an option to export JSON / YAML

Expected Behavior

Should be possible to output the payload as JSON or YAML if needed

Current Behavior

Steps to Reproduce (for bugs)

Context

Your Environment

• pyjaws version used:

implicit context managers

Expected Behavior

Changing the way Cluster context managers work, so that we don't need to assign clusters to tasks inside the with Cluster(...) as cluster block

Current Behavior

Steps to Reproduce (for bugs)

Context

Your Environment

• pyjaws version used:

Airflow DAG converter

Expected Behavior

Users should be able to convert Airflow DAGs to PyJaws ones

Current Behavior

Steps to Reproduce (for bugs)

Context

Your Environment

• pyjaws version used:

context manager capabilities for cluster config

Expected Behavior

Should be possible to do something like: with Cluster(...) as cluster: and declare tasks within this with block. This would result in the cluster object being assigned to all tasks declared within this block.

Current Behavior

NA

Steps to Reproduce (for bugs)

Context

Your Environment

• pyjaws version used:

Ability to do `display(workflow)` from a Jupyter notebook and visualize a graphical representation of the workflow

Expected Behavior

Ability to do display(workflow) from a Jupyter notebook and visualize a graphical representation of the workflow

Current Behavior

NA

Steps to Reproduce (for bugs)

Context

Your Environment

• pyjaws version used:

Ability to define task dependencies with rshift and lshift (e.g. like in Airflow DAGs)

Expected Behavior

Ability to define task dependencies with rshift and lshift (e.g. like in Airflow DAGs)

Current Behavior

NA

Steps to Reproduce (for bugs)

Context

Your Environment

• pyjaws version used:

support for notebook tasks

increase test coverage

• test coverage is currently sitting at 77%, would be great to increase it to > 90%