Add DatabricksCluster class

[dask-contrib]

Add DatabricksCluster class

[benrutter]

I think scaling is theoretically possible with something like this: https://docs.databricks.com/api/workspace/clusters#resize I guess the tricky bits are:

• Additional access tokens etc required to manage cluster with the api
• Not accidently giving people footguns to scale above previously specified max-clusters
• Maybe most imporantly, databricks provides "autoscale" functionality, which I think is normally on (although I have no stats at all). Worst case scenario would be the dask cluster and databricks getting in an argument over who's in charge of scaling.

[jacobtomlinson]

In #9 @benrutter added an initial implementation of this, but we should explore migrating the base class over to Cluster instead of LocalCluster.

[jacobtomlinson]

I'd also be very curious to know if we can somehow scale the Databricks cluster from within our notebooks. If so we could wire up the DatabricksCluster.scale() method.

[jacobtomlinson]

Thanks @benrutter these are all good points. It might not be a sensible thing to do. Your comment does lead me to wonder how the autoscaling is triggered, is it based on node properties such as CPU/Memory or does Spark make those decisions? If it's Spark then there is no way for Dask to trigger that scale-up.

In general, I think autoscaling Dask on Databricks is an advanced topic that is best saved for later.