delta-incubator/delta-sharing-rs

A Minimalistic Rust Implementation of Delta Sharing Server.

delta-sharing-rs's Issues

Add documentation site

Contact Details

No response

Is your feature request related to a problem? Please describe.

There aren't any docs.

Describe the solution you'd like.

It would be cool to have a documentation site on how to use this project, like this one: https://delta-io.github.io/delta-rs/

We should probably wait till this project is promoted to the delta-io organization, so the URL is stable.

Code of Conduct

• I agree to follow this project's Code of Conduct

Streaming support

https://github.com/delta-io/delta-sharing#streaming

Register token at GET admin::profile

Add metastore feature

Update API Response Format

• Add decimal primitive type

https://github.com/delta-io/delta-sharing/blob/main/PROTOCOL.md#primitive-types

Project scope and organisation

There have been several discussions about the organisation and scope of this project, for example #19 and #22. Let's discuss some options here.

Microsoft Azure backed Delta tables

Contact Details

[email protected]

Is your feature request related to a problem? Please describe.

I was trying to connect sharing server with ADLS gen 2, but I am getting internal server error once I execute request.

Describe the solution you'd like.

Hi,
is supporting Microsoft Azure backed Delta tables in development? Is there some estimate when it is expected to be done, or if help is needed, I can try to help, with some minor guidelines and info from someone in charge :) ?
Thanks in advance!

Code of Conduct

• I agree to follow this project's Code of Conduct

Improve authorization flow

The authentication process is handled in the front end and BFF, so the admin/auth middleware should be switched based on the feature flag to enable or disable it:

https://medium.com/@ognis1205/deltahub-data-sharing-platform-1-bdee60e74245

Implement CDF related API

Protocol Specification

Replace rusoto with aws-sdk-rust

https://aws.amazon.com/sdk-for-rust/

Move azure signing to object_store crate

Contact Details

No response

Is your feature request related to a problem? Please describe.

When trying to update dependencies, there were some issues since our server (axum) and azure crates have conflicting dependencies on the base64 crate.

Updating dependencies is particularly important, since right now we have a large number of security findings from dependabot.

Describe the solution you'd like.

Use object_store for URL signing, such that we no longer need to pull in azure-* dependencies.

Code of Conduct

• I agree to follow this project's Code of Conduct

Move URL signing to object_store crate

In the discussion on #22, @tustvold and @roeap pointed out the rust ecosystem is building abstractions over object stores (and consequently URL signing) in the object_store crate. Moving to object_store will simplify the implementation.

Dependencies

Before object_store can be integrated, it needs to support signing URLs for Azure and GCP. Alternatively we could also implement the Signer trait in this crate, but I think that would be wasteful.

• apache/arrow-rs#5232
• apache/arrow-rs#5233

References:

object_store crate

Add CONTRIBUTING.md

Seperate protocol routes from admin routes

Currently the admin routes and protocol routes (also known as the guest routes) are coupled. Separating them would allow people to implement their own admin routes, while still depending on a sound implementation of the open Delta Sharing protocol.

Some adjustments due to the move

• Cargo.toml
• kotosiro -> delta_sharing
• README.md

GitHub actions
DockerHub

Rethink filtering order

https://github.com/kotosiro/sharing/blob/main/src/server/services/deltalake.rs#L240

Allow for configuring multiple locations per provider

Contact Details

No response

Is your feature request related to a problem? Please describe.

The current setup assumes that there is either a single bucket per provider, or at least that all locations share the same credential.

We should allow for configuring dedicated locations, possibly assigning a default credential that is used, when no dedicated credential for that provider is available.

Describe the solution you'd like.

Define some sort of registry, that takes the form of the well know URLs, including the host.

• az://container <-- requires additional account into
• s3://bucket
• gs://bucket

Code of Conduct

• I agree to follow this project's Code of Conduct

Microsoft Azure backed Delta tables

Contact Details

[email protected]

Is your feature request related to a problem? Please describe.

I was trying to connect sharing server with ADLS gen 2, but I am getting internal server error once I execute request.

Describe the solution you'd like.

Hi,
is supporting Microsoft Azure backed Delta tables in development? Is there some estimate when it is expected to be done, or if help is needed, I can try to help, with some minor guidelines and info from someone in charge :) ?
Thanks in advance!

Code of Conduct

• I agree to follow this project's Code of Conduct

Support capabilities header

The Delta Sharing protocol has been extended with capability headers. This allows sharing of table with more advanced Delta Lake features in the future.

The official protocol can be referenced for implementation:
https://github.com/delta-io/delta-sharing/blob/main/PROTOCOL.md#delta-sharing-capabilities-header

Associate share with its owner and change authorization process

Introduce extension points for share storage

Currently the Delta Sharing server depends on a postgres database to store information about the defined shares, schemas, and tables. For a variety of reasons users may want to use a different store. My proposed solution if to introduce a ShareStore trait (open to naming suggestions) and let the app state depend on a concrete implementation defined in a library or user supplied.