Try implementing this lib with delta-rs

[MrPowers]

Try implementing this lib with delta-rs

[rajagurunath]

@MrPowers - Thanks for the easy-to-try example!!

The above code snippet definitely works, But I am afraid of the cases where the different versions of the delta table having different schema/columns and when combining all the versions with different columns was not supported out of the box in dd.read_parquet ( i.e, Schema Evolution was not supported ) for this we are heavily dependant on pyarrow dataset.

Pyarrow dataset able to read parquet files with different schema/columns into a single data frame (by filling NANs in the non-available columns)

currently tried adding a wrapper for delta-rs in this branch: https://github.com/rajagurunath/dask_deltatable/tree/feature/delta-rs, after some more documentation and testing we will create a pull request and release a new package.

cheers

[rajagurunath]

Hi @MrPowers,

That's a great Suggestion. 👍

We initially discussed integrating delta-rs and dask here, and at that
time, we are facing some problems to read data from s3, AzureFS, GCFS.

Now I think there was explicit support for the pyarrow filesystem to read from a different backend.

After your Suggestion, Now I have started working on delta-rs integration, and currently trying to figure out how to parallelize IO using Dask (reading multiple parquet files). if we rely on delta-rs to_pyarraow_dataset function all the parquet files are read in the same thread (Dask task) I guess. so planning to take all the advantages of delta-rs and implementing only the parallel reading function using Dask Delayed separately.

Any other Suggestions for parallelizing the deltalake read using Dask ?

And Once again thanks for trying this out and opening an issue here.

[MrPowers]

@rajagurunath - I actually think this is going to be super easy.

Think this will work:

from deltalake import DeltaTable
import dask.dataframe as dd

dt = DeltaTable("tmp/some-delta-pyspark")
ddf = dd.read_parquet(dt.files())

The read_parquet method takes a list of file names.

Can you try and let me know if it works?

[MrPowers]

Sounds good, keep me posted ;)

[rajagurunath]

Hi @MrPowers,

Integrating delta-rs and dask work was almost completed,

if you have some time can you please review the PR created here #3. and let me know your suggestions for the same.

Thanks in Advance.