Pass `filter` to `file_uris` to skip partitions

[TomAugspurger]

Pass `filter` to `file_uris` to skip partitions

[TomAugspurger]

There's one distinction I didn't appreciate before: the current filter argument is a row filter. But file_uris is expecting a partition filter. It's OK to pass a row filter to a partition filter, but it's not OK to try to pass a partition filter if the dataset isn't actually partitioned on that expression.

I'm going to try briefly to see if we can automatically infer what subset of filter to pass to partition_filters. If I can't get that working, then maybe we add another keyword, partition_filter, to handle that.

[MrPowers]

@TomAugspurger - here's the syntax for filtering the partitions and by the column file statistics:

import pyarrow as pa
import pyarrow.dataset as ds
from deltalake import DeltaTable, write_deltalake

sample_data = pa.table({
    'part': pa.array(['a', 'a', 'b', 'b', 'c', 'c']),
    'x': pa.array([1, 1, 2, 2, 3, 3])
})

write_deltalake('sample_data', sample_data, partition_by=['part'])

dt = DeltaTable('sample_data')
dt.file_uris(partition_filters=[('part', '>', 'a')])
# ['/Users/willjones/Documents/notebooks/sample_data/part=c/0-acc806b2-a96d-47cc-ac80-7607abca0a66-0.parquet',
# '/Users/willjones/Documents/notebooks/sample_data/part=b/0-acc806b2-a96d-47cc-ac80-7607abca0a66-0.parquet']

# With Dataset.get_fragments
filter = (ds.field('part') > 'a') & (ds.field('x') < 3)
fragments = dt.to_pyarrow_dataset().get_fragments(filter=filter)
list(fragment.path for fragment in fragments)
# ['part=b/0-acc806b2-a96d-47cc-ac80-7607abca0a66-0.parquet']

Courtesy of @wjones127 in this issue.

[TomAugspurger]

Apparently passing a filter to file_uris is somewhat new in deltalake. I didn't find a changelog, but it is present in 0.8.1.

[rajagurunath]

Wow, This looks like a Massive improvement! Thanks a lot for this detailed analysis and suggestion! Happy to help wherever I can!

[rajagurunath]

• Makes Complete Sense, Actually I am also inclined towards having paritition_filter and filer_uris as separate Keyword arguments to the function. let me know your thoughts here!

[MrPowers]

I recently created a delta-rs issue to add a filters argument to file_uris() that will behave as expected.

[fjetter]

As an end user I would hope to not have to care about the difference between partition_filter and filter.

How about a read_delta API that is only accepting filter and in the background it splits the partition vs file filters itself.

In @MrPowers example this would look like

ddf = read_delta(
    'sample_data',
    filters=[
        ('part', '>', 'a'),
        ('x', '<', '3'),
    ]
)
assert ddf.npartitions == 1

Is there any reason why this should not be the top-level API?