hive partitioning unable to skip files when casting column to date

[mwhithead]

hive partitioning unable to skip files when casting column to date

[c-peters]

Thanks, I edited the path when reading from data/mw/ to ./

[c-peters]

We (currently) allow comparisons in the form of pl.col("") > | < | == | != literal. Would there be a problem with the first case? ,like casting the datetime

first.filter(pl.col("date") == dt.date(2024, 2, 1).strftime(...)).explain()

Any expression that might alter the value of the column (e.g. pl.col("date").str.to_date()) would significantly increase the complexity and overhead of hive partitioning as we would need to run the expression for every file instead of comparing the statistic in the path directly to the literal.

It would be nice if we can provide feedback, a warning in case a hive partitioned dataset is scanning the whole dataset

[mwhithead]

Ah, i guess i made the example slightly too small. What I am actually using, and what is most powerful (i think)

first.filter(pl.col("date").str.to_date().is_between(start, end)).explain()

Otherwise, to using the literal comparisons i need to do a loop over the required dates, and & them all together.
I currently do this in my environment, but it is pretty hacky.

[ritchie46]

I agree this should be fixed. We first need to do proper schema inference on hive partitions. Once that is in place we can use a similar architecture we use for parquet statistic pruning for hive partitions.

[mwhithead]

Awesome, is there an issue for tracking the schema inference? I can just follow along on that.
Thanks

[c-peters]

Not yet, I have created an issue for hive partition schema (#14838)