[BUG] Missing Partition Filters on renamed Columns after Join

[markus-raster]

[BUG] Missing Partition Filters on renamed Columns after Join

[NuthanReddy]

@markus-raster

This issue is observed on
Delta Lake version: 3.1.0
Spark version: 3.5.0

and

Delta Lake version: 2.4.0
Spark version: 3.4.1

right?

[NuthanReddy]

Can someone assign this to me. I can work on this.

[NuthanReddy]

As a initial check, I ran the same locally
using spark 3.5.1 and delta 3.1.0
and observed predicate pushdown for renamed column as well.

[markus-raster]

Hi @NuthanReddy,

Thank you for your reply. I also changed to my local file system, but I still have problems with either of the mentioned version combinations. Is the screenshot you are displaying from the first or the second explain()? The join in the parquet format will explicitly lead to the pushdown of the filter. Using the delta format, this is not the case. In the following screenshot, you can see the change in behavior of reading files with delta vs. parquet.