Polars job faces timeouts while reading multiple parquet files on a AWS EKS pod

[arnaud-vennin]

Polars job faces timeouts while reading multiple parquet files on a AWS EKS pod

[arnaud-vennin]

Hi,
We investigated with the Ops team which manages the EKS cluster on which the Polars script runs.
Apparently it may be due to the fact that the pod does not have time to fully gets up before Polars starts.
A simple solution was proposed : adding a 10 seconds sleep function at the beginning of the script.
Polars version was also turn back to 0.20.10 since we faces concurrency tuner deadlock as mentioned on Discord (#14827).
The data ingestion will run for several days to fully validate this solution.

[arnaud-vennin]

Hi,
I let the Polars job runs for several days : 15 launches, no retry nor timeout. It works perfectly.
The solution was to add a 10 seconds sleep function at the beginning of the main.py to give time to the EKS pod to get up completely.