wirte_parquet() and write_delta() panics when writing large dataset sizes

[edgBR]

wirte_parquet() and write_delta() panics when writing large dataset sizes

[ritchie46]

Are you entirely sure it is that amount of rows? Have you tried polars-u64-idx?

[edgBR]

Hi @ritchie46

Yes, I am entirely sure.

The data I'm writing is the one corresponding to :

INFO:__main__:Joined data row count (36657598, 41)

With regards to installing polars-u64-idx. The error persists even I install the package.

Also there is no dataset in my code bigger than 2573952862. And after the aggregation Im eliminating the intermediate dataset (before aggregation) just to be sure memory is okay.

[edgBR]

Hi,

Small update, I think I have narrowed down the issue.

It seems one of my aggregations did not work as expected due to a bug in the data. This resulted in the creation of a list type column with considerable size. Concretely every row of the aggregation had this list column of average size 127750 elements. As my data frame has 36.5M rows this might have been the case to this warning.

The code works perfectly fine when I removed this column.

Would it be possible to keep this open? Even that the issue is probably not related with the delta writer I found the exception message a bit confusing.

[stinodego]

I'm closing this issue as this doesn't have anything to do with the write functionality.

Feel free to open a separate issue about improving the error message of the index overflow, if there isn't one yet.