Parquet file writer uses non-compliant list element field name

[cgbur]

Parquet file writer uses non-compliant list element field name

[ritchie46]

I believe we fixed this recently. @coastalwhite can confirm.

[coastalwhite]

No, this is not yet resolved. It requires filtering the item name when we convert to a Parquet schema. Since just mapping all items named item to element seems quite naive, I did not immediately solve this.

[whichwit]

Following. Experiencing the same issue where list type columns in Polars cannot be used by PyIceberg (via PyArrow).

Will this be resolved soon (with the solution potentially naive) or is there a workaround?

[ohines]

I am running into a possibly related issue.
I am trying to replace some pandas code with a polars implementation, but am seeing downstream issues because polars uses arrow large_list, but pandas uses just list. Is is possible to cast to list during parquet writing with polars (with or without pyarrow)?

Extending the example in the issue description:

import polars as pl
import pyarrow.parquet as pq

df = pl.DataFrame(
    {
        "a": [[1, 2], [1, 1, 1]],
    }
)

df.write_parquet("example.parquet", use_pyarrow=False)
print("with polars")
print(pq.read_schema("example.parquet"))
print()
df.write_parquet("example.parquet", use_pyarrow=True)
print("with pyarrow")
print(pq.read_schema("example.parquet"))
df.to_pandas().to_parquet("example.parquet")
print()
print("with pandas")
print(pq.read_schema("example.parquet"))

with polars
a: large_list<item: int64>
  child 0, item: int64

with pyarrow
a: large_list<element: int64>
  child 0, element: int64

with pandas
a: list<element: int64>
  child 0, element: int64
-- schema metadata --
pandas: '{"index_columns": [{"kind": "range", "name": null, "start": 0, "' + 365