LazyFrame - Unnested columns are missing in Lazy Frame

[herrmann1981]

LazyFrame - Unnested columns are missing in Lazy Frame

[herrmann1981]

Interestingly from your above example this fails:

lf_result.sink_parquet("lazy_frame.parquet")

But this succeeds:

lf_result.collect().lazy().sink_parquet("lazy_frame.parquet")

[herrmann1981]

@cmdlineluser Thanks for your feedback. When I specify the full schema then streaming works for me. Also my complete use case not only the example above.
Problem is solved for me.

[jpfeuffer]

Sorry, but this cannot be the solution. What if you have an unknown amount of fields that are inferred by polars in the unnest function.
You cannot specify a schema in that case without replicating the inference process of polars.
This is an obvious bug to me and IMHO should be reopened.

[coastalwhite]

This might be very well be caused by #16274 and fixed by #16463.

EDIT: nevermind, this is not related to that at all

[cmdlineluser]

I'm not sure if it is the underlying cause, but whilst trying to reduce down your example, the sink_parquet panics:

import polars as pl

lf = pl.LazyFrame({
    "index": [0, 1, 2],
    "payload": [1, 1, 1],
    "category": ["a", "b", "c"]
})

lf_temp = (
    lf.with_columns(
        pl.struct("index", "payload", "category").map_elements(lambda row: 
            [
                {"a": row["index"], "b": row["payload"], "c": row["category"]},
                {"a": row["index"] + 1, "b": row["payload"] + 1, "c": row["category"]}
            ],
            return_dtype=pl.List(pl.Struct)
        )
        .alias("struct_column")
    )
)

lf_result = lf_temp.explode("struct_column").unnest("struct_column")

lf_result.sink_parquet("lazy_frame.parquet")
# pyo3_runtime.PanicException: called `Result::unwrap()` on an `Err` value: 
# ComputeError(ErrString("a StructArray must contain at least one field"))

The .collect() output looks OK:

shape: (6, 6)
┌───────┬─────────┬──────────┬─────┬─────┬─────┐
│ index ┆ payload ┆ category ┆ a   ┆ b   ┆ c   │
│ ---   ┆ ---     ┆ ---      ┆ --- ┆ --- ┆ --- │
│ i64   ┆ i64     ┆ str      ┆ i64 ┆ i64 ┆ str │
╞═══════╪═════════╪══════════╪═════╪═════╪═════╡
│ 0     ┆ 1       ┆ a        ┆ 0   ┆ 1   ┆ a   │
│ 0     ┆ 1       ┆ a        ┆ 1   ┆ 2   ┆ a   │
│ 1     ┆ 1       ┆ b        ┆ 1   ┆ 1   ┆ b   │
│ 1     ┆ 1       ┆ b        ┆ 2   ┆ 2   ┆ b   │
│ 2     ┆ 1       ┆ c        ┆ 2   ┆ 1   ┆ c   │
│ 2     ┆ 1       ┆ c        ┆ 3   ┆ 2   ┆ c   │
└───────┴─────────┴──────────┴─────┴─────┴─────┘

[cmdlineluser]

Yeah, the collect also panics with streaming enabled:

lf_result.collect(streaming=True)

# PanicException: called `Result::unwrap()` on an `Err` value: 
# ComputeError(ErrString("a StructArray must contain at least one field"))

[cmdlineluser]

I just tried this again on the latest main and it still fails.

Specifying the full schema allows it to run as expected.

return_dtype=pl.List(pl.Struct({'a': pl.Int64, 'b': pl.Int64, 'c': pl.String}))

It seems allowing an empty pl.Struct leads to the problem here?

[jpfeuffer]

@cmdlineluser Thank you for reopening 🙏🏻 or linking