Comments (8)
Interestingly from your above example this fails:
lf_result.sink_parquet("lazy_frame.parquet")
But this succeeds:
lf_result.collect().lazy().sink_parquet("lazy_frame.parquet")
from polars.
@cmdlineluser Thanks for your feedback. When I specify the full schema then streaming works for me. Also my complete use case not only the example above.
Problem is solved for me.
from polars.
Sorry, but this cannot be the solution. What if you have an unknown amount of fields that are inferred by polars in the unnest function.
You cannot specify a schema in that case without replicating the inference process of polars.
This is an obvious bug to me and IMHO should be reopened.
from polars.
This might be very well be caused by #16274 and fixed by #16463.
EDIT: nevermind, this is not related to that at all
from polars.
I'm not sure if it is the underlying cause, but whilst trying to reduce down your example, the sink_parquet
panics:
import polars as pl
lf = pl.LazyFrame({
"index": [0, 1, 2],
"payload": [1, 1, 1],
"category": ["a", "b", "c"]
})
lf_temp = (
lf.with_columns(
pl.struct("index", "payload", "category").map_elements(lambda row:
[
{"a": row["index"], "b": row["payload"], "c": row["category"]},
{"a": row["index"] + 1, "b": row["payload"] + 1, "c": row["category"]}
],
return_dtype=pl.List(pl.Struct)
)
.alias("struct_column")
)
)
lf_result = lf_temp.explode("struct_column").unnest("struct_column")
lf_result.sink_parquet("lazy_frame.parquet")
# pyo3_runtime.PanicException: called `Result::unwrap()` on an `Err` value:
# ComputeError(ErrString("a StructArray must contain at least one field"))
The .collect()
output looks OK:
shape: (6, 6)
┌───────┬─────────┬──────────┬─────┬─────┬─────┐
│ index ┆ payload ┆ category ┆ a ┆ b ┆ c │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ str ┆ i64 ┆ i64 ┆ str │
╞═══════╪═════════╪══════════╪═════╪═════╪═════╡
│ 0 ┆ 1 ┆ a ┆ 0 ┆ 1 ┆ a │
│ 0 ┆ 1 ┆ a ┆ 1 ┆ 2 ┆ a │
│ 1 ┆ 1 ┆ b ┆ 1 ┆ 1 ┆ b │
│ 1 ┆ 1 ┆ b ┆ 2 ┆ 2 ┆ b │
│ 2 ┆ 1 ┆ c ┆ 2 ┆ 1 ┆ c │
│ 2 ┆ 1 ┆ c ┆ 3 ┆ 2 ┆ c │
└───────┴─────────┴──────────┴─────┴─────┴─────┘
from polars.
Yeah, the collect also panics with streaming enabled:
lf_result.collect(streaming=True)
# PanicException: called `Result::unwrap()` on an `Err` value:
# ComputeError(ErrString("a StructArray must contain at least one field"))
from polars.
I just tried this again on the latest main and it still fails.
Specifying the full schema allows it to run as expected.
return_dtype=pl.List(pl.Struct({'a': pl.Int64, 'b': pl.Int64, 'c': pl.String}))
It seems allowing an empty pl.Struct
leads to the problem here?
from polars.
@cmdlineluser Thank you for reopening 🙏🏻 or linking
from polars.
Related Issues (20)
- BinViewChunkedBuilder has incorrect docs
- Series.hist resulting series name changes with include_breakpoint=False, include_category=False HOT 2
- Series.hist adds two bins when specifying bins HOT 1
- data spilled to disk not cleaned up on failure HOT 1
- Cannot serialize polars.LazyFrame (`Expr::RenameAlias cannot be serialized`)
- Big difference in iteration speed over GroupBy object depending on dataFrame construction HOT 1
- Can't `sink_parquet` on a sorted LazyFrame containing decimal columns HOT 4
- `write_database` closes adbc connection HOT 3
- `list_concat([list<T>, list<T>])` gives `list<T>`, not `list<list<T>>` HOT 10
- mismatching schemas when opening csv file
- SchemaError for Non-Exiting dtype on Concat HOT 3
- LazyFrame select in 0.20.31 includes hive partition column even when not in specified columns HOT 2
- cross-join should not work on any key HOT 1
- `.sink_parquet()` sometimes panics when `statistics` has `"null_count": False` HOT 1
- separate `pl.list()` and `pl.concat_list` HOT 4
- Panick since first release candidate when expressions in `select()` return a different number of rows HOT 3
- Question, how do you generate and test the Python examples in the README? HOT 2
- min/max operations on i16 list with None elements HOT 1
- Panic when mismatching types between glob files HOT 8
- `write_database` fails for UInts and Time dtypes when ADBC used HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from polars.