Comments (4)
@avimallu Weird. Its a 14 core processor (13th gen i7)
from polars.
.over()
performs a group_by e.g.
(df2.group_by("group")
.agg(
pl.when(pl.col("random_value")==1)
.then(2)
.otherwise(pl.col("random_value"))
.alias("result")
)
)
But the lists would be exploded and the results joined back in the correct row order.
from polars.
I find it strange that it doesnt try to do 1 operation across all the groups simultaneously.
Why do you find it strange? You're explicitly asking to do a group-by (using over
) operation, and it's doing that.
What you seem to be asking for is for Polars to ignore that you've asked for an over
calculation, because the calculation's result is the same regardless of whether or not the group is present. That's something that could be achieved using LazyFrame
s, but I don't think it's implemented right now.
I'm also not able to reproduce your 20x slower timings, so there seems to be some optimization going on - in my case, it could be that it's parallelizing across cores efficiently - your recent issue (#15480) might point to having a processor with fewer available cores? 🤔
24.8 ms ± 886 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
107 ms ± 569 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
384 ms ± 10.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
from polars.
Hmm, forgot to turn off battery saver. Turned everything to max and activated all cores manually and got this
44.2 ms ± 2.74 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
24.8 ms ± 1.45 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
477 ms ± 15.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
Idk. Maybe i just dont understand the underlying process of .over(). Feels like it still has potential to do it better in the given example
from polars.
Related Issues (20)
- Error checking equality of `List(Struct(Enum)))` HOT 1
- Implement rolling functions for dtypes Int16, Int8, Boolean
- Regression in when/then/otherwise performance 0.20.15 -> 0.20.16 HOT 1
- Cannot drop a column in UDF when using `.pipe` HOT 1
- Document that `quantile` ignores `nan` values HOT 2
- Support NumPy 2.0 HOT 7
- Join on multiple keys with Lazy DataFrames panics HOT 2
- DataFrame constructor fails for Uint64 arrays and lists with large values HOT 2
- Panic when sorting DataFrame by Null column HOT 1
- Cannot sort by multiple columns including Decimal column
- Stream processing support (in the style of flink/risingwave/arroyo/etc.) HOT 1
- Scan csv.gz tables HOT 2
- Create `pl.lit` from a Python `set` object? HOT 2
- Unexpected result when melting categorical columns
- Expose query execution metrics HOT 2
- Implement coverage mark tests for critical performance paths
- Cannot import name 'async_sessionmaker' from 'sqlalchemy.ext.asyncio' in 1.0.0a1 HOT 2
- `pl.scan_ipc(…)` fails when followed by `.head(0)` HOT 3
- `join` on a `literal` causes segfault when either frame has 0 rows. HOT 1
- Filter doesn't return correct rows HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from polars.