Comments (11)
When adding maintain_order=True to the .group_by(pl.col('disk_hex')) call, the nondeterminism of the grouping appears to cease, but the resulting dataframe still includes duplicate values within the Disk ID column.
I am having the same issue working with dataframes larger than 20K records. I found that adding a sort operation on the grouping col before the group by solves the non-deterministic and non-uniqueness behavior for me on 0.20.11
. E.g., in_ the gist, adding .sort(by='disk_hex')
before the .group_by(pl.col('disk_hex'))
I think returns the expected result if I am interpreting the log correctly.
from polars.
@mirkomiorelli I'm seeing that as well, adding .sort(by='disk_hex')
before the .group_by(pl.col('disk_hex'))
returns the expected number of rows (17,148). This also corrects the issue when grouping on a larger dataframe with greater than 250,000 rows as well.
from polars.
I'm seeing something similar for max().over()
, it seems to be when the input isn't sorted and happens when n_rows > 1,000. Here's a minimal example:
import numpy as np
import polars as pl
n_groups = 250
rows_per_group = 4
(
pl.DataFrame(
{
'grp': np.repeat(np.arange(n_groups), rows_per_group),
'x': np.tile(np.arange(rows_per_group), n_groups)
}
)
.sample(fraction=1.0, shuffle=True)
.select(pl.col('x').max().over('grp'))
['x']
.value_counts()
)
at 1,000 samples this returns the expected
# x count
# i64 u32
# 3 1000
but at 1,004 (n_groups=251) it returns the following:
# x count
# i64 u32
# 2 269
# 1 150
# 3 493
# 0 92
from polars.
Related Issues (20)
- Python Polars 0.20.11 release workflow was failed HOT 1
- `scan_parquet` segfault on main HOT 1
- "expected <datatype>, got <datatype>" - use datatype display repr consistently? HOT 1
- docs(python): Add database and excel pages in User Guide IO
- `index_column` and `period` are not required to be of the same "type" HOT 2
- Bug Report - SchemaError with apply Function in Polars 0.20.3 HOT 3
- Allow converting unknown values to `null` in `pl.Enum` HOT 4
- Weakref to LazyFrame/DataFrame/Series is broken HOT 4
- polars 0.20.10 not compatible with connectorx on python 3.12
- concat failure between Datetime[UTC] column and Datetime column with only nulls
- Can't read parquet from a path with square brackets: RecursionError HOT 1
- Non-Deterministic group_by Sum HOT 3
- write_excel() - support additional engines HOT 5
- rolling by not yet supported for series with null value, panics for dynamic but not for fixed window HOT 4
- Reconsider the use of `Self` in type annotations of `DataFrame` and `LazyFrame` HOT 3
- Dataframe creation for row-oriented data is slow/uses lots of memory
- cum_sum operation not supported for decimals
- Generalized ufunc functions have inconsistent behavior between NumPy and Polars when returning scalars HOT 1
- Infer `time_unit` in `pl.duration` when nanoseconds is specified HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from polars.