Non-deterministic and incomplete group_by in 0.20.11

[seanlane]

Non-deterministic and incomplete group_by in 0.20.11

[mirkomiorelli]

When adding maintain_order=True to the .group_by(pl.col('disk_hex')) call, the nondeterminism of the grouping appears to cease, but the resulting dataframe still includes duplicate values within the Disk ID column.

I am having the same issue working with dataframes larger than 20K records. I found that adding a sort operation on the grouping col before the group by solves the non-deterministic and non-uniqueness behavior for me on 0.20.11. E.g., in_ the gist, adding .sort(by='disk_hex') before the .group_by(pl.col('disk_hex')) I think returns the expected result if I am interpreting the log correctly.

[seanlane]

@mirkomiorelli I'm seeing that as well, adding .sort(by='disk_hex') before the .group_by(pl.col('disk_hex')) returns the expected number of rows (17,148). This also corrects the issue when grouping on a larger dataframe with greater than 250,000 rows as well.

[jmoralez]

I'm seeing something similar for max().over(), it seems to be when the input isn't sorted and happens when n_rows > 1,000. Here's a minimal example:

import numpy as np
import polars as pl

n_groups = 250
rows_per_group = 4
(
    pl.DataFrame(
        {
            'grp': np.repeat(np.arange(n_groups), rows_per_group),
            'x': np.tile(np.arange(rows_per_group), n_groups)
        }
    )
    .sample(fraction=1.0, shuffle=True)
    .select(pl.col('x').max().over('grp'))
    ['x']
    .value_counts()
)

at 1,000 samples this returns the expected

# x	count
# i64	u32
# 3	1000

but at 1,004 (n_groups=251) it returns the following:

# x	count
# i64	u32
# 2	269
# 1	150
# 3	493
# 0	92