Sampling GroupedDataFrames (rand)

[quachpas]

Sampling GroupedDataFrames (rand)

[nalimilan]

Duplicate of #2097. It would make sense to define rand on both GroupedDataFrame and DataFrame, as we implement shuffle for it (#3010).

For DataFrame, we could also allow specifying a number of rows to draw. That wouldn't work for GroupedDataFrame but we could print an error message with a hint about what to do. Or we could automatically create a new integer grouping column that would allow repeating a group multiple times if it's been drawn more than once. FWIW, this feature has been requested in dplyr but hasn't been implemented: tidyverse/dplyr#361, tidyverse/dplyr#6518.

[bkamins]

We could add it. @nalimilan, what do you think about adding:

Random.rand(rng::Random.AbstractRNG, ::Random.SamplerTrivial{<:GroupedDataFrame}) = gdf[rand(rng, 1:length(gdf))]

?

[bkamins]

Ah - good catch.

So - now I responded positively as rand's API specifies:

Pick a random element or array of random elements

And the key word is array. Which means that with rand(gdf, 10, 10) we would return a 10x10 array of SubDataFrame.

If we also added rand for data frame then writing rand(df, 10, 10) would return a 10x10 array of DataFrameRow.

I am not sure this is useful, but this could work. This is different from shuffle as shuffle does not promise to return an array, but a permuted copy. While rand promises to return a single element or an array.

The question is if users would find it intuitive and useful?

[quachpas]

Thanks for all the answers! Sorry about the missed duplicate issue.

The question is if users would find it intuitive and useful?

AFAIK the only interface that DataFrames.jl provides for Random is shuffle and shuffle!, which both return a permuted DataFrame. Since DataFrame does not support rand either, I was probably in the wrong to expect GroupedDataFrame to behave like an Array.

As for usefulness, in my case, I was looking to sample groups of data (hence the groupby), and it did feel jarring that I couldn't just sample the GroupedDataFrame. I am not sure it is strictly useful, but it is certainly more straightforward than the following

N = 100
tdf = transform(df, [:x1, :x2] => ByRow(string))
keys = unique(tdf[!, :x1_x2_string])
subset(tdf, :x1_x2_string => ByRow(in(rand(keys, N)))) # DataFrame, have to drop :x1_x2_string

VS

N = 100
gdf = groupby(df, [:x1, :x2])
rand(gdf, N) # Array of GroupedDataFrame? GroupedDataFrame?

I don't think it's intuitive for rand(gdf, 10, 10) to return an array. If shuffle returns a permuted copy, I would expect rand to always return a (Grouped)DataFrame (although that sounds like a lot of work for not much).

P.S.: I did not go into the implementation of GroupedDataFrame in details, but is there a reason why getindex(gd, idxs) does not support duplicates idxs?

[bkamins]

but is there a reason why getindex(gd, idxs) does not support duplicates idxs?

This is the same reason why Dict does not allow for duplicate keys. Group ids must be unique.

---

Adding shuffle and shuffle! to GroupedDataFrame is easy to do - we could add it if you would find it useful.