Support datasets that contain more than 2B values

[jpountz]

Support datasets that contain more than 2B values

[tdunning]

I can do that but the current limit is typically more generous than 2^31 samples. Instead, it is more like two billion samples in a single centroid. If you don't have repeated values the limit is about 1000 times that.

Furthermore, it is common to have many aggregates going at the same time each of which has this limit separately.

So my question to you is: how real is this request? How many problematic examples have you seen?

How much do you have aggre

Sent from my iPhone

On Mar 18, 2014, at 13:02, Adrien Grand [email protected] wrote:

Current implementation uses ints in order to represent counts of values. It would be useful to switch to longs so that quantile estimations would still work if more than 2B values are accumulated.

—
Reply to this email directly or view it on GitHub.

[jpountz]

I don't have any problematic example yet since the feature that we are working on and which leverages t-digest is not released yet. However, some of our users store more than 2B documents and I was checking whether quantile estimations would work on such large datasets.

Maybe something simple that can be done to raise the maximum dataset size while having negligible impact on memory usage would be to just change ArrayDigest.totalWeight and TreeDigest.count to longs?

[tdunning]

On Tue, Mar 18, 2014 at 2:13 PM, Adrien Grand [email protected]:

Maybe something simple that can be done to raise the maximum dataset size
while having negligible impact on memory usage would be to just change
ArrayDigest.totalWeight and TreeDigest.count to longs?

Good point. I will do that sometime this week unless I see a pull request
from you sooner.