memory usage

[danni-m]

memory usage

[danni-m]

The memory usage in rdb and in memory are not the same.
The rdb format at the moment is as a pair of timestamp+value (per sample).

While the in memory usage is a little different: each TS key will have a minimum of 1 chunk and each chunk has a (default) of 360 samples.
Each sample is 12 bytes, but since we are using chunks for data partitioning there will be overhead of 360 % Nsamples per key.

If the keys are quite small as you say, I will recommend using smaller chunks.

[danni-m]

Also if you can upload the RDB file and make it available I might be able to get some more information about the usage.

[trist4n]

Thank you very much for the advice, I appreciate you being so responsive.

The difference between rdb and in memory makes sense, and definitely matches what I was seeing.
I'm working on a test dataset that could be possibly be sanitised, but unfortunately I'm developing this at my dayjob and can't just dump it as is.

If I understand correctly, when a TS key is created using the default sample size (360), the space is eagerly allocated (e.g. it immediately uses 360*12 bytes of space, before any samples are added). From this the overhead as you have said is the difference between the # of samples actually inserted, and the pre configured bucket size?

If the above is correct, creation of smaller buckets will reduce my memory usage, but possibly slicing buckets too small would create a slower query speed? (which I am OK with, but i'd like to understand the tradeoffs).

A separate point I would like to clarify, just to ensure that the behaviour I am seeing is as intended and stable, I have the following keys:

• Master TS key ("base"), with a 60 second retention period
• Hourly downsample TS key ("1h"), with 7 day retention period

and the following compaction rules:

• ts.createrule base sum 604800 1h

I then stream my data into the base TS key, and it is immediately downsampled into the appropriate bucket (1h).
Is the above a correct setup if I only require data at 1h granularity for 7 days?
Secondly, am i correct in saying it is safe to have only a 60 second retention period on the base ts key, as the downsampling is performed immediately after inserting into the base key (e.g. it does not need to buffer in the base key before being downsampled)

Thanks again! Things are looking great with this module so far.

[danni-m]

You are correct with your analysis.
Its important to mention that downsampling has some limitations if the redis is being restarted.
There's a ticket for it: #22

[danni-m]

@trist4n also, I've added support for global settings.
You can configure automatic downsampling rules for any created key including: default chunk size and retention time.

[danni-m]

@trist4n do you have anything else on this matter or can we close this?

[trist4n]

@danni-m right now i'm all set, i've had to put this project down temporarily but appreciate your help.

[danni-m]

@trist4n Sorry for the late response, was is due to shortcomings in the module?

[trist4n]

@danni-m i wouldn't say it is a shortcoming. for my particular use case, the main issue I could not easily fix was what happens when i got two samples with the same key on the same second, which wasn't avoidable the way I have modelled the system.

[danni-m]

@trist4n what do you expect to get in this situation?
The current logic is to use the last sample that was received.