danni-m/redis-timeseries

Future development of redis-timeseries is at github.com/RedisLabsModules/redis-timeseries.

Future development of redis-timeseries is at github.com/RedisLabsModules/redis-timeseries. Please submit issues at github.com/RedisLabsModules/redis-timeseries.

redis-timeseries's Issues

Redis crashes when bucket size in aggregated query is 0

Replacing the current value in a series corrupts aggregate values

> ts.create root 100
> ts.create test 100
> ts.createrule root sum 10 test
> ts.range root 0 100
(empty list or set)
> ts.range test 0 100
(empty list or set)
> ts.add root 10 1
> ts.range root 0 100
1) 1) (integer) 10
   2) "1”
> ts.range test 0 100
1) 1) (integer) 10
   2) "1"
> ts.add root 10 5
> ts.range root 0 100
1) 1) (integer) 10
   2) "5"
> ts.range test 0 100
1) 1) (integer) 10
   2) "6"

We need to store the last calculated value in aggregate contexts and then either add a replaceValue method that works off of the last calculated value as opposed to the current one, or pass timestamp to appendValue and add a branch to do what replaceValue would do.

For backwards compatibility and code style reasons I’d prefer—I think—to add replaceValue, and I’ll submit a pull doing so.

Add NX argument to TS.CREATE

Hi,

What are your thoughts on adding a NX (or similar) argument to TS.CREATE that will only create a new TS key if it doesn't already exist? This way if a process is restarted, it doesn't have to query Redis to determine what TS keys exist or not, and instead will issue a TS.CREATE for any update. For some use cases the extra command in a pipeline might not be acceptable, but for lower-throughput uses the overhead is not measurable.

Thanks,
Troy

ADDing without CREATEing

It looks like the previous version CREATEd a TS when you ADDed to a nonexistant key. Now it gives an error. I'd like to revert back to the old behaviour, any tips on where to look?

Timeseries starting from 0 are unsupported

> ts.create test 100
> ts.add test 0 5
> ts.range test 0 100
(empty list or set)
> ts.add test 1 5
> ts.range test 0 100
1) 1) (integer) 1
   2) "5"

This is likely due to a comparison check between timestamp and lastTimestamp. This isn’t actually a personal issue for me, but it’s irksome and I’d like to submit a pull for it if there’s not a reasonable reason for this behavior.

Note that in addition it’s perhaps wise not to consider the timestamp as any specific unit to begin with as I don’t think such an assumption is made anywhere in code (beyond just variable names) and truly our only constraint is that the value for timestamp be monotonic and >= 0 (though currently with this bug we require >0), which additionally should allow us to use uint32_t for timestamp_t/api_timestamp_t, unless there’s a place doing subtraction with them that I haven’t come across.

Timestamp with milliseond precision

Hi, your (excellent) module does not seem to support millisecond precision as it throughs a "timestamp is too old" error. Any chance this might be added in the future?

Multiple compaction rules on a key cause crash on redis startup

> ts.create test.1m 7200
> ts.create test.30m 7200
> ts.create test.1h 7200
> ts.createrule test.1m max 1800 test.30m
> ts.createrule test.1m max 3600 test.1h

Restarting redis then causes a crash:

=== REDIS BUG REPORT START: Cut & paste starting from here ===
64756:M 10 May 14:57:27.805 # Redis 4.0.8 crashed by signal: 11
64756:M 10 May 14:57:27.805 # Crashed running the instruction at: 0x10a4039f2
64756:M 10 May 14:57:27.805 # Accessing address: 0x2f
64756:M 10 May 14:57:27.806 # Failed assertion: <no assertion failed> (<no file>:0)

------ STACK TRACE ------
EIP:
0   redis-tsdb-module.so                0x000000010a4039f2 series_rdb_load + 386

Backtrace:
0   redis-server                        0x000000010a29fa32 logStackTrace + 110
1   redis-server                        0x000000010a29fdbf sigsegvHandler + 236
2   libsystem_platform.dylib            0x00007fff631a1f5a _sigtramp + 26
3   libsystem_c.dylib                   0x00007fff62f2a770 __vfprintf + 16214
4   redis-server                        0x000000010a282afa rdbLoadObject + 2435
5   redis-server                        0x000000010a28349e rdbLoadRio + 1358
6   redis-server                        0x000000010a283692 rdbLoad + 70
7   redis-server                        0x000000010a268e4a loadDataFromDisk + 292
8   redis-server                        0x000000010a26979e main + 1224
9   libdyld.dylib                       0x00007fff62e93015 start + 1
10  ???                                 0x0000000000000002 0x0 + 2

------ INFO OUTPUT ------
# Server
redis_version:4.0.8
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:d791831611c260ce
redis_mode:standalone
os:Darwin 17.5.0 x86_64
arch_bits:64
multiplexing_api:kqueue
atomicvar_api:atomic-builtin
gcc_version:4.2.1
process_id:64756
run_id:cd77c354c689286f253611fd19420d2ccdafe71d
tcp_port:6379
uptime_in_seconds:0
uptime_in_days:0
hz:50
lru_clock:16040007
executable:redis-server
config_file:redis.conf

# Clients
connected_clients:0
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0

# Memory
used_memory:971632
used_memory_human:948.86K
used_memory_rss:0
used_memory_rss_human:0B
used_memory_peak:971632
used_memory_peak_human:948.86K
used_memory_peak_perc:inf%
used_memory_overhead:955640
used_memory_startup:955536
used_memory_dataset:15992
used_memory_dataset_perc:99.35%
total_system_memory:17179869184
total_system_memory_human:16.00G
used_memory_lua:37888
used_memory_lua_human:37.00K
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
mem_fragmentation_ratio:0.00
mem_allocator:libc
active_defrag_running:0
lazyfree_pending_objects:0

# Persistence
loading:1
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1525989447
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
loading_start_time:1525989447
loading_total_bytes:214
loading_loaded_bytes:0
loading_loaded_perc:0.00
loading_eta_seconds:1

# Stats
total_connections_received:0
total_commands_processed:0
instantaneous_ops_per_sec:0
total_net_input_bytes:0
total_net_output_bytes:0
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
evicted_keys:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0

# Replication
role:master
connected_slaves:0
master_replid:4cd9caf7397c0a9c41ecac930b289bd6f72baf4e
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:0.01
used_cpu_user:0.01
used_cpu_sys_children:0.00
used_cpu_user_children:0.00

# Commandstats

# Cluster
cluster_enabled:0

# Keyspace
db0:keys=1,expires=0,avg_ttl=0

------ CLIENT LIST OUTPUT ------

------ REGISTERS ------
64756:M 10 May 14:57:27.806 # 
RAX:00007f9ee4c03dd0 RBX:00007ffee59a3260
RCX:0000000000000007 RDX:0000000000000001
RDI:0000000000000e10 RSI:00007f9ee4c03e00
RBP:00007ffee59a3250 RSP:00007ffee59a31b0
R8 :9e3779b97f4a7c55 R9 :0000000000000001
R10:0000000000000001 R11:00007f9ee4c00000
R12:4d20c1f9360f1000 R13:0000000000000007
R14:ffffffffffffffff R15:00007f9ee4c033c0
RIP:000000010a4039f2 EFL:0000000000010206
CS :000000000000002b FS:0000000000000000  GS:0000000000000000
64756:M 10 May 14:57:27.806 # (00007ffee59a31bf) -> 0000000000000168
64756:M 10 May 14:57:27.806 # (00007ffee59a31be) -> 0000000000000002
64756:M 10 May 14:57:27.806 # (00007ffee59a31bd) -> 00007f9ee4c03c20
64756:M 10 May 14:57:27.806 # (00007ffee59a31bc) -> 0000000000000007
64756:M 10 May 14:57:27.806 # (00007ffee59a31bb) -> 00007f9ee4c03cb0
64756:M 10 May 14:57:27.806 # (00007ffee59a31ba) -> 00000001f9c1204d
64756:M 10 May 14:57:27.806 # (00007ffee59a31b9) -> 00007f9ee4c03dc0
64756:M 10 May 14:57:27.806 # (00007ffee59a31b8) -> 0000000000000e10
64756:M 10 May 14:57:27.806 # (00007ffee59a31b7) -> 0000000000000002
64756:M 10 May 14:57:27.806 # (00007ffee59a31b6) -> 00007f9ee4c03dd0
64756:M 10 May 14:57:27.806 # (00007ffee59a31b5) -> 0000000000000008
64756:M 10 May 14:57:27.806 # (00007ffee59a31b4) -> 000000010a282eb8
64756:M 10 May 14:57:27.806 # (00007ffee59a31b3) -> 000000010a282ee4
64756:M 10 May 14:57:27.806 # (00007ffee59a31b2) -> 00007ffee59a31f0
64756:M 10 May 14:57:27.806 # (00007ffee59a31b1) -> 0000000000000008
64756:M 10 May 14:57:27.806 # (00007ffee59a31b0) -> 000000010a282eb8

------ DUMPING CODE AROUND EIP ------
Symbol: series_rdb_load (base: 0x10a403870)
Module: redis-tsdb-module.so (base 0x10a400000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=0x10a403870 -D -b binary -m i386:x86-64 /tmp/dump.bin
------
64756:M 10 May 14:57:27.806 # dump of function (hexdump of 514 bytes):
554889e54881eca000000048897df08975ec837dec000f843b000000488d3536330000488d1535330000488d05ff3a0000488b00488b7df048898568ffffffb000488b8d68ffffffffd148c745f800000000e9da010000488d05c23a0000488b00488b7df0ffd0488d3db23a0000488945e0488b07488b7df0ffd0488d3d9e3a0000488945d8488b07488b7df0ffd0488945d0488b45e089c1488b45d86689c289cf0fbff2e8b6f6ffff4c8d05df390000488945c8498b00488b7df0ffd0488945b8c745b400000000486345b4483b45d00f83ca000000488d05323a0000488b00488b7df0ffd0488d3d323a0000488945a8488b07488b7df0ffd0488d3d1e3a0000488945a0488b07488b7df0ffd0488d3d2239000048894598488b07488b7db8488b75a8ffd0488d35a23a0000488945a8488b06488b7db8488b75a8ffd0488b7da8488b459889c1488b45a089c289cee8dafbffff48894590488b45c848837820000f8511000000488b4590488b4dc848894120e914000000488b4590488b4dc048894128488b4590488945c0e9000000008b45b483c0018945b4e928ffffff488d0578390000488b00488b7df0ffd04889458848c7458000000000488b4580483b45880f835e000000488d054e390000488b00488b7df0ffd0488d3d1639000089c1898d7cffffff488b07488b7df0ffd0f20f118570ffffff488b7dc88bb57c
Function at 0x10a402fd0 is NewSeries
Function at 0x10a4035a0 is NewRule

=== REDIS BUG REPORT END. Make sure to include from START to END. ===

My guess is that there’s some issue with (de)serializing multiple compaction rules on a series to/from the rdb file, but I’m still investigating.

how does retentionSecs works?

I am trying to create a new time-series bucket with retentionSec option, but somehow it does not trim the data after a given time.
here is what im doing..

TS.CREATE helloWorld 10
TS.add helloWorld 1530610870 1
TS.add helloWorld 1530610871 2

after 10 sec, i try the range query, but the data is still there
in my understanding retentionSec will remove the data after time is passed
my question is my understanding is right or did i missing something here?

Implement RDB and AOF methods

This is crucial for storage persistence

How to use int64 timestamp

redis-cli

127.0.0.1:6379> ts.range key 0 999999999999
(empty list or set)
127.0.0.1:6379> ts.range key 0 "999999999999"
(empty list or set)
127.0.0.1:6379> ts.range key 0 999999999999
(empty list or set)
127.0.0.1:6379> ts.range key 0 1530025201
    1) 1) (integer) 1530025200
    2) "0"

Do you support int64? How can I query with a large timestamp?

Contributing a feature - is it relevant?

Hey there,
I'm using your TS module, and i encountered a use case in which i need to:

1. update existing value for given timestamp
2. add values for history (i.e. with timestamps that are older than existing ones)

I'm thinking about developing those features, and opening a pull request, But i was wondering if it is at all relevant, and will you be willing to merge and add those features in to the project?
(of course assuming my code will be decent :-) )

Support automatic timestamps for arbitrary values

In order to avoid out-of-order timestamps (which cause run-time errors), it would be very useful IMO to be able to delegate time-stamping to the module, e.g. via:

TS.ADD key * value

memory usage

Hi, not a bug bit looking for advice.

I'm developing a POC using this module which operates in a stream of metrics, creating individual timeseries for different keys within this stream. the features available here has been very useful so far.

The only thing I am wondering is what settings (configurations, compaction rules etc) i need to consider when trying to estimate memory usage.
best case scenario would be some basic rules of thumb estimates :)

Currently i've created about 1 million timeseries keys (all of which are quite small), and have observed a memory usage of around 8gb.

This seems quite high, and in digging into a dump.rdb (using sripathikrishnan/redis-rdb-tools) i discovered that the memory usage is underreported by that tool, as the total of my dump analysis is only about 1.2gb. this has stymied analysis a little.

i'm quite early in figuring this out, but it seems possible that there is a non trivial overhead to create many small timeseries keys?

Support adding multiple or weighted values

In case a client or intermediate layer has already aggregated several samples (e.g. an average) and the client does not have (or does not need) the originals to submit to the time series, it could be useful to have ability to add a value with a "weight", so that it will count as several samples when aggregating.

For example, I may have 10 samples whose average value is 5.25 but I do not know what each sample value is and I don't even care because I am only interested in the rolling average. One solution would be to do:

MULTI
TS.ADD key t 5.25
TS.ADD key t 5.25
...
TS.ADD key t 5.25
EXEC

But a better solution could be:

TS.ADD key t 5.25 5.25 ... 5.25

or

TS.ADD key t 5.25 REPEAT 10

or

TS.ADD key t 5.25 WEIGHT 10

If multiple fields per sample are supported then one of them could simply be the weight and an aggregation could reduce (weight, value) => (weight * value)

Timestamp Too Old

"TSDB: timestamp is too old" errors are being thrown. I checked the timestamp via epoch conversion and they correctly reflect only two days ago. The TTL is set to 2 weeks. How can I resolve this error?

Retain compaction context in rdb

Add compaction

Add a compaction command. It should take in a time range and some operation. It will then mutate the underlying chunks so that there are fewer chunks that represent the same time range, but are reduced by the specified function.
Some good functions to support are min, avg, max

This would allow the use case of having a fixed size of memory, yet still be able to retain some historic data at a lower granularity in time. Having different functions allows for an opinionated yet versatile way of reducing the data.

Add TS.RANGE command complexity description

Hello danni-m! Please tell about the time complexity of TS.RANGE in the README.md.

Calculate correlation between two keys

Hi,

I'm suggesting a feature to calculate Pearson Correlation Coefficient between two keys.
Two applications I could think of for this feature include:

• Gene Expression Correlation: for example, for constructing Gene co-expression networks
• System monitoring: for example, create alert when a negative correlation is detected between CPU usage and incoming requests

I hope you'll find it interesting.
Please see PR #23
Thanks 🌷

TS.RANGE should return a list of 2-tuple instead of a list

This will make the result set cleaner.
Hopefully, parsing will be a lot easier too.

Question on chunks

Hey so for my purposes I want to separate chunks by a certain time range. For example, the last 5 minutes- so after 5 minutes whatever samples have arrived are "locked in" so-to-speak and a new chunk is started. In other words, separate chunks by N seconds, as opposed to N-samples. Is there a way to do this already? If not, how would you recommend attacking?

Compressing timestamps

I think compressing timestamps before writing to disk would be cool. Even when storing on ram. I don't know any compression libraries for doubles(data type that used for values), so I can't suggest anything to compress values.

How about this idea?

Example integer compression libraries:
https://github.com/lemire/simdcomp (BSD)
https://github.com/powturbo/TurboPFor (GPL V2)

has it tested on arm?

It is really a great module!
I want use it on arm Linux platform ,as i know Starting with Redis 4.0 ARM and the Raspberry Pi are officially supported platforms.How about this module?

add count aggregated range query

Compilation Issue on MacOS Sierra v10.12.6

When I tried to compile on MacOS, I get the following:
$make all
/Applications/Xcode.app/Contents/Developer/usr/bin/make -C ../RedisModulesSDK/rmutil
gcc -g -fPIC -O3 -std=gnu99 -Wall -Wno-unused-function -I../ -c -o periodic.o periodic.c
periodic.c:34:5: warning: implicit declaration of function 'clock_gettime' is invalid in C99
[-Wimplicit-function-declaration]
clock_gettime(CLOCK_REALTIME, &ts);
^
periodic.c:34:19: error: use of undeclared identifier 'CLOCK_REALTIME'
clock_gettime(CLOCK_REALTIME, &ts);
^
1 warning and 1 error generated.
make[1]: *** [periodic.o] Error 1
make: *** [rmutil] Error 2

How can resolve the issue? Where should I declare CLOCK_REALTIME?
The command, $uname -s returns the value as Darwin

Also, I am new to use of redis module. So, appreciate your time to resolve this issue. Thank you,

Redis rdb save crashes if no samples yet

I actually added a fix locally and didn't have time to PR. But looking over my fix, I'm not sure it's totally correct. From what I remember, the problem is redis will crash when trying to save to rdb if there are no samples yet. I discovered it by setting my redis.conf to save every minute, regardless of changes to keyspace, in order to test a backup copying cron job. I traced it down the function below, and seen is my go at fixing. This does seem to work, but not sure about the if-else logic.

 Sample *SeriesItertorGetNext(SeriesItertor *iterator) {
     while (iterator->currentChunk != NULL)
     {
        Chunk *currentChunk = iterator->currentChunk;
 -      if (ChunkGetLastSample(currentChunk)->timestamp < iterator->minTimestamp)
 +      Sample * lastSample = ChunkGetLastSample(currentChunk);
 +      if ( !lastSample )
 +         return NULL;
 +
 +      if (lastSample->timestamp < iterator->minTimestamp)
        {
           iterator->currentChunk = currentChunk->nextChunk;
           iterator->chunkIterator = NewChunkIterator(iterator->currentChunk);
           continue;
        }
 -      else if (ChunkGetFirstSample(currentChunk)->timestamp > iterator->maxTimestamp)
 +      else if ( (ChunkGetFirstSample(currentChunk)) &&
 +            ChunkGetFirstSample(currentChunk)->timestamp > iterator->maxTimestamp)
        {
           break;
        }