Is this still maintained?

[Sinderella]

Is this still maintained?

[TheOldMan2000]

Hi @Sinderella

If you need an alternative until this is fixed, I managed to get the other Redis Infrastructure v0.0.3, it comes with the Redis Dashboard for Prometheus Redis Exporter 1.x and once I fixed a mistake I made, it works fine and connects immediately.

I found out it does work with with multiple instances/pools on the same host, I had my list of instances in my Prometheus folder named as a yml file instead of json, my bad. I had installed the full Redis Exporter plugin and it works great but maybe the built-in one in Grafana Agent would have been sufficient. The Dashboard is a bit basic but is neat and tidy.

[Sinderella]

That's neat! I tried to use that one earlier but it didn't work for me. Just the "Redis" one worked for me, but still a bit buggy. I guess I'll have to do more debugging. Thanks!

[slorello89]

FYI @Sinderella - my team is taking over maintenance of this plugin - we'll be adding a few new features, but there's also of course a bunch of issues that have piled up in the backlog that we'll be taking a look at and addressing.

@NikolaBorisov @Sinderella RE TS.MRANGE -it's one command we have particular interest in - be curious to know what your usage is like and what sorts of issues you're seeing with some reliable ways of reproducing them so we can look into it.

[TheOldMan2000]

Sadly I think it’s dead. Doesn’t work for me at all using a single local machine.

[NikolaBorisov]

I'm trying to use as datasource for TS.MRANGE command. Seems very buggy and not deterministic. Running the same query shows me reasonable graph every 5th request. Does anyone know of a Grafana datasource plugin for redis time series data?

[Sinderella]

I'm trying to use as datasource for TS.MRANGE command. Seems very buggy and not deterministic. Running the same query shows me reasonable graph every 5th request. Does anyone know of a Grafana datasource plugin for redis time series data?

@NikolaBorisov TS.MRANGE works fine for me for a narrow range. It's a bit slow when the range is too wide, sometimes it became frozen. I don't think there's an alternative.

[NikolaBorisov]

What version of Grafana do you have?

[Sinderella]

What version of Grafana do you have?

I've been on the latest branch of version 8 the last few months, just switched to 9.5.1, so issue so far.

[NikolaBorisov]

Good to hear @slorello89. We are using latest version of redis stack. We have time series data in redis about machine learning model usage time. When I try to make a grafana dashboard with this data, the dashboard is empty. Every 5th refresh some data shows up. I have grafana 9.3.1.

I'm also not sure how to use the "value label" filed?

[slorello89]

Hi @NikolaBorisov - interesting, still trying to work out how everything works - it looks like (at least in my configuration) the TS.MRANGE command will send a query for a range over the last hour - you can verify this by looking at what query it sends:

I'm still working out where that bit of configuration comes from (I'm not setting it). But that's what's telling it what range to pull from Redis.

I'd be particularly interested to see what that interval is for those dead instances - for what it's worth whenever I run it it seems to work just fine.

I don't suppose you can monitor redis while you're running those? redis-cli monitor | grep ts.mrange would be able to show you what's flowing through to Redis without cluttering you with all the TS.ADDs

[slorello89]

Ahah - now I see - it's going off the range from the time box of course 😆

so sounds like almost definitely something's up with the returned payload or the state of your time-series or perhaps the way it's formatting the command.

[NikolaBorisov]

Here is what i got from monitor

"ts.mrange" "1686039976565" "1686083176565" "AGGREGATION" "sum" "60000" "WITHLABELS" "FILTER" "series=usage_time"

I think this is correct. The strange bug I'm seeing is after I click refresh in grafana 80% of the time I don't see a new TS.MRANGE coming to redis. Also the screen clears. Also the fill missing button does not seem to work. Also there are no values...

[slorello89]

I don't know how frequently these time series are being added to - but I wonder if this is just due to the enormity of what you're requesting here - you are requesting 12 hours of data (43,200,000 milliseconds) for 10 timeseries - so even if you were only populating those time-series once a second (I don't know the frequency of their updates - but once a second would be what we consider lightly used) you are looking at 432,000 unique records the TS.MRANGE has to analyze - TS.MRANGE's time complexity is based off the number of records it's ranging over.

I'd be curious to see what the slowlog looks like after one of these SLOWLOG GET

You might want to consider not leaning into the aggregation in the MRANGE itself and rather using compaction rules within your timeseries so it doesn't have to to do all those calculations. If you set your rules up correctly you wouldn't need the aggregation at all and you'd have a guaranteed 7200 records (60 minutes * 12 hours * 10 time-series) that you'd have to range over - which should be quite reasonable.

[NikolaBorisov]

The amount of data is not a problem here. I have not written a lot of data in this series and I ask for the aggregation of 1min. The redis command completes in <100ms. The bugs are in the grafana redis plugin.

1. After refresh of the graph sometimes the query does not make it to redis. Graph is empy, but series are listed in the legend
2. legend had series names, but no values.
3. the fill if empty is not working. should draw 0 for all the times where there is no data.

[slorello89]

Hi @NikolaBorisov - what does your setup look like - I cannot get this to happen on Grafana 9.3.1 - but I have everything running locally so may not be representative of what you are experiencing. Each time I run the TS.MRANGE I have no issues. If the commands are only taking <100ms than it's probably not a perf issue (FWIW 100ms is solidly what we could call a slow operation - but still shouldn't cause what you are seeing.)

[NikolaBorisov]

I'm running Grafana and redis in k8s. I did some more digging and it looks like sometimes the grafana server returns large set of results(400+) with maybe all the data and sometimes it returns smaller set of result (121)

[NikolaBorisov]

I think the issue is probably in the grafana server side. To see the issue probably you need large number of series to return. When I get the small responses I don't see a query going to redis from grafana. So i think it is trying to return results from some sort of cache or something?

[slorello89]

Interesting @NikolaBorisov - so you are expecting to have 400+ time-series in your result set? Sorry only saw 10 in the graph you sent before.

I tried this with 400 time series - all with the same label foo=bar - this time I inserted 1 record per second to a random time series (note 1 per second total not 1 per second per time series). The operation in Redis itself only takes about 15ms (beast) - but it takes several minutes for anything to render on the front-end - and when it does Grafana notes that you can only see the first 20 of these time series.

So I expect what's happening is that either the backend plugin or the front-end is getting hung by the enormous amount of data it's getting. Considering the javascript is literally hanging to the extent that it's giving me the "do you want to kill this page" messages - I suspect it might be the frontend but not sure

At least I can reproduce it now - so I can take a look.

[slorello89]

@NikolaBorisov - in each of those frames how large are the arrays of values JSON path $.results.A.frames[0].data.values[0].length (check on something like https://jsonpath.com/) (assuming you are using query A? Should be something ~ 720 give or take I think?

[NikolaBorisov]

My time series are very sparse most of them had no data points during this time range (60%). a few will have 720 points.
The response from the grafana backend to the frontend is only 150kb for my data.

[slorello89]

@NikolaBorisov - so if you are using fill values - that would fill the values of all the time-series present in the result set (might only be between the min and max time in the series result-set). I just noticed something in the code that might explain your issue?

Is your deployment of Redis a cluster? If so are you using Redis Cloud/Redis Enterprise?

If yes - you are in a cluster, and no - you are not using Redis Enterprise - that could be the problem.

It looks like the from field is being fed to the client as a key. If it's a cluster deployment, the key is hashed to determine which instance the client should go to to service it's request.

Effectively this means the shard it would go to for the TS.MRANGE would be random - in a Enterprise/cloud deployment that's actually ideal because the DMC proxy will fan the TS.MRANGE out to all the shards in the cluster - bringing you back a complete result set. However, if you are not using RE/Cloud - it will not fan the TS.MRANGE out, limiting your result set to what's on the shard it happens to go to. I think that would more or less match the behavior you're seeing - if you are monitoring a Redis instance and you had 5 master shards only 1/5th of the commands would reach your shard, and your result set would change more or less on every request given the from changes each time you reissue the request.

[NikolaBorisov]

I have just one instance of redis stack not in a cluster.

…

On Wed, Jun 14, 2023, 8:53 AM Steve Lorello ***@***.***> wrote: @NikolaBorisov <https://github.com/NikolaBorisov> - so if you are using fill values - that would fill the values of all the time-series present in the result set (might only be between the min and max time in the series result-set). I just noticed something in the code that might explain your issue? Is your deployment of Redis a cluster? If so are you using Redis Cloud/Redis Enterprise? If *yes* - you are in a cluster, and *no* - you are not using Redis Enterprise - that could be the problem. It looks like the from field is being fed to the client as a key. If it's a cluster deployment, the key is hashed to determine which instance the client should go to to service it's request. Effectively this means the shard it would go to for the TS.MRANGE would be random - in a Enterprise/cloud deployment that's actually ideal because the DMC proxy will fan the TS.MRANGE out to all the shards in the cluster - bringing you back a complete result set. However, if you are *not* using RE/Cloud - it will *not* fan the TS.MRANGE out, limiting your result set to what's on the shard it happens to go to. I think that would more or less match the behavior you're seeing - if you are monitoring a Redis instance and you had 5 master shards only 1/5th of the commands would reach your shards, and your result set would change more or less on every request given the from changes each time you reissue the request. — Reply to this email directly, view it on GitHub <#296 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABNAAXF7UNE7GQHT22XY7LXLG64LANCNFSM6AAAAAAXOILVT4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[slorello89]

@NikolaBorisov - I am going to close this issue as the fundamental question of the issue has been answered, split out your issue into #303