doublemarket/grafana-rrd-server

A HTTP server that helps Grafana read RRD files via Grafana Simple JSON Datasource plugin

grafana-rrd-server's Issues

Can not get any data inside Grafana

Hi,
thanks for this tool.
I could successfully add the grafana-rrd-server as a new Data-Source in Grafana v7.3.2.
But when I want to add a query, there are no entries shown.

I used even your sample files for testing, but still no success.
Here my configuration:

copied example files directly to the server (for testing purposes)
name@server:~/go/bin$ ls
annotations.csv grafana-rrd-server percent-idle.rrd percent-user.rrd sample.rrd

starting grafana-rrd-server
name@server:~/go/bin$ sudo ./grafana-rrd-server -p 9000 -s 60 -r ./ -a ./annotations.csv

On a manual call with a webbrowser at the /search site: http://192.168.x.17:9000/search
the grafana-rrd-server gives following output:

name@server:~/go/bin$ ./grafana-rrd-server -p 9000 -s 60 -r ./ -a ./annotations.csv
ERROR: Cannot decode the request
EOF
Updating search cache.

As URL for the Grafana Simple JSON Datasource I used "192.168.x.17:9000". Must there be defined any name behind the port?
Can you give me any hints?
Thanks!

Alerting within Grafana

Not a question about grafana-rrd-server itself, but is there any way to generate alerts on the data from this? From my reading so far the Simple JSON datasource can't be used with alerting, https://github.com/simPod/GrafanaJsonDatasource seems to support the data format used by grafana-rrd-server, but also doesn't support alerting.

Thank you for writing this.

More granular logging

• HTTP format logging
• Ability to specify a log file path

Grafana fails when no datapoints

Grafana fails when there are no datapoints.

I think response is something like this:
[{"target":"acat:0244:INOCTETS","datapoints":null},{"target":"acat:0002:OUTOCTETS","datapoints":null}]

Follow symbolic links

• /search
  • doesn't follow symbolic links because it uses filepath.Walk(), which doesn't follow symlinks.
• /query
  • doesn't follow symbolic links but it uses go-zglob, which says it follows symlinks.
  • follows symbolic links in the middle of the path.
  • doesn't follow symbolic links in the top directory.

for an initial version

critical

• Respond empty data when choosing last XX

fixme

• Parameters are fixed and not configurable
  • rrd file path
  • default step
  • port number
• Does not recursively search RRD files
• Cannot retrieve multiple DSs (e.g. cannot retrieve DSs from load.rrd)
• Panic occurs when requesting non-existing metric

Prebuilt binary crashing on RHEL 6

I've downloaded the prebuilt binary, and it keeps crashing with this error:

grafana-rrd-server: symbol lookup error: grafana-rrd-server: undefined symbol: rrd_info_r

This happens every time it's sent a POST from Grafana.

It appears to be looking for an old version of librrd:

root@host:~ ❱❱❱ objdump -x /usr/local/bin/grafana-rrd-server  | grep librrd
  NEEDED               librrd.so.4

librrd.so.4 is present on my machine (version 1.3.8, installed by yum) but it does not have that symbol. I've also got librrd.so.8 (version 1.6.0 installed from source) and it does have it:

root@host:~ ❱❱❱ objdump -x /usr/lib64/librrd.so.4 | grep rrd_info_r
root@host:~ ❱❱❱ objdump -x /usr/local/lib/librrd.so.8 | grep rrd_info_r
0000000000030860 g     F .text	0000000000000c06              rrd_info_r
root@host:~ ❱❱❱ 

Also checked RHEL 7; it also provides librrd.so.4 version 1.4.8, and doesn't contain that symbol either.

Add capacity to ignore hidden/temporary files for target directory

Hello,

Great project you have there.
I ran into an issue, the target directory i'm using for grafana-rrd has some temporary hidden files (used by rsync). These hidden files creates some exceptions when trying to read/display data on my graphs on grafana, making me unable to use the simplejson plugin.

So, i think it could by great to have support for hidden files, so that we can specify whether we want to include all files in the target directory or ignore hidden files (.hiddenfile).

License?

Before I download this and give it a shot, is there a license for this?

cannot unmarshal string into Go struct field QueryRequest.panelId of type int64

#$HOME/go/bin/grafana-rrd-server -r doublemarket/grafana-rrd-server/sample/
ERROR: Cannot decode the request
json: cannot unmarshal string into Go struct field QueryRequest.panelId of type int64
Updating search cache.
ERROR: Cannot decode the request
json: cannot unmarshal string into Go struct field QueryRequest.panelId of type int64
ERROR: Cannot decode the request
json: cannot unmarshal string into Go struct field QueryRequest.panelId of type int64
^C

When Grafana query, cli report error.

Does it support linux/386?

Hi, i am new here. And I am wondering if this support linux/386 OS? My RRD is running on CentOS 6 386 OS. It would be highly appreciated if you can provide linux/386 release.

Thank you.

Metric Alias File

Hello,
This feature wish maybe is similar to #49 and #30:

Like the annotation file, I would like a "metric human name" file, that is read and send as metric name to Grafana.

My use-case/example:
We've a generic RRD Statistic Collector (https://github.com/mschlenstedt/LoxBerry-Plugin-Stats4Lox) where we use incremental numbers for RRD files (0001, 0002,...) and manage the statistic name, entered by the user, in an own database.
Now, for the Grafana users it's difficult to find out what the metric 0034 is.

It would be nice, if there would be a "matching" file, e.g. for a 0024.rrd a 0024.inf file, that is read and holds a human readable name ("Outdoor temperature") sent as metric name.

Please correct me, if something like this is already possible!

Christian

HTTP panics

Unable to run the prebuilt binary (#37) I tried building my own. Pretty simple process:

yum install golang
PKG_CONFIG_PATH=/usr/local/lib/pkgconfig go get github.com/doublemarket/grafana-rrd-server
./go/bin/grafana-rrd-server -r /var/lib/mrtg

Everything compiled without issue, but whenever it gets any HTTP request:

2018/03/01 12:59:00 http: panic serving [::1]:37128: runtime error: invalid memory address or nil pointer dereference
goroutine 18 [running]:
net/http.(*conn).serve.func1(0xc420100000)
	/usr/lib/golang/src/net/http/server.go:1697 +0xd0
panic(0x6aaa80, 0xa50db0)
	/usr/lib/golang/src/runtime/panic.go:491 +0x283
main.search.func1(0x700b10, 0x9, 0x0, 0x0, 0xa5c660, 0xc4200fa1e0, 0x10, 0x6b8900)
	/root/go/src/github.com/doublemarket/grafana-rrd-server/rrdserver.go:125 +0x46
path/filepath.Walk(0x700b10, 0x9, 0xc420108040, 0xc420121ca0, 0x42ae86)
	/usr/lib/golang/src/path/filepath/path.go:401 +0x76
main.search(0xa60220, 0xc42013c000, 0xc42010a400)
	/root/go/src/github.com/doublemarket/grafana-rrd-server/rrdserver.go:123 +0x98
net/http.HandlerFunc.ServeHTTP(0x7138e0, 0xa60220, 0xc42013c000, 0xc42010a400)
	/usr/lib/golang/src/net/http/server.go:1918 +0x44
net/http.(*ServeMux).ServeHTTP(0xa8a2c0, 0xa60220, 0xc42013c000, 0xc42010a400)
	/usr/lib/golang/src/net/http/server.go:2254 +0x130
net/http.serverHandler.ServeHTTP(0xc420072f70, 0xa60220, 0xc42013c000, 0xc42010a400)
	/usr/lib/golang/src/net/http/server.go:2619 +0xb4
net/http.(*conn).serve(0xc420100000, 0xa60760, 0xc420106040)
	/usr/lib/golang/src/net/http/server.go:1801 +0x71d
created by net/http.(*Server).Serve
	/usr/lib/golang/src/net/http/server.go:2720 +0x288

I'm completely unfamiliar with Go, so if there's any way I can provide more details please let me know! Running RHEL 6.9, go 1.9.2, and rrdtool 1.6.0

Query fails if path contains white-space

I am trying to query data from a 3d party RRD database, so I have no control over the name of the RRD files/paths. Some of the paths contain a space character. The grafana-rrd-server always returns a 500 status.
I've tried escaping the space with a backslash '\'.
And I've tried using the URL encoding '%20'.
And I've tried wrapping the entire name in double-quotes (").
Nothing seems to work.

Example target:
$testhost/os/interface/Ethernet 1_1/ifInOctets:ds0

SimpleJson is deprecated - which plugin to choose

SimpleJson is marked as deprecated with the following information:

Angular plugin
This data source plugin uses a deprecated, legacy platform based on AngularJS and will stop working in future releases of Grafana.

Do you have any other recommendation for a working plugin?

Wildcard in a query target

e.g. Giving mysql.query_count.* as a target in a query, then the query endpoint returns datapoints of mysql.query_count.select, mysql.query_count.update and so on.

Can't execute grafana-rrd-server

Hello,

I can't execute grafana-rrd-server command on my CentOS 7...

I have :
"error while loading shared libraries: librrd.so.8: cannot open shared object file: No such file or directory"

I don't understand where this error can from.

I have never learned or used GO language :/

Thank you

Wildcard in a query target in the ds

I would like to use a wildcard in the ds part of the target like mentioned in #22.
*:host-metric-one:42. Why is it not possible to use wildcards in ds?

Support for Annotations

The /annotations endpoint currently returns the one message. It should support reading a JSON file and returning the content as a return value.

Aliase for the Devices

Hi,

instead of the device ID and Datasource ID can we have a corresponding list to map it to a device name and datasource name?

Considering different extensions RDD files

Hi,

I'm using a collector which use rrd files with different extensions : "rsm", "rrm", "xrm", and "db".

When i try to use grafana-rrd-server with these files, it seems that it is not working.

Is it a limitation or an issue ?

Thank you.

delay with real content in rrd file

Hello
I use different releases from 0.0.2 to 0.0.5 and get strange behaviour

0:Array[6467703.333333333,1612262700000]
1:Array[6467743.333333333,1612263000000]
2:Array[6467768.666666667,1612263300000]

this is in grafana but if i use rrdtool directly i see last timestamp data

1612261800: 6,4685253333e+06
1612262100: 6,4677073333e+06
1612262400: 6,4677633333e+06
1612262700: 6,4677293333e+06
1612263000: 6,4677033333e+06
1612263300: 6,4677433333e+06
1612263600: 6,4677686667e+06
1612263900: -nan

how to fix it?

rrdcached

Is rrdcached supported?

hardcoded consolidation function (CF)

Hi,

It seems that consolidation function (CF) is hardcoded:

grafana-rrd-server/rrdserver.go

Line 193 in 3e319cf

fetchRes, err := rrd.Fetch(filePath, "AVERAGE", from, to, time.Duration(config.Server.Step)*time.Second)

For many purposes, average is not right consolidation function. Please consider exposing all consolidation functions available in rrd files.

Shorter target names

When an RRD file is in a deep directory, the target name will be long. If we have an option to pick only some directory names or a DS for the target name in the path, that makes graph legend names shorter.

Execute ./grafana-rrd-server

I'm using CentOS 6.9.
When i execute ./grafana-rrd-server it gives the error:
./grafana-rrd-server: line 1: syntax error near unexpected token `newline'
./grafana-rrd-server: line 1: '!DOCTYPE html'

Please help me.

Please exporter the rdd data to prometheus

Graph aggregation / item renaming

Hello,

I'm not sure if my question related more to your project or simple json datasource, but here it is: is there a way to get a agregated graph from two values, let say, uplink A usage and uplink B usage -> Total Uplink usage graph?

As far as I see, there is no way to do so, or even simply to rename item:

Thank you in advance for the answer.

Grafana 7, Expected response data to be array, got object

With Grafana 7, when I add a graph and select the JSON datasource to the grafana-rrd-server I get an error saying:
Expected response data to be array, got object.

With F12 I do see /search return my data:

["pdns_recursor:concurrent-queries", "pdns_recursor:cache-hits", "pdns_recursor:server-parse-errors",…]

The data is generated by PowerDNS, nothing I can really do about it.

build and publish a new release featuring search"target" and cache

Support cache and target in search endpoint is impemented but not published as a standalone release. Some customers prefer not to build the application from source locally

We are considering using the grafana-rrd-server in our health monitoring environment featuring the really big number of rrd files. The ability to filter the data scan is vital to us.