No new data - even when restarting the services

[yunohost-apps]

No new data - even when restarting the services

[JimboJoe]

Actually, NetData sends data to influxdb (the database), and Grafana queries influxdb to put up its charts.
Is influxdb up and running?
What logs do you have in netdata logs relative to backend?

[lapineige]

Yes, influxdb up and running.
Netdata is working too on its own app.

I'll check the logs :)

[lapineige]

Hum.. where is the log located ? I'm not able to find it (using find command)

[JimboJoe]

Here they are: /opt/netdata/var/log/netdata/

[lapineige]

Thanks :)

Access.log and debug.log are empty.
In error.log, I have this:

2017-08-05 14:51:27: netdata: ERROR: PROCFILE: Cannot open file '/proc/diskstats' (errno 2, No such file or directory)
2017-08-05 14:51:28: netdata: ERROR: PROCFILE: Cannot open file '/proc/diskstats' (errno 2, No such file or directory)
2017-08-05 14:51:29: netdata: ERROR: PROCFILE: Cannot open file '/proc/diskstats' (errno 2, No such file or directory)
2017-08-05 14:51:30: netdata: ERROR: PROCFILE: Cannot open file '/proc/diskstats' (errno 2, No such file or directory)
2017-08-05 14:51:31: netdata: ERROR: PROCFILE: Cannot open file '/proc/diskstats' (errno 2, No such file or directory)
2017-08-05 14:51:32: netdata: ERROR: PROCFILE: Cannot open file '/proc/diskstats' (errno 2, No such file or directory)
2017-08-05 14:51:33: netdata: ERROR: PROCFILE: Cannot open file '/proc/diskstats' (errno 2, No such file or directory)
2017-08-05 14:51:34: netdata: ERROR: PROCFILE: Cannot open file '/proc/diskstats' (errno 2, No such file or directory)
2017-08-05 14:51:35: netdata: Too many logs (201 logs in 200 seconds, threshold is set to 200 logs in 1200 seconds). Preventing more logs from process 'netdata' for 1000 seconds.

(and the list of this error is kind of huge)

[JimboJoe]

@lapineige you'll have to take the time to analyze everything as these errors are usual (it automatically looks for available monitoring means).
Look for messages relative to "backend"

[lapineige]

I made a search for "backend".

I have this:

2017-08-04 10:55:07: netdata: ERROR: Health alarm 'netdata.backend_metrics.backend_last_buffering' has update every 10, less than chart update every 60. Setting alarm update frequency to 60.
2017-08-04 10:55:07: netdata: ERROR: Health alarm 'netdata.backend_metrics.backend_metrics_sent' has update every 10, less than chart update every 60. Setting alarm update frequency to 60.
2017-08-04 10:55:07: netdata: ERROR: Health alarm 'netdata.backend_metrics.backend_metrics_lost' has update every 10, less than chart update every 60. Setting alarm update frequency to 60.
2017-08-04 10:55:07: netdata: INFO : BACKEND configured ('opentsdb' on 'localhost:4242' sending 'average' data, every 60 seconds, as host 'MyHost', with prefix 'netdata')

Some lines like this one:

netdata: ERROR: File /opt/netdata/var/cache/netdata/netdata.backend_ops/write.db is too old. Clearing it.

If I look around the timestamp for the last data point I have, I see nothing related to backend.

But during the day I have this :

netdata: ERROR: Failed to update database backend 'localhost:4242'
And
alarm-notify.sh: INFO: sent email notification for: mydomain.tld netdata.backend_metrics.backend_metrics_lost is CLEAR to 'root'

[JimboJoe]

OK, so it looks like update queries to influxdb are failing... what about influxdb logs ?

[lapineige]

/var/log/influxdb is empty :/

[JimboJoe]

I see influxdb logs are written in /var/log/syslog.

[lapineige]

Thanks.

The oldest file I have for syslog (and older and archived files) is too recent to contain the log related to the day the issue was raised.

Regarding the log I have right now, I don't see anything special (but maybe it's because I don't know what to search ^^): SQL queries, HTTP request (GET), ...

Maybe those lines are not not normal ?

failed to store statistics: timeout service=monitor

failed to write point batch to database "opentsdb": timeout service=opentsdb

(These are the only lines I get when searching for "fail" word)

[JimboJoe]

Hello,
Can you confirm your problems started happening with NetData 1.7.0 update? The time-frame seems to coincide with that update.
Because I've discovered this NetData issue that confirms that some metrics' identifiers have changed in this version. Do you confirm you don't have any data point for every metric? Are you using the dashboard provided by default or custom one?
I'm using the default one and "some" metrics are now empty (Disk I/O's for instance), but most are still valid (CPU utilization for instance).

[lapineige]

I had no data points at all (for every metric) except the ones gathered before the issue appeared.
Default dashboard.

I did a fresh reinstall of grafana a few hours ago, but before re-installing I removed every influxdb+grafana file that I found (some are not deleted during removal).
It works fine for the moment for CPU, Memory, Networking, System Load, and Netdata Backend. I was waiting for a few hours to be sure the issue doesn't raise again.
But indeed, I don't have all the metric (disk I/O indeed).

I'm quite sure that I did this update right after its release.
I'll check netdata version just in case.

[JimboJoe]

I propose to close this issue in the absence of activity. Please feel free to reopen it if the problem occurs again for you!