Cannot visualize/retrieve data from Grafana for period longer than 7 days

[balena-labs-projects]

Cannot visualize/retrieve data from Grafana for period longer than 7 days

[vschinazi]

Just to follow up on my previous message. It now now seems that my system is having a hard time connecting to the Influx DB and this may be the source of the porblem. I am including a snippet from my log which notes a series of server errors:

10.11.19 17:39:10 (+0100) <telegraf> 2019-11-10T16:39:10Z E! [agent] Error writing to outputs.influxdb: could not write any address 10.11.19 17:39:12 (+0100) Failed to kill service 'influxdb sha256:ea8f584a602a29e0d1da6eec13fec9238f9a3eca7add5eba06d92b93f0959f89' due to '(HTTP code 500) server error - cannot stop container: e88093528741c6ccd7c66cd08ff3bc4f0fad79567fcbee81ba0425ff22012841: Cannot kill container e88093528741c6ccd7c66cd08ff3bc4f0fad79567fcbee81ba0425ff22012841: connection error: desc = "transport: dial unix /var/run/balena-engine/containerd/balena-engine-containerd.sock: connect: connection refused": unknown ' 10.11.19 17:39:12 (+0100) Failed to kill service 'sensor sha256:57eb63236b553c575df4a4fc6e32bcabe20c48471a8206cce2d8243f436b1e77' due to '(HTTP code 500) server error - cannot stop container: e498d6085bda4f533622c7c7ae33d409e3701535fd8c121c4052abbab636175f: Cannot kill container e498d6085bda4f533622c7c7ae33d409e3701535fd8c121c4052abbab636175f: connection error: desc = "transport: dial unix /var/run/balena-engine/containerd/balena-engine-containerd.sock: connect: connection refused": unknown ' 10.11.19 17:39:13 (+0100) Failed to kill service 'grafana sha256:5e6679ac15b8eeb36e20fd93834639800b9b4262d136f484d99c9e2edecc5a21' due to '(HTTP code 500) server error - cannot stop container: 5d7d7c98f0dfe35559cbc7f4551c2ad4668ab292c2ba361551e01e67ab635613: Cannot kill container 5d7d7c98f0dfe35559cbc7f4551c2ad4668ab292c2ba361551e01e67ab635613: connection error: desc = "transport: dial unix /var/run/balena-engine/containerd/balena-engine-containerd.sock: connect: connection refused": unknown ' 10.11.19 17:39:13 (+0100) Failed to kill service 'telegraf sha256:d45bb176d41c53cce96dcc089ee784bb6b8d3116f697e31c9c9596cdfc10a958' due to '(HTTP code 500) server error - cannot stop container: d5d2dacecdbb07b595e4d784ad1b9b9c723c2c1678112ca44e49905f54b3be16: Cannot kill container d5d2dacecdbb07b595e4d784ad1b9b9c723c2c1678112ca44e49905f54b3be16: connection error: desc = "transport: dial unix /var/run/balena-engine/containerd/balena-engine-containerd.sock: connect: connection refused": unknown ' 10.11.19 17:39:14 (+0100) Killing service 'telegraf sha256:d45bb176d41c53cce96dcc089ee784bb6b8d3116f697e31c9c9596cdfc10a958' 10.11.19 17:39:14 (+0100) Killing service 'grafana sha256:5e6679ac15b8eeb36e20fd93834639800b9b4262d136f484d99c9e2edecc5a21' 10.11.19 17:39:14 (+0100) Killing service 'sensor sha256:57eb63236b553c575df4a4fc6e32bcabe20c48471a8206cce2d8243f436b1e77' 10.11.19 17:39:14 (+0100) Killing service 'influxdb sha256:ea8f584a602a29e0d1da6eec13fec9238f9a3eca7add5eba06d92b93f0959f89' 10.11.19 17:39:17 (+0100) Failed to kill service 'telegraf sha256:d45bb176d41c53cce96dcc089ee784bb6b8d3116f697e31c9c9596cdfc10a958' due to '(HTTP code 500) server error - cannot stop container: d5d2dacecdbb07b595e4d784ad1b9b9c723c2c1678112ca44e49905f54b3be16: Cannot kill container d5d2dacecdbb07b595e4d784ad1b9b9c723c2c1678112ca44e49905f54b3be16: connection error: desc = "transport: dial unix /var/run/balena-engine/containerd/balena-engine-containerd.sock: connect: connection refused": unknown ' 10.11.19 17:39:17 (+0100) Failed to kill service 'influxdb sha256:ea8f584a602a29e0d1da6eec13fec9238f9a3eca7add5eba06d92b93f0959f89' due to '(HTTP code 500) server error - cannot stop container: e88093528741c6ccd7c66cd08ff3bc4f0fad79567fcbee81ba0425ff22012841: Cannot kill container e88093528741c6ccd7c66cd08ff3bc4f0fad79567fcbee81ba0425ff22012841: connection error: desc = "transport: dial unix /var/run/balena-engine/containerd/balena-engine-containerd.sock: connect: connection refused": unknown '

Please let me know if you have any idea on how to troubleshoot this.

Thank you

[chrisys]

Hey @vschinazi I would start by checking your SD card. In a lot of instances, strange behaviour such as this can be attributed to a failing SD card. You could start by using the device diagnostics feature within the device page on balenaCloud (it's bottom left in the menu). This will do some basic checks for you.

[vschinazi]

Hi Chris

I finally fixed this. It was my SD. Sorry forgot to close this issue.