jsdelivr/globalping-hwprobe

OS image for Globalping hardware probes. Become a GitHub Sponsor to get yours!

Globalping Hardware Probe Firmware

This is the firmware of the hardware probe we ship to our supporters. It was tested only on our specific ARM-v6 probes and we don't guarantee it will work correctly on other similar devices. As a user it is not necessary to update your firmware unless something breaks or you need to replace the SD card.

How to get a Globalping hardware probe

Download the latest firmware

Check the Releases

The gz file is ready to get flashed to your SD Card, no preparation required.

Flashing the SD card

The compiled firmware can be flashed to an SD card using Raspberry PI Imager, balenaEtcher, Rufus or other similar software.

After the SD Card is correctly flashed and verified it can be inserted into the Globalping hardware probe and the probe powered up.

Hardware Probe startup process

1. After power-up the red LED will be on for the first 17 seconds
2. After this, the red LED will turn off and the green LED will start blinking
3. When the probe software has been started the green LED will go solid.

Errors

• Solid red LED ( the green LED never turns on)

  If during the startup process the green LED never turn's on, it could be a flash sdcard issue or the sdcard is not correctly installed on the slot.

• Blinking red LED

  Probe Software has failed, and the software restart is being done (should jump to solid Green when finishes).

Updates

The actual probe code that runs inside a docker container on the device is automatically updated. Learn more about the probe code

Accessing the probe

For security reasons there is no way to get shell access of a running probe. But for debugging purposes you can connect to it via SSH to get the container logs of the software probe. To connect:

1. Login into your router's web UI and check the list of connected devices. You should be able to spot the globalping hardware probe in the list.
2. Get the IP address of the probe
3. Login via ssh ssh logs@IP_ADDRESS e.g. ssh [email protected]
4. You can now see the log output of the docker container running the software probe on your device.

To access firmware logs (only really useful to firmware devs/firmware debug), repeat the above steps but with "devlogs" instead of "logs" as the user

Security

In addition to the security features of the software probe these are the extra safe guards we used to make the hardware device as secure as possible:

• The rootfs of the probe OS is read-only
• The kernel configuration was tunned to reduce the size and exploitable area
• The probe container runs completely from RAM
• The OS will automatically reboot every 3 days + random amount of hours between 1 and 48.
• The only user that is eligible to use SSH is the "logs" user, without shell access
• The OS was trimmed to have minimum attack surface

Building the firmware

The script was tested on Ubuntu 22.04 LTS. First install the required software:

apt update -y && apt install gawk wget git diffstat unzip texinfo gcc build-essential chrpath socat cpio python3 python3-pip python3-pexpect xz-utils debianutils iputils-ping python3-git python3-jinja2 libegl1-mesa libsdl1.2-dev xterm python3-subunit mesa-common-dev zstd liblz4-tool

Next create a user for the compilation process and clone this repo locally.

useradd -m compiler
su compiler
bash
cd /home/compiler
git clone https://github.com/jsdelivr/globalping-hwprobe

You can now run the bash script that will download all the necessary dependencies and build the firmware. NOTE: This process can take a couple of hours

cd globalping-hwprobe
bash build_firmware.sh 

After the build is done a firmware file with the extension ".sunxi-sdimg" will appear in the current directory.

USB update

The probe firmware can upgrade the container itself using a USB flash drive as the initiator for the process but not as a method of offline upgrade. This is done as a security measure to avoid "Evil maid " type of attacks.

USB update key

To create the USB update key is just a matter of using a USB flash drive formatted as fat with a file named "JSDELIVR.UPD" in its root. The file can be empty as its presence is checked, not the file content.

After the USB drive preparation is done, plug the USB Flash drive in the probe and power cycle it The probe will detect the USB flash and check for the presence of the update key file. If it's present, the upgrade process will start, with a rapid flashing of the GREEN led. At the end of the process, the initiator file "JSDELIVR.UPD" will be erased, and the probe automatically reboots.

USB factory reset key

If by any chance there is a need to go back to the container version bundled with the probe firmware, this can be quickly done by doing the same process as the USB Update, but instead with a file named "JSDELIVR.RESET"

globalping-hwprobe's Issues

restarts every 30 min or less

          Hey, the restarts are normal and they are there for security and stability reasons. The SSH connection is only there for rarely debugging, so no need to monitor it constantly.

Originally posted by @jimaek in #12 (comment)

Is it possible to change that behaviour through settings?
Is it normal that the restarts happening every 30 minutes, this arent the weekly updates/restarts.

Consider adding a second IPv6 registry

If GitHub doesn't add IPv6 support before we ship new probes. Related to jsdelivr/globalping-probe#189.

Log/report firmware version

It seems we don't log the firmware version at all, so I have no idea which one my probe currently has. We should log that on startup and possibly also expose it to the API, same as #27?

todo

• Make the device detectable by name "globalping-probe" on LAN
• Get IP via DHCP
• Fallback to a DNS resolver if DHCP doesnt provide one
• Show logs on SSH connection
• Limit SSH port to LAN connections

[2023-08-02 16:17:11] [WARN] [status-manager] Quality control ping test result is unsuccessful: ns1.dns.nl 14.2857% packet loss. Test pass.

hwprobe:
[2023-08-02 16:17:11] [WARN] [status-manager] Quality control ping test result is unsuccessful: ns1.dns.nl 14.2857% packet loss. Test pass.
This continues until probe gets disconnected.

from the same network and from different devices (not filtering activated) all ping commands run without any problem:

Ping-Statistik für 194.0.28.53: Pakete: Gesendet = 4, Empfangen = 4, Verloren = 0 (0% Verlust), Ca. Zeitangaben in Millisek.: Minimum = 35ms, Maximum = 40ms, Mittelwert = 36ms

So its an hwprobe issue

Originally posted by @skapytek in #22 (comment)

Quality control ping test result is unsuccessful

[2023-07-21 14:10:31] [DEBUG] [general] 'http' request 3Yx2QcDSKzwFKhDz received.
[2023-07-21 16:31:41] [WARN] [status-manager] Quality control ping test result is unsuccessful: k.root-servers.net 14.2857% packet loss. Test pass.
[2023-07-21 17:07:09] [DEBUG] [general] 'http' request WMVQfaScKvz0dFVP received.
[2023-07-21 17:31:51] [WARN] [status-manager] Quality control ping test result is unsuccessful: ns1.registry.in 14.2857% packet loss. Test pass.
[2023-07-21 18:52:04] [WARN] [status-manager] Quality control ping test result is unsuccessful: ns1.registry.in 14.2857% packet loss. Test pass.
[2023-07-21 19:03:26] [DEBUG] [general] 'ping' request xCRY5PDyw1FRLfDI received.
[2023-07-21 19:32:16] [WARN] [status-manager] Quality control ping test result is unsuccessful: ns1.registry.in 14.2857% packet loss. Test pass.

Strange cause other devices in the same network dont have any problems pinging the domains

Hardware probe not getting correct time or updated to new version

Hi globalping team,

I recently received my hardware probe and had it running for a couple of weeks. I noticed in the API page that my probe was one of the few that were not updated to the latest version. Also, logs from ssh logs@[ip] show that it has a wrong system date, and has been consistently reporting error.

Could not chdir to home directory /home/logs: No such file or directory
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
Checking for the latest version
Current version 0.10.1
Latest version
[2021-11-19 17:22:47] [INFO] [688] [general] Start probe version 0.10.1 in a production mode
[2021-11-19 17:22:48] [DEBUG] [688] [general] connection to API established
[2021-11-19 17:22:49] [INFO] [688] [api:connect] connected from <redacted>
node:internal/process/promises:288
            triggerUncaughtException(err, true /* fromPromise */);
            ^

RequestError: certificate is not yet valid
    at ClientRequest.<anonymous> (file:///app/node_modules/got/dist/source/core/index.js:760:107)
    at Object.onceWrapper (node:events:628:26)
    at ClientRequest.emit (node:events:525:35)
    at TLSSocket.socketErrorListener (node:_http_client:481:9)
    at TLSSocket.emit (node:events:513:28)
    at emitErrorNT (node:internal/streams/destroy:151:8)
    at emitErrorCloseNT (node:internal/streams/destroy:116:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
    at TLSSocket.onConnectSecure (node:_tls_wrap:1534:34)
    at TLSSocket.emit (node:events:513:28)
    at TLSSocket._finishInit (node:_tls_wrap:948:8)
    at ssl.onhandshakedone (node:_tls_wrap:729:12) {
  input: undefined,
  code: 'CERT_NOT_YET_VALID',
  timings: {
    start: 1637343110967,
    socket: 1637343110992,
    lookup: 1637343110995,
    connect: 1637343111094,
    secureConnect: undefined,
    upload: undefined,
    response: undefined,
    end: undefined,
    error: 1637343111283,
    abort: undefined,
    phases: {
      wait: 25,
      dns: 3,
      tcp: 99,
      tls: undefined,
      request: undefined,
      firstByte: undefined,
      download: undefined,
      total: 316
    }
  },
  options: Options {
    _unixOptions: undefined,
    _internals: {
      request: undefined,
      agent: { http: undefined, https: undefined, http2: undefined },
      h2session: undefined,
      decompress: true,
      timeout: {
        connect: undefined,
        lookup: undefined,
        read: undefined,
        request: 15000,
        response: undefined,
        secureConnect: undefined,
        send: undefined,
        socket: undefined
      },
      prefixUrl: '',
      body: undefined,
      form: undefined,
      json: undefined,
      cookieJar: undefined,
      ignoreInvalidCookies: false,
      searchParams: undefined,
      dnsLookup: undefined,
      dnsCache: undefined,
      context: {},
      hooks: {
        init: [],
        beforeRequest: [],
        beforeError: [],
        beforeRedirect: [],
        beforeRetry: [],
        afterResponse: []
      },
      followRedirect: true,
      maxRedirects: 10,
      cache: undefined,
      throwHttpErrors: true,
      username: '',
      password: '',
      http2: false,
      allowGetBody: false,
      headers: {
        'user-agent': 'got (https://github.com/sindresorhus/got)',
        accept: 'application/json',
        'accept-encoding': 'gzip, deflate, br'
      },
      methodRewriting: false,
      dnsLookupIpVersion: undefined,
      parseJson: [Function: parse],
      stringifyJson: [Function: stringify],
      retry: {
        limit: 2,
        methods: [ 'GET', 'PUT', 'HEAD', 'DELETE', 'OPTIONS', 'TRACE' ],
        statusCodes: [
          408, 413, 429, 500,
          502, 503, 504, 521,
          522, 524
        ],
        errorCodes: [
          'ETIMEDOUT',
          'ECONNRESET',
          'EADDRINUSE',
          'ECONNREFUSED',
          'EPIPE',
          'ENOTFOUND',
          'ENETUNREACH',
          'EAI_AGAIN'
        ],
        maxRetryAfter: undefined,
        calculateDelay: [Function: calculateDelay],
        backoffLimit: Infinity,
        noise: 100
      },
      localAddress: undefined,
      method: 'GET',
      createConnection: undefined,
      cacheOptions: {
        shared: undefined,
        cacheHeuristic: undefined,
        immutableMinTimeToLive: undefined,
        ignoreCargoCult: undefined
      },
      https: {
        alpnProtocols: undefined,
        rejectUnauthorized: undefined,
        checkServerIdentity: undefined,
        certificateAuthority: undefined,
        key: undefined,
        certificate: undefined,
        passphrase: undefined,
        pfx: undefined,
        ciphers: undefined,
        honorCipherOrder: undefined,
        minVersion: undefined,
        maxVersion: undefined,
        signatureAlgorithms: undefined,
        tlsSessionLifetime: undefined,
        dhparam: undefined,
        ecdhCurve: undefined,
        certificateRevocationLists: undefined
      },
      encoding: undefined,
      resolveBodyOnly: false,
      isStream: false,
      responseType: 'text',
      url: <ref *1> URL {
        [Symbol(context)]: URLContext {
          flags: 400,
          scheme: 'https:',
          username: '',
          password: '',
          host: 'api.github.com',
          port: null,
          path: [
            'repos',
            'jsdelivr',
            'globalping-probe',
            'releases',
            'latest'
          ],
          query: null,
          fragment: null
        },
        [Symbol(query)]: URLSearchParams {
          [Symbol(query)]: [],
          [Symbol(context)]: [Circular *1]
        }
      },
      pagination: {
        transform: [Function: transform],
        paginate: [Function: paginate],
        filter: [Function: filter],
        shouldContinue: [Function: shouldContinue],
        countLimit: Infinity,
        backoff: 0,
        requestLimit: 10000,
        stackAllItems: false
      },
      setHost: true,
      maxHeaderSize: undefined
    },
    _merging: false,
    _init: [ { timeout: { request: 15000 } } ]
  }
}

Node.js v18.7.0
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
Checking for the latest version
Current version 0.10.1
Latest version
[2021-11-19 17:32:10] [INFO] [688] [general] Start probe version 0.10.1 in a production mode
[2021-11-19 17:32:11] [DEBUG] [688] [general] connection to API established
[2021-11-19 17:32:11] [INFO] [688] [api:connect] connected from <redacted>
node:internal/process/promises:288
            triggerUncaughtException(err, true /* fromPromise */);
            ^

... (more identical stack traces and error logs omitted)

It seems like the stack trace was also caused by the incorrect system date.

Also, I'm not sure whether the incorrect date was caused by missing NTP client or the NTP calls being blocked by the Chinese firewall.

What is the recommended action to be taken here? Thanks!

how to check if hw-probe is working

Hi,
im using the hardware probe from my fiber internet at my home.
How can i check if my device is connected to the api and resolving requests?
I cant find any ID or something similair to check it on the web api site.

Probe not visible on the webapp

Error logs: https://paste.debian.net/1282752/

Docker log spamming

How is the current docker logging logic configured? What if the container decides to write 100TB of logs?
There are options to limit that, e.g. https://docs.docker.com/config/containers/logging/local/
Are we using them?

Upgrade Yocto and use the new CDN

We are now the new CDN of Yocto, lets use it ourselves for the FW https://git.yoctoproject.org/poky/commit/?id=a6753de86bd9f7d9ec8fbc70ccb36294d1497e88

Detect hardware probes as such

We didn't think about it originally but its useful to know. We need a way to detect a probe as software or hardware.

I think passing an env var to the container is the best way to do that.

GP_HOST_HW=true
GP_HOST_DEVICE=v1

And need to make sure we dont forget to update the device value when we start using new hardware

Using username "logs". Could not chdir to home directory /home/logs: No such file or directory

After self update my probe, stuck no such file or directory and force close

Shows error

Using username "logs".
Could not chdir to home directory /home/logs: No such file or directory

Try to flash 1.4 on new micro sd card also the same error

Accessing non-container logs

I got to a state where the probe booted, started a container update, and that update failed. Connecting via SSH to get the logs was not helpful in this case because it only said Error: No such container: globalping-probe - I already found what's most likely the problem in this case, but:

1. I can see there are some log messages like echo "Starting container update process" > /dev/tty4 in the code. Are those logs accessible in any way?
2. If not, we can we collect the logs in one file and make that part of the SSH output?

Minor: /home/logs missing on hardware probes

Noticed /home/logs is missing when ssh'ing to the hardware probes logs account (first line after ssh command):

% ssh [email protected]
Could not chdir to home directory /home/logs: No such file or directory
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
Checking for the latest version
Current version 0.10.1
Latest version 0.11.0
Start self-update process
Self-update finished
[2023-01-13 22:32:48] [INFO] [695] [general] Start probe version 0.11.0 in a production mode
[2023-01-13 22:32:50] [DEBUG] [695] [general] connection to API established

7 packets transmitted, 6 received

from every other pc there are no packets dropped

[2023-06-10 13:37:48] [WARN] [696] [status-manager] ping test result don't match criterias:
{
  status: 'finished',
  resolvedAddress: '193.0.14.129',
  resolvedHostname: 'k.root-servers.net',
  timings: [
    { ttl: 57, rtt: 27 },
    { ttl: 57, rtt: 16.7 },
    { ttl: 57, rtt: 15.2 },
    { ttl: 57, rtt: 19.6 },
    { ttl: 57, rtt: 15.3 },
    { ttl: 57, rtt: 16.5 }
  ],
  stats: {
    min: 15.215,
    avg: 18.392,
    max: 26.984,
    total: 7,
    loss: 14.2857,
    rcv: 6,
    drop: 1
  },
  rawOutput: 'PING k.root-servers.net (193.0.14.129) 56(84) bytes of data.\n' +
    '64 bytes from k.root-servers.net (193.0.14.129): icmp_seq=1 ttl=57 time=27.0 ms\n' +
    '64 bytes from k.root-servers.net (193.0.14.129): icmp_seq=3 ttl=57 time=16.7 ms\n' +
    '64 bytes from k.root-servers.net (193.0.14.129): icmp_seq=4 ttl=57 time=15.2 ms\n' +
    '64 bytes from k.root-servers.net (193.0.14.129): icmp_seq=5 ttl=57 time=19.6 ms\n' +
    '64 bytes from k.root-servers.net (193.0.14.129): icmp_seq=6 ttl=57 time=15.3 ms\n' +
    '64 bytes from k.root-servers.net (193.0.14.129): icmp_seq=7 ttl=57 time=16.5 ms\n' +
    '\n' +
    '--- k.root-servers.net ping statistics ---\n' +
    '7 packets transmitted, 6 received, 14.2857% packet loss, time 1213ms\n' +
    'rtt min/avg/max/mdev = 15.215/18.392/26.984/4.101 ms'
}

Probe not working / segfaulting

I wasn't sure where to report this, and I guess here is as good a place as any.

I've had one of the hardware probes since last year. It was working great, but now I don't see it on the Globalping site. When I ssh to logs@, it throws a segfault:

Could not chdir to home directory /home/logs: No such file or directory
unexpected fault address 0x284bf7c
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x2 addr=0x284bf7c pc=0x284bf7c]

goroutine 1 [running, locked to thread]:
runtime.throw(0x1b9c5c5, 0x5)
        /usr/lib/go/src/runtime/panic.go:1117 +0x60 fp=0x329f63c sp=0x329f628 pc=0x816a78
runtime.sigpanic()
        /usr/lib/go/src/runtime/signal_unix.go:741 +0x1d0 fp=0x329f654 sp=0x329f63c pc=0x8300f8
github.com/docker/cli/vendor/k8s.io/api/apps/v1.init.0()
        /home/compiler/globalping-hwprobe/poky/build/tmp/work/cortexa7t2hf-neon-poky-linux-gnueabi/docker-ce/20.10.8-ce+gitd24c6dc5cf5e68dfb30027b2db454099566a9b9e-r0/git/src/import/.gopath/src/github.com/docker/cli/vendor/k8s.io/api/apps/v1/generated.pb.go:835 +0x4ec fp=0x329f658 sp=0x329f658 pc=0x177490c
github.com/docker/cli/vendor/k8s.io/api/apps/v1.init.0()
        /home/compiler/globalping-hwprobe/poky/build/tmp/work/cortexa7t2hf-neon-poky-linux-gnueabi/docker-ce/20.10.8-ce+gitd24c6dc5cf5e68dfb30027b2db454099566a9b9e-r0/git/src/import/.gopath/src/github.com/docker/cli/vendor/k8s.io/api/apps/v1/generated.pb.go:835 +0x4ec fp=0x329f658 sp=0x329f658 pc=0x177490c
runtime.doInit(0x2c83490)
        /usr/lib/go/src/runtime/proc.go:6292 +0x50 fp=0x329f784 sp=0x329f658 pc=0x828fec
runtime.doInit(0x2c88ab0)
        /usr/lib/go/src/runtime/proc.go:6292 +0x50 fp=0x329f8b0 sp=0x329f784 pc=0x828fec
runtime.doInit(0x2c85688)
        /usr/lib/go/src/runtime/proc.go:6292 +0x50 fp=0x329f9dc sp=0x329f8b0 pc=0x828fec
runtime.doInit(0x2c88280)
        /usr/lib/go/src/runtime/proc.go:6292 +0x50 fp=0x329fb08 sp=0x329f9dc pc=0x828fec
runtime.doInit(0x2c89d98)
        /usr/lib/go/src/runtime/proc.go:6292 +0x50 fp=0x329fc34 sp=0x329fb08 pc=0x828fec
runtime.doInit(0x2c85420)
        /usr/lib/go/src/runtime/proc.go:6292 +0x50 fp=0x329fd60 sp=0x329fc34 pc=0x828fec
runtime.doInit(0x2c85af8)
        /usr/lib/go/src/runtime/proc.go:6292 +0x50 fp=0x329fe8c sp=0x329fd60 pc=0x828fec
runtime.doInit(0x2c850b0)
        /usr/lib/go/src/runtime/proc.go:6292 +0x50 fp=0x329ffb8 sp=0x329fe8c pc=0x828fec
runtime.main()
        /usr/lib/go/src/runtime/proc.go:208 +0x278 fp=0x329ffe4 sp=0x329ffb8 pc=0x81978c
runtime.goexit()
        /usr/lib/go/src/runtime/asm_arm.s:841 +0x4 fp=0x329ffe4 sp=0x329ffe4 pc=0x84dcc0

goroutine 21 [chan receive]:
github.com/docker/cli/vendor/k8s.io/klog.(*loggingT).flushDaemon(0x2cef958)
        /home/compiler/globalping-hwprobe/poky/build/tmp/work/cortexa7t2hf-neon-poky-linux-gnueabi/docker-ce/20.10.8-ce+gitd24c6dc5cf5e68dfb30027b2db454099566a9b9e-r0/git/src/import/.gopath/src/github.com/docker/cli/vendor/k8s.io/klog/klog.go:1010 +0x70
created by github.com/docker/cli/vendor/k8s.io/klog.init.0
        /home/compiler/globalping-hwprobe/poky/build/tmp/work/cortexa7t2hf-neon-poky-linux-gnueabi/docker-ce/20.10.8-ce+gitd24c6dc5cf5e68dfb30027b2db454099566a9b9e-r0/git/src/import/.gopath/src/github.com/docker/cli/vendor/k8s.io/klog/klog.go:411 +0x13c
Connection to 10.30.2.2 closed.

root-servers.net

Could not chdir to home directory /home/logs: No such file or directory
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
Checking for the latest version
Current version 0.14.0
Latest version 0.20.0
Start self-update process
Self-update finished
[2023-04-12 22:55:13] [INFO] [695] [general] Start probe version 0.20.0 in a production mode
[2023-04-12 22:55:14] [DEBUG] [695] [general] connection to API established
[2023-04-12 22:55:14] [INFO] [695] [api:connect] connected from (Laakirchen, AT, EU) (lat: 47.9849 long: 13.8183)
[2023-04-12 23:05:17] [WARN] [695] [status-manager] ping test result don't match criterias:
{
status: 'finished',
resolvedAddress: '199.7.83.42',
resolvedHostname: 'l.root-servers.net',
timings: [
{ ttl: 56, rtt: 37.3 },
{ ttl: 56, rtt: 30.9 },
{ ttl: 56, rtt: 31.1 },
{ ttl: 56, rtt: 30.9 },
{ ttl: 56, rtt: 30.1 },
{ ttl: 56, rtt: 34.5 }
],
stats: {
min: 30.106,
avg: 32.471,
max: 37.342,
total: 7,
loss: 14.2857,
rcv: 6,
drop: 1
},
rawOutput: 'PING l.root-servers.net (199.7.83.42) 56(84) bytes of data.\n' +
'64 bytes from l.root-servers.net (199.7.83.42): icmp_seq=1 ttl=56 time=37.3 ms\n' +
'64 bytes from l.root-servers.net (199.7.83.42): icmp_seq=3 ttl=56 time=30.9 ms\n' +
'64 bytes from l.root-servers.net (199.7.83.42): icmp_seq=4 ttl=56 time=31.1 ms\n' +
'64 bytes from l.root-servers.net (199.7.83.42): icmp_seq=5 ttl=56 time=30.9 ms\n' +
'64 bytes from l.root-servers.net (199.7.83.42): icmp_seq=6 ttl=56 time=30.1 ms\n' +
'64 bytes from l.root-servers.net (199.7.83.42): icmp_seq=7 ttl=56 time=34.5 ms\n' +
'\n' +
'--- l.root-servers.net ping statistics ---\n' +
'7 packets transmitted, 6 received, 14.2857% packet loss, time 1206ms\n' +
'rtt min/avg/max/mdev = 30.106/32.471/37.342/2.597 ms'
}
[2023-04-12 23:05:17] [WARN] [695] [status-manager] ping test result don't match criterias:
{
status: 'finished',
resolvedAddress: '193.0.14.129',
resolvedHostname: 'k.root-servers.net',
timings: [
{ ttl: 57, rtt: 18.7 },
{ ttl: 57, rtt: 14.2 },
{ ttl: 57, rtt: 14.4 },
{ ttl: 57, rtt: 14.9 },
{ ttl: 57, rtt: 15.1 },
{ ttl: 57, rtt: 15 }
],
stats: {
min: 14.189,
avg: 15.381,
max: 18.725,
total: 7,
loss: 14.2857,
rcv: 6,
drop: 1
},
rawOutput: 'PING k.root-servers.net (193.0.14.129) 56(84) bytes of data.\n' +
'64 bytes from k.root-servers.net (193.0.14.129): icmp_seq=1 ttl=57 time=18.7 ms\n' +
'64 bytes from k.root-servers.net (193.0.14.129): icmp_seq=3 ttl=57 time=14.2 ms\n' +
'64 bytes from k.root-servers.net (193.0.14.129): icmp_seq=4 ttl=57 time=14.4 ms\n' +
'64 bytes from k.root-servers.net (193.0.14.129): icmp_seq=5 ttl=57 time=14.9 ms\n' +
'64 bytes from k.root-servers.net (193.0.14.129): icmp_seq=6 ttl=57 time=15.1 ms\n' +
'64 bytes from k.root-servers.net (193.0.14.129): icmp_seq=7 ttl=57 time=15.0 ms\n' +
'\n' +
'--- k.root-servers.net ping statistics ---\n' +
'7 packets transmitted, 6 received, 14.2857% packet loss, time 1206ms\n' +
'rtt min/avg/max/mdev = 14.189/15.381/18.725/1.531 ms'
}
[2023-04-13 01:05:37] [WARN] [695] [status-manager] ping test result don't match criterias:
{
status: 'finished',
resolvedAddress: '193.0.14.129',
resolvedHostname: 'k.root-servers.net',
timings: [
{ ttl: 57, rtt: 21.5 },
{ ttl: 57, rtt: 12.8 },
{ ttl: 57, rtt: 12.7 },
{ ttl: 57, rtt: 12.6 },
{ ttl: 57, rtt: 12.8 },
{ ttl: 57, rtt: 12.9 }
],
stats: {
min: 12.609,
avg: 14.196,
max: 21.462,
total: 7,
loss: 14.2857,
rcv: 6,
drop: 1
},
rawOutput: 'PING k.root-servers.net (193.0.14.129) 56(84) bytes of data.\n' +
'64 bytes from k.root-servers.net (193.0.14.129): icmp_seq=1 ttl=57 time=21.5 ms\n' +
'64 bytes from k.root-servers.net (193.0.14.129): icmp_seq=3 ttl=57 time=12.8 ms\n' +
'64 bytes from k.root-servers.net (193.0.14.129): icmp_seq=4 ttl=57 time=12.7 ms\n' +
'64 bytes from k.root-servers.net (193.0.14.129): icmp_seq=5 ttl=57 time=12.6 ms\n' +
'64 bytes from k.root-servers.net (193.0.14.129): icmp_seq=6 ttl=57 time=12.8 ms\n' +
'64 bytes from k.root-servers.net (193.0.14.129): icmp_seq=7 ttl=57 time=12.9 ms\n' +
'\n' +
'--- k.root-servers.net ping statistics ---\n' +
'7 packets transmitted, 6 received, 14.2857% packet loss, time 1207ms\n' +
'rtt min/avg/max/mdev = 12.609/14.196/21.462/3.250 ms'
}
[2023-04-13 02:45:56] [WARN] [695] [status-manager] ping test result don't match criterias:
{
status: 'finished',
resolvedAddress: '192.58.128.30',
resolvedHostname: 'j.root-servers.net',
timings: [
{ ttl: 250, rtt: 15.2 },
{ ttl: 250, rtt: 12.9 },
{ ttl: 250, rtt: 12.4 },
{ ttl: 250, rtt: 12.6 },
{ ttl: 250, rtt: 13.6 },
{ ttl: 250, rtt: 12.4 }
],
stats: {
min: 12.428,
avg: 13.2,
max: 15.184,
total: 7,
loss: 14.2857,
rcv: 6,
drop: 1
},
rawOutput: 'PING j.root-servers.net (192.58.128.30) 56(84) bytes of data.\n' +
'64 bytes from j.root-servers.net (192.58.128.30): icmp_seq=1 ttl=250 time=15.2 ms\n' +
'64 bytes from j.root-servers.net (192.58.128.30): icmp_seq=3 ttl=250 time=12.9 ms\n' +
'64 bytes from j.root-servers.net (192.58.128.30): icmp_seq=4 ttl=250 time=12.4 ms\n' +
'64 bytes from j.root-servers.net (192.58.128.30): icmp_seq=5 ttl=250 time=12.6 ms\n' +
'64 bytes from j.root-servers.net (192.58.128.30): icmp_seq=6 ttl=250 time=13.6 ms\n' +
'64 bytes from j.root-servers.net (192.58.128.30): icmp_seq=7 ttl=250 time=12.4 ms\n' +
'\n' +
'--- j.root-servers.net ping statistics ---\n' +
'7 packets transmitted, 6 received, 14.2857% packet loss, time 1209ms\n' +
'rtt min/avg/max/mdev = 12.428/13.200/15.184/0.970 ms'
}
[2023-04-13 05:06:15] [WARN] [695] [status-manager] ping test result don't match criterias:
{
status: 'finished',
resolvedAddress: '193.0.14.129',
resolvedHostname: 'k.root-servers.net',
timings: [
{ ttl: 57, rtt: 23.6 },
{ ttl: 57, rtt: 12.9 },
{ ttl: 57, rtt: 12.6 },
{ ttl: 57, rtt: 12.8 },
{ ttl: 57, rtt: 12.8 },
{ ttl: 57, rtt: 12.5 }
],
stats: {
min: 12.542,
avg: 14.534,
max: 23.597,
total: 7,
loss: 14.2857,
rcv: 6,
drop: 1
},
rawOutput: 'PING k.root-servers.net (193.0.14.129) 56(84) bytes of data.\n' +
'64 bytes from k.root-servers.net (193.0.14.129): icmp_seq=1 ttl=57 time=23.6 ms\n' +
'64 bytes from k.root-servers.net (193.0.14.129): icmp_seq=3 ttl=57 time=12.9 ms\n' +
'64 bytes from k.root-servers.net (193.0.14.129): icmp_seq=4 ttl=57 time=12.6 ms\n' +
'64 bytes from k.root-servers.net (193.0.14.129): icmp_seq=5 ttl=57 time=12.8 ms\n' +
'64 bytes from k.root-servers.net (193.0.14.129): icmp_seq=6 ttl=57 time=12.8 ms\n' +
'64 bytes from k.root-servers.net (193.0.14.129): icmp_seq=7 ttl=57 time=12.5 ms\n' +
'\n' +
'--- k.root-servers.net ping statistics ---\n' +
'7 packets transmitted, 6 received, 14.2857% packet loss, time 1210ms\n' +
'rtt min/avg/max/mdev = 12.542/14.534/23.597/4.055 ms'
}
[2023-04-13 07:06:35] [WARN] [695] [status-manager] ping test result don't match criterias:
{
status: 'finished',
resolvedAddress: '192.58.128.30',
resolvedHostname: 'j.root-servers.net',
timings: [
{ ttl: 250, rtt: 22.7 },
{ ttl: 250, rtt: 12.1 },
{ ttl: 250, rtt: 12.1 },
{ ttl: 250, rtt: 13.1 },
{ ttl: 250, rtt: 12.7 },
{ ttl: 250, rtt: 16.2 }
],
stats: {
min: 12.078,
avg: 14.81,
max: 22.731,
total: 7,
loss: 14.2857,
rcv: 6,
drop: 1
},
rawOutput: 'PING j.root-servers.net (192.58.128.30) 56(84) bytes of data.\n' +
'64 bytes from j.root-servers.net (192.58.128.30): icmp_seq=1 ttl=250 time=22.7 ms\n' +
'64 bytes from j.root-servers.net (192.58.128.30): icmp_seq=3 ttl=250 time=12.1 ms\n' +
'64 bytes from j.root-servers.net (192.58.128.30): icmp_seq=4 ttl=250 time=12.1 ms\n' +
'64 bytes from j.root-servers.net (192.58.128.30): icmp_seq=5 ttl=250 time=13.1 ms\n' +
'64 bytes from j.root-servers.net (192.58.128.30): icmp_seq=6 ttl=250 time=12.7 ms\n' +
'64 bytes from j.root-servers.net (192.58.128.30): icmp_seq=7 ttl=250 time=16.2 ms\n' +
'\n' +
'--- j.root-servers.net ping statistics ---\n' +
'7 packets transmitted, 6 received, 14.2857% packet loss, time 1217ms\n' +
'rtt min/avg/max/mdev = 12.078/14.810/22.731/3.803 ms'
}
[2023-04-13 10:47:07] [WARN] [695] [status-manager] ping test result don't match criterias:
{
status: 'finished',
resolvedAddress: '193.0.14.129',
resolvedHostname: 'k.root-servers.net',
timings: [
{ ttl: 57, rtt: 23.4 },
{ ttl: 57, rtt: 13.1 },
{ ttl: 57, rtt: 13.6 },
{ ttl: 57, rtt: 13.5 },
{ ttl: 57, rtt: 12 },
{ ttl: 57, rtt: 13.5 }
],
stats: {
min: 12.024,
avg: 14.863,
max: 23.421,
total: 7,
loss: 14.2857,
rcv: 6,
drop: 1
},
rawOutput: 'PING k.root-servers.net (193.0.14.129) 56(84) bytes of data.\n' +
'64 bytes from k.root-servers.net (193.0.14.129): icmp_seq=1 ttl=57 time=23.4 ms\n' +
'64 bytes from k.root-servers.net (193.0.14.129): icmp_seq=3 ttl=57 time=13.1 ms\n' +
'64 bytes from k.root-servers.net (193.0.14.129): icmp_seq=4 ttl=57 time=13.6 ms\n' +
'64 bytes from k.root-servers.net (193.0.14.129): icmp_seq=5 ttl=57 time=13.5 ms\n' +
'64 bytes from k.root-servers.net (193.0.14.129): icmp_seq=6 ttl=57 time=12.0 ms\n' +
'64 bytes from k.root-servers.net (193.0.14.129): icmp_seq=7 ttl=57 time=13.5 ms\n' +
'\n' +
'--- k.root-servers.net ping statistics ---\n' +
'7 packets transmitted, 6 received, 14.2857% packet loss, time 1207ms\n' +
'rtt min/avg/max/mdev = 12.024/14.863/23.421/3.864 ms'
}
[2023-04-13 11:17:12] [WARN] [695] [status-manager] ping test result don't match criterias:
{
status: 'finished',
resolvedAddress: '192.58.128.30',
resolvedHostname: 'j.root-servers.net',
timings: [
{ ttl: 250, rtt: 21.5 },
{ ttl: 250, rtt: 16.7 },
{ ttl: 250, rtt: 18.5 },
{ ttl: 250, rtt: 17.2 },
{ ttl: 250, rtt: 16.9 },
{ ttl: 250, rtt: 12.1 }
],
stats: {
min: 12.142,
avg: 17.157,
max: 21.466,
total: 7,
loss: 14.2857,
rcv: 6,
drop: 1
},
rawOutput: 'PING j.root-servers.net (192.58.128.30) 56(84) bytes of data.\n' +
'64 bytes from j.root-servers.net (192.58.128.30): icmp_seq=1 ttl=250 time=21.5 ms\n' +
'64 bytes from j.root-servers.net (192.58.128.30): icmp_seq=3 ttl=250 time=16.7 ms\n' +
'64 bytes from j.root-servers.net (192.58.128.30): icmp_seq=4 ttl=250 time=18.5 ms\n' +
'64 bytes from j.root-servers.net (192.58.128.30): icmp_seq=5 ttl=250 time=17.2 ms\n' +
'64 bytes from j.root-servers.net (192.58.128.30): icmp_seq=6 ttl=250 time=16.9 ms\n' +
'64 bytes from j.root-servers.net (192.58.128.30): icmp_seq=7 ttl=250 time=12.1 ms\n' +
'\n' +
'--- j.root-servers.net ping statistics ---\n' +
'7 packets transmitted, 6 received, 14.2857% packet loss, time 1213ms\n' +
'rtt min/avg/max/mdev = 12.142/17.157/21.466/2.763 ms'
}
[2023-04-13 11:47:16] [WARN] [695] [status-manager] ping test result don't match criterias:
{
status: 'finished',
resolvedAddress: '192.58.128.30',
resolvedHostname: 'j.root-servers.net',
timings: [
{ ttl: 250, rtt: 16.9 },
{ ttl: 250, rtt: 15.7 },
{ ttl: 250, rtt: 15 },
{ ttl: 250, rtt: 12.8 },
{ ttl: 250, rtt: 13.6 },
{ ttl: 250, rtt: 12.8 }
],
stats: {
min: 12.75,
avg: 14.46,
max: 16.874,
total: 7,
loss: 14.2857,
rcv: 6,
drop: 1
},
rawOutput: 'PING j.root-servers.net (192.58.128.30) 56(84) bytes of data.\n' +
'64 bytes from j.root-servers.net (192.58.128.30): icmp_seq=1 ttl=250 time=16.9 ms\n' +
'64 bytes from j.root-servers.net (192.58.128.30): icmp_seq=3 ttl=250 time=15.7 ms\n' +
'64 bytes from j.root-servers.net (192.58.128.30): icmp_seq=4 ttl=250 time=15.0 ms\n' +
'64 bytes from j.root-servers.net (192.58.128.30): icmp_seq=5 ttl=250 time=12.8 ms\n' +
'64 bytes from j.root-servers.net (192.58.128.30): icmp_seq=6 ttl=250 time=13.6 ms\n' +
'64 bytes from j.root-servers.net (192.58.128.30): icmp_seq=7 ttl=250 time=12.8 ms\n' +
'\n' +
'--- j.root-servers.net ping statistics ---\n' +
'7 packets transmitted, 6 received, 14.2857% packet loss, time 1219ms\n' +
'rtt min/avg/max/mdev = 12.750/14.460/16.874/1.528 ms'
}
[2023-04-13 14:37:42] [WARN] [695] [status-manager] ping test result don't match criterias:
{
status: 'finished',
resolvedAddress: '199.7.83.42',
resolvedHostname: 'l.root-servers.net',
timings: [
{ ttl: 56, rtt: 34.2 },
{ ttl: 56, rtt: 33.8 },
{ ttl: 56, rtt: 29.2 },
{ ttl: 56, rtt: 30.1 },
{ ttl: 56, rtt: 29.1 },
{ ttl: 56, rtt: 32.6 }
],
stats: {
min: 29.116,
avg: 31.509,
max: 34.218,
total: 7,
loss: 14.2857,
rcv: 6,
drop: 1
},
rawOutput: 'PING l.root-servers.net (199.7.83.42) 56(84) bytes of data.\n' +
'64 bytes from l.root-servers.net (199.7.83.42): icmp_seq=1 ttl=56 time=34.2 ms\n' +
'64 bytes from l.root-servers.net (199.7.83.42): icmp_seq=3 ttl=56 time=33.8 ms\n' +
'64 bytes from l.root-servers.net (199.7.83.42): icmp_seq=4 ttl=56 time=29.2 ms\n' +
'64 bytes from l.root-servers.net (199.7.83.42): icmp_seq=5 ttl=56 time=30.1 ms\n' +
'64 bytes from l.root-servers.net (199.7.83.42): icmp_seq=6 ttl=56 time=29.1 ms\n' +
'64 bytes from l.root-servers.net (199.7.83.42): icmp_seq=7 ttl=56 time=32.6 ms\n' +
'\n' +
'--- l.root-servers.net ping statistics ---\n' +
'7 packets transmitted, 6 received, 14.2857% packet loss, time 1213ms\n' +
'rtt min/avg/max/mdev = 29.116/31.509/34.218/2.114 ms'
}
[2023-04-13 15:07:46] [WARN] [695] [status-manager] ping test result don't match criterias:
{
status: 'finished',
resolvedAddress: '199.7.83.42',
resolvedHostname: 'l.root-servers.net',
timings: [
{ ttl: 56, rtt: 35.7 },
{ ttl: 56, rtt: 29.2 },
{ ttl: 56, rtt: 32.2 },
{ ttl: 56, rtt: 28.9 },
{ ttl: 56, rtt: 29.1 },
{ ttl: 56, rtt: 30 }
],
stats: {
min: 28.869,
avg: 30.847,
max: 35.724,
total: 7,
loss: 14.2857,
rcv: 6,
drop: 1
},
rawOutput: 'PING l.root-servers.net (199.7.83.42) 56(84) bytes of data.\n' +
'64 bytes from l.root-servers.net (199.7.83.42): icmp_seq=1 ttl=56 time=35.7 ms\n' +
'64 bytes from l.root-servers.net (199.7.83.42): icmp_seq=3 ttl=56 time=29.2 ms\n' +
'64 bytes from l.root-servers.net (199.7.83.42): icmp_seq=4 ttl=56 time=32.2 ms\n' +
'64 bytes from l.root-servers.net (199.7.83.42): icmp_seq=5 ttl=56 time=28.9 ms\n' +
'64 bytes from l.root-servers.net (199.7.83.42): icmp_seq=6 ttl=56 time=29.1 ms\n' +
'64 bytes from l.root-servers.net (199.7.83.42): icmp_seq=7 ttl=56 time=30.0 ms\n' +
'\n' +
'--- l.root-servers.net ping statistics ---\n' +
'7 packets transmitted, 6 received, 14.2857% packet loss, time 1217ms\n' +
'rtt min/avg/max/mdev = 28.869/30.847/35.724/2.445 ms'
}
[2023-04-13 16:38:00] [WARN] [695] [status-manager] ping test result don't match criterias:
{
status: 'finished',
resolvedAddress: '193.0.14.129',
resolvedHostname: 'k.root-servers.net',
timings: [
{ ttl: 57, rtt: 22.5 },
{ ttl: 57, rtt: 13.5 },
{ ttl: 57, rtt: 13 },
{ ttl: 57, rtt: 12.3 },
{ ttl: 57, rtt: 17 },
{ ttl: 57, rtt: 17.1 }
],
stats: {
min: 12.256,
avg: 15.876,
max: 22.486,
total: 7,
loss: 14.2857,
rcv: 6,
drop: 1
},
rawOutput: 'PING k.root-servers.net (193.0.14.129) 56(84) bytes of data.\n' +
'64 bytes from k.root-servers.net (193.0.14.129): icmp_seq=1 ttl=57 time=22.5 ms\n' +
'64 bytes from k.root-servers.net (193.0.14.129): icmp_seq=3 ttl=57 time=13.5 ms\n' +
'64 bytes from k.root-servers.net (193.0.14.129): icmp_seq=4 ttl=57 time=13.0 ms\n' +
'64 bytes from k.root-servers.net (193.0.14.129): icmp_seq=5 ttl=57 time=12.3 ms\n' +
'64 bytes from k.root-servers.net (193.0.14.129): icmp_seq=6 ttl=57 time=17.0 ms\n' +
'64 bytes from k.root-servers.net (193.0.14.129): icmp_seq=7 ttl=57 time=17.1 ms\n' +
'\n' +
'--- k.root-servers.net ping statistics ---\n' +
'7 packets transmitted, 6 received, 14.2857% packet loss, time 1209ms\n' +
'rtt min/avg/max/mdev = 12.256/15.876/22.486/3.503 ms'
}

v2 final testing

After everything is merged and no other features are planned need to make sure the final firmware is stable:

1. Load it on multiple devices. From old batch and new batch for a few weeks.
2. Build a firmware with 0.28 container and test the container update process
3. Build a firmware with 0.29 container and test the container update process (after 0.30 is released)
4. Monitor closely and check logs to ensure its not crashing or has update loops
5. Ensure the weekly reboots don't break anything
6. Ensure the sd card is protected from abuse
7. Ensure RAM is enough to do all of the above, even if the container increases in size
8. Measure time to start the container
9. other?
10. Start shipping

SSH Fingerprint changes on reboot

Hello! I had this problem, where every time I restart the probe, the SSH fingerprint changes, so I have to manually remove the old line from known_hosts to be able to connect.

also I cant select labels on issues

errors after reboot

This is the output from the logs:

      _closeFrameSent: false,
      _closeMessage: <Buffer >,
      _closeTimer: null,
      _extensions: {},
      _protocol: '',
      _readyState: 2,
      _receiver: null,
      _sender: null,
      _socket: null,
      _bufferedAmount: 0,
      _isServer: false,
      _redirects: 0,
      _url: 'ws://api.globalping.io/socket.io/?version=0.14.0&EIO=4&transport=websocket',
      _req: null,
      [Symbol(kCapture)]: false
    },
    [Symbol(kType)]: 'error',
    [Symbol(kError)]: Error: getaddrinfo ENOTFOUND api.globalping.io
        at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:107:26) {
      errno: -3007,
      code: 'ENOTFOUND',
      syscall: 'getaddrinfo',
      hostname: 'api.globalping.io'
    },
    [Symbol(kMessage)]: 'getaddrinfo ENOTFOUND api.globalping.io'
  },
  context: undefined,
  type: 'TransportError'
}
Error: websocket error
    at WS.onError (file:///app/node_modules/engine.io-client/build/esm-debug/transport.js:40:37)
    at ws.onerror (file:///app/node_modules/engine.io-client/build/esm-debug/transports/websocket.js:82:37)
    at WebSocket.onError (/app/node_modules/ws/lib/event-target.js:220:18)
    at WebSocket.emit (node:events:513:28)
    at ClientRequest.<anonymous> (/app/node_modules/ws/lib/websocket.js:728:15)
    at ClientRequest.emit (node:events:513:28)
    at Socket.socketErrorListener (node:_http_client:496:9)
    at Socket.emit (node:events:513:28)
    at emitErrorNT (node:internal/streams/destroy:151:8)
    at emitErrorCloseNT (node:internal/streams/destroy:116:3)
[2021-11-20 19:39:29] [ERROR] [687] [general] connection to API failed: websocket error
{
  description: ErrorEvent {
    [Symbol(kTarget)]: WebSocket {
      _events: [Object: null prototype],
      _eventsCount: 4,
      _maxListeners: undefined,
      _binaryType: 'nodebuffer',
      _closeCode: 1006,
      _closeFrameReceived: false,
      _closeFrameSent: false,
      _closeMessage: <Buffer >,
      _closeTimer: null,
      _extensions: {},
      _protocol: '',
      _readyState: 2,
      _receiver: null,
      _sender: null,
      _socket: null,
      _bufferedAmount: 0,
      _isServer: false,
      _redirects: 0,
      _url: 'ws://api.globalping.io/socket.io/?version=0.14.0&EIO=4&transport=websocket',
      _req: null,
      [Symbol(kCapture)]: false
    },
    [Symbol(kType)]: 'error',
    [Symbol(kError)]: Error: getaddrinfo ENOTFOUND api.globalping.io
        at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:107:26) {
      errno: -3007,
      code: 'ENOTFOUND',
      syscall: 'getaddrinfo',
      hostname: 'api.globalping.io'
    },
    [Symbol(kMessage)]: 'getaddrinfo ENOTFOUND api.globalping.io'
  },
  context: undefined,
  type: 'TransportError'
}
Error: websocket error
    at WS.onError (file:///app/node_modules/engine.io-client/build/esm-debug/transport.js:40:37)
    at ws.onerror (file:///app/node_modules/engine.io-client/build/esm-debug/transports/websocket.js:82:37)
    at WebSocket.onError (/app/node_modules/ws/lib/event-target.js:220:18)
    at WebSocket.emit (node:events:513:28)
    at ClientRequest.<anonymous> (/app/node_modules/ws/lib/websocket.js:728:15)
    at ClientRequest.emit (node:events:513:28)
    at Socket.socketErrorListener (node:_http_client:496:9)
    at Socket.emit (node:events:513:28)
    at emitErrorNT (node:internal/streams/destroy:151:8)
    at emitErrorCloseNT (node:internal/streams/destroy:116:3)

Location not correct

[2023-02-03 17:46:58] [INFO] [688] [api:connect] connected from (Laakirchen, AT, EU) (lat: 47.9819 long: 13.8217)

Its not the correct location.
Its the nearest datacenter from my ISP (im using the hardware-probe from my home location)
Is there any way to change it to my exact home town?

Dev build flag

I currently had to build a custom FW to try out a new release of the probe software without releasing it to all users (changed the source for docker image to my repo). I'm wondering if we could have support for this in the official FW instead. E.g., if JSDELIVR-DEV.UPD file is present, pull ghcr.io/jsdelivr/globalping-probe:dev instead of ghcr.io/jsdelivr/globalping-probe:latest - security-wise, this should be fine, since it would still use the official repo, just a different tag.

beta branch

Is there any beta branch to test?
I have 2 devices here so i could install one with the beta and one with the release version

ns1.registry.in 14.2857% packet loss

Ive tested it with a new pc and setup only hwprobe docker.
DNS same as all other devices and ive tested it with different dns (1.1.1.1 and 8.8.8.8).
New pc is working flawlessy and no dns package losses.
Probe still produces package losses:
[2023-08-20 09:14:19] [WARN] [status-manager] Quality control ping test result is unsuccessful: ns1.registry.in 14.2857% packet loss. Test pass.
[2023-08-20 09:34:22] [WARN] [status-manager] Quality control ping test result is unsuccessful: ns1.registry.in 14.2857% packet loss. Test pass.

          Its not something I can replicate on my probes, they pass all tests 100% of the time. So I dont think its a hardware issue.

If you want you could run a docker container probe on your PC and compare the results.

But its not even a failure, the log you sent is just a warning, you can see at the end that the probe was disconnected Test pass.

Originally posted by @jimaek in #23 (comment)

Update container

We need the firmware to update the container without user interaction as it happens now. We need to find the most optimal way to do so.
Something to consider:

• People dont like slow boots. So pulling the container on boot is probably not the best idea.
• At some point the old container will stop working completely
• SD Cards suck and we need to protect them
• The device can lose power at any moment. This should never lead to corruption.

Can the firmware cache the docker container on card and then update it on a schedule? e.g.

1. We compile the firmware with probe version 0.25.0, the device boots and starts the container.
2. The old pre-baked container will update the code inside itself and start the probe. The container at this point is based on 0.25.0 but runs latest code v0.35.0
3. The probe is online and works
4. A parallel cron/schedule runs docker pull ghcr.io/jsdelivr/globalping-probe once per day and permanently writes it to the SD card. Only if there are changes.
5. Scheduled reboot happens after 5 days.
6. The firmware now boots the cached container from last pull, and not the one it came with. (the original container should probably be replaced completely. We dont need to store multiple versions of it. BUT need to handle cases when the probe reboots during container update)

restarts and connection losses

Hi,
i got my globalping device yesterday and installed it today.
Sometimes i can login with "ssh [email protected]
After some time im getting this error message:
Could not chdir to home directory /home/logs: No such file or directory
Error: No such container: globalping-probe

Also not shown on https://api.globalping.io/demo/

It needs some minutes to let me login again and shows up on the api website.

This is the "normal" output when its shown on https://api.globalping.io/demo/
Could not chdir to home directory /home/logs: No such file or directory
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
Checking for the latest version
Current version 0.10.1
Latest version 0.11.0
Start self-update process
Self-update finished
[2023-01-25 23:17:25] [INFO] [695] [general] Start probe version 0.11.0 in a production mode
[2023-01-25 23:17:26] [DEBUG] [695] [general] connection to API established

Update node.js on the existing probes

Since the existing probes don't have an automated container update mechanism (#28) and we have a similar issue with people who run the probes in VMs and don't update their docker containers, I've been exploring the option of adding an update mechanism for node.js directly into the docker container. The reason is that the node.js version used on the existing probes already reached EOL, which also prevents us from updating npm packages in our code base (since those usually stop supporting EOL versions quickly).

I have a repository hosting a modified version of the probe software, which includes this feature, with the main part being this file. It's based on a nvm script, which handles the whole process of downloading and verification.

I've also built a custom version of the FW to test this out on the HW probes. Unfortunately, while it works reliably in docker on my PC, it fails most of the time on the HW probe. I'm not 100% sure why since the failure results in a restart and loss of all logs, but my guess is that there is not enough RAM to handle the downloaded file combined with the update itself. Occasionally, though, it works - it just takes many reboots to succeed. The problem is it can often take 1 hour or more of retries, and since the update is in-memory only, it needs to happen again if the probe is shut down.

@kernelgurumeditation it would be great if you could take a look at this, verify the real cause of failures, and see if we can make any improvements/changes that would make the update work reliably (it is fine if it takes, let's say, 5 minutes after each startup). However, keep in mind that the goal is to make this work on the existing probes without user interaction, so we're limited to making changes to the update process itself; no FW changes will help here.

Prevent SSH key changes

Every time the probe resets, its SSH key changes, which is super annoying when connecting to the logs account. Can we store the generated key on the SD card to avoid this?

Support new OCI images

#29 (comment)

Version in the new format that should work for testing: globalping/globalping-probe:0.29.0

Probes hostname not resolving

Previously the probe was able to resolve it's name in the LAN to something like globalping-probe-XXXX. But now its just an IP.
Can you please check it works for you @MartinKolarik, maybe its something with my network

https://share.perfstack.net/2024/06/advanced_ip_scanner_2024-06-13_14-58-50.png