Giter Site home page Giter Site logo

netprobe_lite's Introduction

Netprobe

Simple and effective tool for measuring ISP performance at home. The tool measures several performance metrics including packet loss, latency, jitter, and DNS performance. It also has an optional speed test to measure bandwidth. Netprobe aggregates these metrics into a common score, which you can use to monitor overall health of your internet connection.

Support the Project

If you'd like to support the development of this project, feel free to buy me a coffee!

https://buymeacoffee.com/plaintextpm

Full Tutorial

Visit YouTube for a full tutorial on how to install and use Netprobe:

https://youtu.be/Wn31husi6tc

Requirements and Setup

To run Netprobe, you'll need a PC running Docker connected directly to your ISP router. Specifically:

  1. Netprobe requires the latest version of Docker. For instructions on installing Docker, see YouTube, it's super easy.

  2. Netprobe should be installed on a machine (the 'probe') which has a wired Ethernet connection to your primary ISP router. This ensures the tests are accurately measuring your ISP performance and excluding and interference from your home network. An old PC with Linux installed is a great option for this.

Installation

First-time Install

  1. Clone the repo locally to the probe machine:
git clone https://github.com/plaintextpackets/netprobe_lite.git
  1. From the cloned folder, use docker compose to launch the app:
docker compose up
  1. To shut down the app, use docker compose again:
docker compose down

Upgrading Between Versions

When upgrading between versions, it is best to delete the deployment altogether and restart with the new code. The process is described below.

  1. Stop Netprobe in Docker and use the -v flag to delete all volumes (warning this deletes old data):
docker compose down -v
  1. Clone the latest code (or download manually from Github and replace the current files):
git clone https://github.com/plaintextpackets/netprobe_lite.git
  1. Re-start Netprobe:
docker compose up

How to use

  1. Navigate to: http://x.x.x.x:3001/d/app/netprobe where x.x.x.x = IP of the probe machine running Docker.

  2. Default user / pass is 'admin/admin'. Login to Grafana and set a custom password.

How to customize

Enable Speedtest

By default the speed test feature is disabled as many users pay for bandwidth usage (e.g. cellular connections). To enable it, edit the .env file to set the option to 'True':

SPEEDTEST_ENABLED="True"

Note: speedtest.net has a limit on how frequently you can connection and run the test. If you set the test to run too frequently, you will receive errors. Recommend leaving the 'SPEEEDTEST_INTERVAL' unchanged.

Change Netprobe port

To change the port that Netprobe Lite is running on, edit the 'compose.yml' file, under the 'grafana' section:

ports:
    - '3001:3000'

Change the port on the left to the port you want to access Netprobe Lite on

Customize DNS test

If the DNS server your network uses is not already monitored, you can add your DNS server IP for testing.

To do so, modify this line in .env:

DNS_NAMESERVER_4_IP="8.8.8.8" # Replace this IP with the DNS server you use at home

Change 8.8.8.8 to the IP of the DNS server you use, then restart the application (docker compose down / docker compose up)

Use external Grafana

Some users have their own Grafana instance running and would like to ingest Netprobe statistics there rather than running Grafana in Docker. To do this:

  1. In the compose.yaml file, add a port mapping to the Prometheus deployment config:
  prometheus:
    ...
    ports:
      - 'XXXX:9090'    

... where XXXX is the port you wish to expose Prometheus on your host machine

  1. Remove all of the Grafana configuration from the compose.yaml file

  2. Run Netprobe and then add a datasource to your existing Grafana as http://x.x.x.x:XXXX where x.x.x.x = IP of the probe machine running Docker

Data storage - default method

By default, Docker will store the data collected in several Docker volumes, which will persist between restarts.

They are:

netprobe_grafana_data (used to store Grafana user / pw)
netprobe_prometheus_data (used to store time series data)

To clear out old data, you need to stop the app and remove these volumes:

docker compose down
docker volume rm netprobe_grafana_data
docker volume rm netprobe_prometheus_data

When started again the old data should be wiped out.

Data storage - bind mount method

Using the default method, the data is stored within Docker volumes which you cannot easily access from the host itself. If you'd prefer storing data in mapped folders from the host, follow these instructions (thank you @Jeppedy):

  1. Clone the repo

  2. Inside the folder create two directories:

mkdir -p data/grafana data/prometheus 
  1. Modify the compose.yml as follows (volume path as well as adding user ID):
  prometheus:
    restart: always
    container_name: netprobe-prometheus
    image: "prom/prometheus"
    volumes:
      - ./config/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - ./data/prometheus:/prometheus # modify this to map to the folder you created

    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
    networks:
      - custom_network  # Attach to the custom network
    user: "1000" # set this to the desired user with correct permissions to the bind mount

  grafana:
    restart: always
    image: grafana/grafana-enterprise
    container_name: netprobe-grafana
    volumes:
      - ./config/grafana/datasources/automatic.yml:/etc/grafana/provisioning/datasources/automatic.yml
      - ./config/grafana/dashboards/main.yml:/etc/grafana/provisioning/dashboards/main.yml
      - ./config/grafana/dashboards/netprobe.json:/var/lib/grafana/dashboards/netprobe.json
      - ./data/grafana:/var/lib/grafana  # modify this to map to the folder you created
    ports:
      - '3001:3000'
    networks:
      - custom_network  # Attach to the custom network
    user: "1000" # set this to the desired user with correct permissions to the bind mount
  1. Remove the volumes section from compose.yml

Run on startup

Netprobe will automatically restart itself after the host system is rebooted, provided that Docker is also launched on startup. If you want to disable this behavior, modify the 'restart' variables in the compose.yaml file to this:

restart: never

Wipe all stored data

To wipe all stored data and remove the Docker volumes, use this command:

docker compose down -v

This will delete all containers and volumes related to Netprobe.

FAQ & Troubleshooting

Q. How do I reset my Grafana password?

A. Delete the docker volume for grafana. This will reset your password but will leave your data:

docker volume rm netprobe_grafana_data

Q. I am running Pihole and when I enter my host IP under 'DNS_NAMESERVER_4_IP=' I receive this error:

The resolution lifetime expired after 5.138 seconds: Server Do53:192.168.0.91@53 answered got a response from ('172.21.0.1', 53) instead of ('192.168.0.91', 53)

A. This is a limitation of Docker. If you are running another DNS server in Docker and want to test it in Netprobe, you need to specify the Docker network gateway IP:

  1. Stop netprobe but don't wipe it (docker compose down)
  2. Find the gateway IP of your netprobe-probe container:
$ docker inspect netprobe-probe | grep Gateway
            "Gateway": "",
            "IPv6Gateway": "",
                    "Gateway": "192.168.208.1",
                    "IPv6Gateway": "", 
  1. Enter that IP (e.g. 182.168.208.1) into your .env file for 'DNS_NAMESERVER_4_IP='

Q. I constantly see one of my DNS servers at 5s latency, is this normal?

A. 5s is the timeout for DNS queries in Netprobe Lite. If you see this happening for one specific IP, likely your machine is having issues using that DNS server (and so you shouldn't use it for home use).

License

This project is released under a custom license that restricts commercial use. You are free to use, modify, and distribute the software for non-commercial purposes. Commercial use of this software is strictly prohibited without prior permission. If you have any questions or wish to use this software commercially, please contact [[email protected]].

netprobe_lite's People

Contributors

nicolascarpi avatar nordmannphilip avatar plaintextpackets avatar securedpackets avatar synerdjin avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

netprobe_lite's Issues

Docker Image not for ARM

Since you switch to a docker image the image did not work on raspberry any more:
netprobe The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm/v7) and no specific platform was requested …-

DNS latency variable error

Hi,
When I run the docker container I get the following error that keeps displaying with a few seconds intervals.
Grafana is also not displaying any data

netprobe-presentation | Traceback (most recent call last): netprobe-presentation | File "/usr/lib/python3.12/wsgiref/handlers.py", line 137, in run netprobe-presentation | self.result = application(self.environ, self.start_response) netprobe-presentation | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ netprobe-presentation | File "/usr/local/lib/python3.12/dist-packages/prometheus_client/exposition.py", line 128, in prometheus_app netprobe-presentation | status, headers, output = _bake_output(registry, accept_header, accept_encoding_header, params, disable_compression) netprobe-presentation | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ netprobe-presentation | File "/usr/local/lib/python3.12/dist-packages/prometheus_client/exposition.py", line 104, in _bake_output netprobe-presentation | output = encoder(registry) netprobe-presentation | ^^^^^^^^^^^^^^^^^ netprobe-presentation | File "/usr/local/lib/python3.12/dist-packages/prometheus_client/openmetrics/exposition.py", line 21, in generate_latest netprobe-presentation | for metric in registry.collect(): netprobe-presentation | File "/usr/local/lib/python3.12/dist-packages/prometheus_client/registry.py", line 97, in collect netprobe-presentation | yield from collector.collect() netprobe-presentation | File "/netprobe_lite/presentation.py", line 101, in collect netprobe-presentation | if my_dns_latency / threshold_dns_latency >= 1: netprobe-presentation | ^^^^^^^^^^^^^^ netprobe-presentation | UnboundLocalError: cannot access local variable 'my_dns_latency' where it is not associated with a value

unbound variable my_dns_latency

I just tried to run the docker compose and I saw this in the log after a short while:

netprobe-probe         | Error performing DNS resolution on ('My_DNS_Server', '8.8.8.8')Error performing DNS resolution on ('Google_DNS', '8.8.8.8')
netprobe-probe         | The resolution lifetime expired after 5.106 seconds: Server Do53:8.8.8.8@53 answered The DNS operation timed out.
netprobe-probe         | Error performing DNS resolution on ('CloudFlare_DNS', '1.1.1.1')
netprobe-probe         | The resolution lifetime expired after 5.103 seconds: Server Do53:8.8.8.8@53 answered The DNS operation timed out.
netprobe-probe         | 
netprobe-probe         | The resolution lifetime expired after 5.106 seconds: Server Do53:1.1.1.1@53 answered The DNS operation timed out.
netprobe-probe         | Error performing DNS resolution on ('Quad9_DNS', '9.9.9.9')
netprobe-probe         | The resolution lifetime expired after 5.106 seconds: Server Do53:9.9.9.9@53 answered The DNS operation timed out.
netprobe-presentation  | Traceback (most recent call last):
netprobe-presentation  |   File "/usr/lib/python3.12/wsgiref/handlers.py", line 137, in run
netprobe-presentation  |     self.result = application(self.environ, self.start_response)
netprobe-presentation  |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
netprobe-presentation  |   File "/usr/local/lib/python3.12/dist-packages/prometheus_client/exposition.py", line 128, in prometheus_app
netprobe-presentation  |     status, headers, output = _bake_output(registry, accept_header, accept_encoding_header, params, disable_compression)
netprobe-presentation  |                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
netprobe-presentation  |   File "/usr/local/lib/python3.12/dist-packages/prometheus_client/exposition.py", line 104, in _bake_output
netprobe-presentation  |     output = encoder(registry)
netprobe-presentation  |              ^^^^^^^^^^^^^^^^^
netprobe-presentation  |   File "/usr/local/lib/python3.12/dist-packages/prometheus_client/openmetrics/exposition.py", line 21, in generate_latest
netprobe-presentation  |     for metric in registry.collect():
netprobe-presentation  |   File "/usr/local/lib/python3.12/dist-packages/prometheus_client/registry.py", line 97, in collect
netprobe-presentation  |     yield from collector.collect()
netprobe-presentation  |   File "/netprobe_lite/presentation.py", line 106, in collect
netprobe-presentation  |     if my_dns_latency / threshold_dns_latency >= 1:
netprobe-presentation  |        ^^^^^^^^^^^^^^
netprobe-presentation  | UnboundLocalError: cannot access local variable 'my_dns_latency' where it is not associated with a value

It seems no DNS are reachable from the docker, and so the variable my_dns_latency is not properly set and raises an unbound error. When I change the default local DNS IP, it works as expected (except that the others DNS are stil not reachable).

Failed to start the docker container

Hello,

I tried to clone the repo and start the container.

✘ Network netprobe_lite_custom_network  Error                                                                                                                                                               0.0s
failed to create network netprobe_lite_custom_network: Error response from daemon: Pool overlaps with other one on this address space

Not sure what I could change on the container network to avoid this.

PiHoleDNS in docker causing issues with DNS response times

I'm trying to use this tool, and also have a custom PiHole DNS server running on my local network in another docker container. My machines all access this dns server on the hosts IP (192.168.0.91) which is also the same machine I'm trying to run this on, however when I set that as my DNS server in DNS_NAMESERVER_4_IP I get a consistent timeout with a response from a docker IP (although interestingly not the IP that portainer lists as pihole's IP. Is there someway this setup can be supported, or something i can do to work around the issue (aside from simply not monitoring my pihole DNS server

The resolution lifetime expired after 5.138 seconds: Server Do53:192.168.0.91@53 answered got a response from ('172.21.0.1', 53) instead of ('192.168.0.91', 53); Server Do53:192.168.0.91@53 answered got a response from ('172.21.0.1', 53) instead of ('192.168.0.91', 53); Server Do53:192.168.0.91@53 answered got a response from ('172.21.0.1', 53) instead of ('192.168.0.91', 53); Server Do53:192.168.0.91@53 answered got a response from ('172.21.0.1', 53) instead of ('192.168.0.91', 53); Server Do53:192.168.0.91@53 answered got a response from ('172.21.0.1', 53) instead of ('192.168.0.91', 53); Server Do53:192.168.0.91@53 answered got a response from ('172.21.0.1', 53) instead of ('192.168.0.91', 53)

Use absolute paths in `docker-compose.yml`

Hello, good day!

Your Docker Compose file uses relative paths (e.g., ./config/...), which can be problematic when deploying through Portainer because the context directory might not be what you expect. Portainer might not have access to the same filesystem structure as your local development environment. Here are some steps and adjustments you can make to ensure it works properly:

General Adjustments

  1. Use Absolute Paths: When deploying with Portainer, it's better to use absolute paths for volumes to ensure the paths are correctly resolved.

  2. Verify Paths: Ensure all paths referenced in your docker-compose.yml actually exist on your host machine and are accessible.

  3. Permissions: Ensure the user running Portainer has the necessary permissions to access the files and directories specified.

One step to the right direction would be is to specify these /absolute/path/to/

version: '3'
name: netprobe

networks:
  netprobe-net:

services:
  redis:
    restart: always
    container_name: netprobe-redis
    image: "redis:latest"
    volumes:
      - /absolute/path/to/config/redis/redis.conf:/etc/redis/redis.conf
    networks:
      - netprobe-net
    dns:
      - 8.8.8.8
      - 8.8.4.4      

  netprobe:
    restart: always
    container_name: netprobe-probe
    image: "plaintextpackets/netprobe:latest"
    pull_policy: always
    volumes:
      - /absolute/path/to/netprobe_lite:/netprobe_lite
    environment:
      MODULE: "NETPROBE"
    networks:
      - netprobe-net
    dns:
      - 8.8.8.8
      - 8.8.4.4      

  speedtest:
    restart: always
    container_name: netprobe-speedtest
    image: "plaintextpackets/netprobe:latest"
    pull_policy: always
    volumes:
      - /absolute/path/to/netprobe_lite:/netprobe_lite
    environment:
      MODULE: "SPEEDTEST"
    networks:
      - netprobe-net
    dns:
      - 8.8.8.8
      - 8.8.4.4      

  presentation:
    restart: always
    container_name: netprobe-presentation
    image: "plaintextpackets/netprobe:latest"
    pull_policy: always
    volumes:
      - /absolute/path/to/netprobe_lite:/netprobe_lite
    environment:
      MODULE: "PRESENTATION"
    networks:
      - netprobe-net
    dns:
      - 8.8.8.8
      - 8.8.4.4      

  prometheus:
    restart: always
    container_name: netprobe-prometheus
    image: "prom/prometheus"
    volumes:
      - /absolute/path/to/config/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus  # Persistent local storage for Prometheus data
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
    networks:
      - netprobe-net
    dns:
      - 8.8.8.8
      - 8.8.4.4      

  grafana:
    restart: always
    image: grafana/grafana-enterprise
    container_name: netprobe-grafana
    volumes:
      - /absolute/path/to/config/grafana/datasources/automatic.yml:/etc/grafana/provisioning/datasources/automatic.yml
      - /absolute/path/to/config/grafana/dashboards/main.yml:/etc/grafana/provisioning/dashboards/main.yml
      - /absolute/path/to/config/grafana/dashboards/netprobe.json:/var/lib/grafana/dashboards/netprobe.json
      - grafana_data:/var/lib/grafana
    ports:
      - '3001:3000'
    networks:
      - netprobe-net
    dns:
      - 8.8.8.8
      - 8.8.4.4      
      
volumes:
  prometheus_data:
  grafana_data:

Applying these changes should help resolve #29

If any of the config files are containing some path on your machine please also make it clear if it required to be specified.

example:

improvements

hi i recently stumbled upon your video, and after having successfully diagnosed problems with my own ISP who erroneously claimed it was my own fault I thought maybe they could be included too.

I noticed that during periods of time, my connection would drop but only aspects of it, for example PPPoE would remain connected.

I managed to find out who the culprit was, by being lucky enough to run a trace route during the periods where it wouldn't pass traffic.

This information alongside a total bandwidth usage would be extremely useful inclusions. SNMP / NetFlow support would also be a nice inclusion but I doubt many are making use of that.

Running constant trace routes every 5 seconds or something? then if anybody experiences the same problems that I did, they would be able to look back and see which parts of their ISP weren't properly passing traffic from one to the other.

Exited with code 1 & 2

netprobe-grafana exited with code 1
netprobe-prometheus | ts=2024-05-14T19:34:12.604Z caller=main.go:573 level=info msg="No time or size retention was set so using the default time retention" duration=15d
netprobe-prometheus | ts=2024-05-14T19:34:12.605Z caller=main.go:617 level=info msg="Starting Prometheus Server" mode=server version="(version=2.52.0, branch=HEAD, revision=879d80922a227c37df502e7315fad8ceb10a986d)"
netprobe-prometheus | ts=2024-05-14T19:34:12.605Z caller=main.go:622 level=info build_context="(go=go1.22.3, platform=linux/amd64, user=root@1b4f4c206e41, date=20240508-21:56:43, tags=netgo,builtinassets,stringlabels)"
netprobe-prometheus | ts=2024-05-14T19:34:12.605Z caller=main.go:623 level=info host_details="(Linux 6.5.11-7-pve #1 SMP PREEMPT_DYNAMIC PMX 6.5.11-7 (2023-12-05T09:44Z) x86_64 69e28a7855d4 (none))"
netprobe-prometheus | ts=2024-05-14T19:34:12.605Z caller=main.go:624 level=info fd_limits="(soft=1048576, hard=1048576)"
netprobe-prometheus | ts=2024-05-14T19:34:12.605Z caller=main.go:625 level=info vm_limits="(soft=unlimited, hard=unlimited)"
netprobe-prometheus | ts=2024-05-14T19:34:12.607Z caller=query_logger.go:93 level=error component=activeQueryTracker msg="Error opening query log file" file=/prometheus/queries.active err="open /prometheus/queries.active: permission denied"
netprobe-prometheus | panic: Unable to create mmap-ed active query log
netprobe-prometheus |
netprobe-prometheus | goroutine 1 [running]:
netprobe-prometheus | github.com/prometheus/prometheus/promql.NewActiveQueryTracker({0x7ffc58e61f79, 0xb}, 0x14, {0x44aa9e0, 0xc00055cc30})
netprobe-prometheus | /app/promql/query_logger.go:123 +0x425
netprobe-prometheus | main.main()
netprobe-prometheus | /app/cmd/prometheus/main.go:748 +0x82fe
netprobe-prometheus exited with code 2

Could you point in the right direction to solve this?

Running netprobe with external Grafana instance

Hi!

I have a question, i'm interested in running netprobe, but i have an already instance of Grafana running on my net (running on Home Assistant, different host that is running docker). I'd like to include netprobe dashboard to my already running instance.
The point is, i'm not super familiar with docker compose files. Are you able to help me?

I assume that i need to (obviously) import dashboards definitions from repo to my grafana, comment grafana section in compose file.
Then instruct somehow that prometheus should be exposed to my internal network.

But what worries me the most, is that i cant figure out where Grafana in this repo keeps URL for datasource?

Need help, not running - Portainer issues

NNN---@NNN-TEST:/netprobe_lite$ docker compose up
WARN[0000] /home/NNN---/netprobe_lite/compose.yml: version is obsolete
[+] Running 5/0
✔ Container netprobe-presentation Created 0.0s
✔ Container netprobe-grafana Created 0.0s
✔ Container netprobe-probe Created 0.0s
✔ Container netprobe-redis Created 0.1s
✔ Container netprobe-prometheus Created 0.0s
Attaching to netprobe-grafana, netprobe-presentation, netprobe-probe, netprobe-prometheus, netprobe-redis
netprobe-prometheus | ts=2024-05-03T02:41:39.181Z caller=main.go:573 level=info msg="No time or size retention was set so using the default time retention" duration=15d
netprobe-prometheus | ts=2024-05-03T02:41:39.181Z caller=main.go:617 level=info msg="Starting Prometheus Server" mode=server version="(version=2.51.2, branch=HEAD, revision=b4c0ab52c3e9b940ab803581ddae9b3d9a452337)"
netprobe-prometheus | ts=2024-05-03T02:41:39.181Z caller=main.go:622 level=info build_context="(go=go1.22.2, platform=linux/amd64, user=root@b63f02a423d9, date=20240410-14:05:54, tags=netgo,builtinassets,stringlabels)"
netprobe-prometheus | ts=2024-05-03T02:41:39.181Z caller=main.go:623 level=info host_details="(Linux 5.4.0-177-generic #197-Ubuntu SMP Thu Mar 28 22:45:47 UTC 2024 x86_64 b856a8d8dbca (none))"
netprobe-prometheus | ts=2024-05-03T02:41:39.181Z caller=main.go:624 level=info fd_limits="(soft=1048576, hard=1048576)"
netprobe-prometheus | ts=2024-05-03T02:41:39.181Z caller=main.go:625 level=info vm_limits="(soft=unlimited, hard=unlimited)"
netprobe-prometheus | ts=2024-05-03T02:41:39.184Z caller=web.go:568 level=info component=web msg="Start listening for connections" address=0.0.0.0:9090
netprobe-prometheus | ts=2024-05-03T02:41:39.185Z caller=main.go:1129 level=info msg="Starting TSDB ..."
netprobe-prometheus | ts=2024-05-03T02:41:39.186Z caller=dir_locker.go:77 level=warn component=tsdb msg="A lockfile from a previous execution already existed. It was replaced" file=/prometheus/lock
netprobe-prometheus | ts=2024-05-03T02:41:39.192Z caller=tls_config.go:313 level=info component=web msg="Listening on" address=[::]:9090
netprobe-prometheus | ts=2024-05-03T02:41:39.192Z caller=tls_config.go:316 level=info component=web msg="TLS is disabled." http2=false address=[::]:9090
netprobe-prometheus | ts=2024-05-03T02:41:39.194Z caller=head.go:616 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any"
netprobe-prometheus | ts=2024-05-03T02:41:39.194Z caller=head.go:698 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.95µs
netprobe-prometheus | ts=2024-05-03T02:41:39.194Z caller=head.go:706 level=info component=tsdb msg="Replaying WAL, this may take a while"
netprobe-prometheus | ts=2024-05-03T02:41:39.225Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3
netprobe-prometheus | ts=2024-05-03T02:41:39.226Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3
netprobe-prometheus | ts=2024-05-03T02:41:39.227Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3
netprobe-prometheus | ts=2024-05-03T02:41:39.227Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3
netprobe-prometheus | ts=2024-05-03T02:41:39.227Z caller=head.go:815 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=95.788µs wal_replay_duration=32.827096ms wbl_replay_duration=161ns total_replay_duration=32.974262ms
netprobe-prometheus | ts=2024-05-03T02:41:39.230Z caller=main.go:1150 level=info fs_type=EXT4_SUPER_MAGIC
netprobe-prometheus | ts=2024-05-03T02:41:39.230Z caller=main.go:1153 level=info msg="TSDB started"
netprobe-prometheus | ts=2024-05-03T02:41:39.230Z caller=main.go:1335 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
netprobe-prometheus | ts=2024-05-03T02:41:39.233Z caller=main.go:1372 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=2.440331ms db_storage=1.321µs remote_storage=1.21µs web_handler=592ns query_engine=1.12µs scrape=2.11521ms scrape_sd=38.265µs notify=795ns notify_sd=1.333µs rules=1.361µs tracing=6.411µs
netprobe-prometheus | ts=2024-05-03T02:41:39.233Z caller=main.go:1114 level=info msg="Server is ready to receive web requests."
netprobe-prometheus | ts=2024-05-03T02:41:39.233Z caller=manager.go:163 level=info component="rule manager" msg="Starting rule manager..."
netprobe-redis | 1:C 03 May 2024 02:41:39.248 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see jemalloc/jemalloc#1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
netprobe-redis | 1:C 03 May 2024 02:41:39.248 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
netprobe-redis | 1:C 03 May 2024 02:41:39.248 * Redis version=7.2.4, bits=64, commit=00000000, modified=0, pid=1, just started
netprobe-redis | 1:C 03 May 2024 02:41:39.248 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
netprobe-redis | 1:M 03 May 2024 02:41:39.249 * monotonic clock: POSIX clock_gettime
netprobe-redis | 1:M 03 May 2024 02:41:39.250 * Running mode=standalone, port=6379.
netprobe-redis | 1:M 03 May 2024 02:41:39.251 * Server initialized
netprobe-redis | 1:M 03 May 2024 02:41:39.251 * Ready to accept connections tcp
netprobe-probe | /netprobe_lite/helpers/network_helper.py:25: SyntaxWarning: invalid escape sequence '|'
netprobe-probe | ping = subprocess.getoutput(f"ping -n -i 0.1 -c {count} {site} | grep 'rtt|loss'")
Error response from daemon: driver failed programming external connectivity on endpoint netprobe-grafana (588de0b3fe6d13d712538a22086b31bbbe5bd9f4c0ce4753f92b700e8aafff26): Bind for 0.0.0.0:3001 failed: port is already allocated
NNN---@NNN-TEST:
/netprobe_lite$ sudo vim compose.yml
NNN---@NNN-TEST:~/netprobe_lite$ docker compose up
WARN[0000] /home/NNN---/netprobe_lite/compose.yml: version is obsolete
[+] Running 5/0
✔ Container netprobe-probe Running 0.0s
✔ Container netprobe-redis Running 0.0s
✔ Container netprobe-prometheus Running 0.0s
✔ Container netprobe-grafana Recreated 0.0s
✔ Container netprobe-presentation Running 0.0s
Attaching to netprobe-grafana, netprobe-presentation, netprobe-probe, netprobe-prometheus, netprobe-redis
Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/home/NNN---/netprobe_lite/config/grafana/dashboards/netprobe.json" to rootfs at "/var/lib/grafana/dashboards/netprobe.json": mount /home/NNN---/netprobe_lite/config/grafana/dashboards/netprobe.json:/var/lib/grafana/dashboards/netprobe.json (via /proc/self/fd/6), flags: 0x5000: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

Helm chart

I run a Kubernetes cluster at home (microk8s) and I'm thinking of creating a helm chart out of this repository so we can easily run it in there instead of having to deal with docker compose. At this point I'm wonder if maybe somebody already did this, and if not, if people might be interested in running this on Kubernetes.

Failing to start

Running this on Ubuntu 20.
Running Docker with portainer on top.

When I try to start it, it gives an error:
/bin/bash: ./entrypoint.sh: No such file or directory

Any advice or tips?

Thanks

Not seeming to gather any, Logs showing this.

I'm running from docker compose, only edits were to put my own local ISP DNS in the .env though I note that removing that change makes no difference to this.

netprobe-presentation  | Traceback (most recent call last):
netprobe-presentation  |   File "/usr/lib/python3.12/wsgiref/handlers.py", line 137, in run
netprobe-presentation  |     self.result = application(self.environ, self.start_response)
netprobe-presentation  |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
netprobe-presentation  |   File "/usr/local/lib/python3.12/dist-packages/prometheus_client/exposition.py", line 128, in prometheus_app
netprobe-presentation  |     status, headers, output = _bake_output(registry, accept_header, accept_encoding_header, params, disable_compression)
netprobe-presentation  |                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
netprobe-presentation  |   File "/usr/local/lib/python3.12/dist-packages/prometheus_client/exposition.py", line 104, in _bake_output
netprobe-presentation  |     output = encoder(registry)
netprobe-presentation  |              ^^^^^^^^^^^^^^^^^
netprobe-presentation  |   File "/usr/local/lib/python3.12/dist-packages/prometheus_client/openmetrics/exposition.py", line 21, in generate_latest
netprobe-presentation  |     for metric in registry.collect():
netprobe-presentation  |   File "/usr/local/lib/python3.12/dist-packages/prometheus_client/registry.py", line 97, in collect
netprobe-presentation  |     yield from collector.collect()
netprobe-presentation  |   File "/netprobe_lite/presentation.py", line 101, in collect
netprobe-presentation  |     if my_dns_latency / threshold_dns_latency >= 1:
netprobe-presentation  |        ^^^^^^^^^^^^^^
netprobe-presentation  | UnboundLocalError: cannot access local variable 'my_dns_latency' where it is not associated with a value
netprobe-presentation  | Traceback (most recent call last):
netprobe-presentation  |   File "/usr/lib/python3.12/wsgiref/handlers.py", line 137, in run
netprobe-presentation  |     self.result = application(self.environ, self.start_response)
netprobe-presentation  |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
netprobe-presentation  |   File "/usr/local/lib/python3.12/dist-packages/prometheus_client/exposition.py", line 128, in prometheus_app
netprobe-presentation  |     status, headers, output = _bake_output(registry, accept_header, accept_encoding_header, params, disable_compression)
netprobe-presentation  |                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
netprobe-presentation  |   File "/usr/local/lib/python3.12/dist-packages/prometheus_client/exposition.py", line 104, in _bake_output
netprobe-presentation  |     output = encoder(registry)
netprobe-presentation  |              ^^^^^^^^^^^^^^^^^
netprobe-presentation  |   File "/usr/local/lib/python3.12/dist-packages/prometheus_client/openmetrics/exposition.py", line 21, in generate_latest
netprobe-presentation  |     for metric in registry.collect():
netprobe-presentation  |   File "/usr/local/lib/python3.12/dist-packages/prometheus_client/registry.py", line 97, in collect
netprobe-presentation  |     yield from collector.collect()
netprobe-presentation  |   File "/netprobe_lite/presentation.py", line 101, in collect
netprobe-presentation  |     if my_dns_latency / threshold_dns_latency >= 1:
netprobe-presentation  |        ^^^^^^^^^^^^^^
netprobe-presentation  | UnboundLocalError: cannot access local variable 'my_dns_latency' where it is not associated with a value

Package differently?

I thank you so much for writing this and putting it together.
I wonder if now that you have it rolling along, you might consider publishing out an image. We could just pull down the image, create our volumes and drop the env file(s) into the external volume.
The way it is now, I build everything by following your directions, but then I make a Docker Compose inside Portainer that references the previously-built images.

Could be so much more convenient if you published the image. But I know, only so much time in the day!

Not Storing Data?????

Debian 12 Container on Proxmox

Starts and Runs Fine but,
Grafani wont Store new Password
Prometheus isnt storing Data

Happens with CT Reboot on Docker down

Builds no longer working

Self-opening this

For some reason when building manually now (docker build .) pip no longer recognizes the ' --break-system-packages' option:

321.4 Setting up iputils-ping (3:20211215-1) ...
323.4
323.4 Usage:
323.4 pip install [options] [package-index-options] ...
323.4 pip install [options] -r [package-index-options] ...
323.4 pip install [options] [-e] ...
323.4 pip install [options] [-e] ...
323.4 pip install [options] <archive url/path> ...
323.4
323.4 no such option: --break-system-packages

netprobe as standalone app

Hi there, is there a way to setup netprobe lite as standalone app where redis, prometheus and grafana are external instances?
I have these setup already in my home environment and would like to use it instead of just setting new instances of these dockers via docker-compose?

Network Pool overlap

On trying to to a "docker compose up -d" I get the following error:

Network netprobe_custom_network Error
failed to create network netprobe_custom_network: Error response from daemon: Pool overlaps with other one on this address space

Docker compose startup errors on MacOS

Thanks for building the Netprobe monitor! I did note a couple of issues that I wanted to share.

  1. The installation on MacOS Sonoma of https://github.com/plaintextpackets/netprobe_lite throws the error "failed to solve: process "/bin/sh -c apt-get update -y && apt-get install -y python3 && apt-get install -y python3-pip && apt-get install -y iputils-ping && pip install -r /netprobe_lite/requirements.txt" did not complete successfully: exit code: 1" after "docker compose up".

  2. After correcting the issue 1, the error "debconf: unable to initialize frontend: Dialog" appeared. This indicates that debconf, a configuration system for Debian packages, is trying to use a dialog-based frontend, but it fails because the Docker container does not have a user interface for dialogs. Fixed this issue by configuring debconf to use a non-interactive frontend.

  3. After issue 2 was fixed, the apt-get install throws the error "error: externally-managed-environment". Worked around this error by configuring a virtual environment.

The dockerfile below is the version I used to successfully startup Netprobe.

`FROM ubuntu:latest

COPY requirements.txt /netprobe_lite/requirements.txt
ENV DEBIAN_FRONTEND=noninteractive

Install python/pip

ENV PYTHONUNBUFFERED=1
RUN apt-get update -y
&& apt-get install -y python3
&& apt-get install -y python3-pip
&& apt-get install -y iputils-ping
&& apt-get install -y python3-venv

Setup a virtual environment if needed

RUN python3 -m venv /venv
ENV PATH="/venv/bin:$PATH"

RUN pip install --no-cache-dir -r /netprobe_lite/requirements.txt

ENV DEBIAN_FRONTEND=dialog

WORKDIR /netprobe_lite

ENTRYPOINT [ "/bin/bash", "./entrypoint.sh" ]`

Unauthorized and constant reloading

When deploying via docker-compose grafana will give you constant "Unauthorized" messages wich will trigger reloads when viewing the dashboard that's been provisioned.

In the current state it is more or less unusable.
I've been fiddling around with this and changing the grafana version to 10.1.6 solves the problem completely.

if you open the web developer console you will see the following messages spammed:
msg=Unauthorized error="user token not found"

DNS, Encrypted, my_dns_latency error

Not sure what the issue is. I def had to go through a few steps to get things to work.
had to disable TinyFirewall to get docker to load, then working... I ran into this issue (see log file below)
docker compose up runs and works and then log outputs this on repeat. I did have encrypted DNS in windows natively, i disabled it and reset to auto DHCP DNS, the DNS returned to be default 1.1.1.1 9.9.9.9 (even restarting it didnt reset to my gateway), but it clearly says unencrypted now.
so this error log looks like DoH / encrypted DNS, but not sure what else to do that is the issue.

Log

netprobe-probe | Error performing DNS resolution on ('Google_DNS', '8.8.8.8')
netprobe-probe | The resolution lifetime expired after 5.101 seconds: Server Do53:8.8.8.8@53 answered The DNS operation timed out.
netprobe-probe | Error performing DNS resolution on ('Quad9_DNS', '9.9.9.9')
netprobe-probe | The resolution lifetime expired after 5.106 seconds: Server Do53:9.9.9.9@53 answered The DNS operation timed out.
netprobe-probe | Error performing DNS resolution on ('CloudFlare_DNS', '1.1.1.1')
netprobe-probe | The resolution lifetime expired after 5.106 seconds: Server Do53:1.1.1.1@53 answered The DNS operation timed out.
netprobe-probe | Error performing DNS resolution on ('My_DNS_Server', '8.8.8.8')
netprobe-probe | The resolution lifetime expired after 5.106 seconds: Server Do53:8.8.8.8@53 answered The DNS operation timed out.
netprobe-presentation | Traceback (most recent call last):
netprobe-presentation | File "/usr/lib/python3.12/wsgiref/handlers.py", line 137, in run
netprobe-presentation | self.result = application(self.environ, self.start_response)
netprobe-presentation | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
netprobe-presentation | File "/usr/local/lib/python3.12/dist-packages/prometheus_client/exposition.py", line 128, in prometheus_app
netprobe-presentation | status, headers, output = _bake_output(registry, accept_header, accept_encoding_header, params, disable_compression)
netprobe-presentation | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
netprobe-presentation | File "/usr/local/lib/python3.12/dist-packages/prometheus_client/exposition.py", line 104, in _bake_output
netprobe-presentation | output = encoder(registry)
netprobe-presentation | ^^^^^^^^^^^^^^^^^
netprobe-presentation | File "/usr/local/lib/python3.12/dist-packages/prometheus_client/openmetrics/exposition.py", line 21, in generate_latest
netprobe-presentation | for metric in registry.collect():
netprobe-presentation | File "/usr/local/lib/python3.12/dist-packages/prometheus_client/registry.py", line 97, in collect
netprobe-presentation | yield from collector.collect()
netprobe-presentation | File "/netprobe_lite/presentation.py", line 106, in collect
netprobe-presentation | if my_dns_latency / threshold_dns_latency >= 1:
netprobe-presentation | ^^^^^^^^^^^^^^
netprobe-presentation | UnboundLocalError: cannot access local variable 'my_dns_latency' where it is not associated with a value

Errors launching from docker-compose on Ubuntu

After downloading the .env and docker-compose.yml files, I only changed the "DNS_NAMESERVER_4_IP" value.

I see 2 errors: a missing ./entrypoint and a directory named ./config/prometheus/prometheus.yml is a directory, not a fie.

user@librenms ~/docker/netprobe$ docker-compose up
WARN[0000] /home/user/docker/netprobe/docker-compose.yml: `version` is obsolete 
[+] Running 1/6
 ✔ Network netprobe_custom_network  Created                                                                                                                                                                 0.0s 
 ⠋ Container netprobe-presentation  Created                                                                                                                                                                 6.0s 
 ⠋ Container netprobe-grafana       Created                                                                                                                                                                 6.0s 
 ⠋ Container netprobe-prometheus    Created                                                                                                                                                                 6.0s 
 ⠋ Container netprobe-redis         Created                                                                                                                                                                 6.0s 
 ⠋ Container netprobe-probe         Created                                                                                                                                                                 6.0s 
Attaching to netprobe-grafana, netprobe-presentation, netprobe-probe, netprobe-prometheus, netprobe-redis
netprobe-presentation  | /bin/bash: ./entrypoint.sh: No such file or directory
netprobe-redis         | 1:C 02 May 2024 15:08:58.911 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
netprobe-redis         | 1:C 02 May 2024 15:08:58.911 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
netprobe-redis         | 1:C 02 May 2024 15:08:58.911 * Redis version=7.2.4, bits=64, commit=00000000, modified=0, pid=1, just started
netprobe-redis         | 1:C 02 May 2024 15:08:58.911 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
netprobe-redis         | 1:M 02 May 2024 15:08:58.912 * monotonic clock: POSIX clock_gettime
netprobe-redis         | 1:M 02 May 2024 15:08:58.912 * Running mode=standalone, port=6379.
netprobe-redis         | 1:M 02 May 2024 15:08:58.913 * Server initialized
netprobe-redis         | 1:M 02 May 2024 15:08:58.913 * Ready to accept connections tcp
netprobe-probe         | /bin/bash: ./entrypoint.sh: No such file or directory
netprobe-grafana       | logger=settings t=2024-05-02T15:08:59.121729802Z level=info msg="Starting Grafana" version=10.4.2 commit=22809dea50455ae875624827dd277b17621b4ed4 branch=HEAD compiled=2024-05-02T15:08:59Z
netprobe-grafana       | logger=settings t=2024-05-02T15:08:59.121908214Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini
netprobe-grafana       | logger=settings t=2024-05-02T15:08:59.121914704Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini
netprobe-grafana       | logger=settings t=2024-05-02T15:08:59.121917612Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana"
netprobe-grafana       | logger=settings t=2024-05-02T15:08:59.121936956Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana"
netprobe-grafana       | logger=settings t=2024-05-02T15:08:59.121939178Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins"
netprobe-grafana       | logger=settings t=2024-05-02T15:08:59.121941478Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning"
netprobe-grafana       | logger=settings t=2024-05-02T15:08:59.121945416Z level=info msg="Config overridden from command line" arg="default.log.mode=console"
netprobe-grafana       | logger=settings t=2024-05-02T15:08:59.121949903Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana"
netprobe-grafana       | logger=settings t=2024-05-02T15:08:59.122041216Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana"
netprobe-grafana       | logger=settings t=2024-05-02T15:08:59.122045193Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins"
netprobe-grafana       | logger=settings t=2024-05-02T15:08:59.12204732Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning"
netprobe-grafana       | logger=settings t=2024-05-02T15:08:59.1220522Z level=info msg=Target target=[all]
netprobe-grafana       | logger=settings t=2024-05-02T15:08:59.122092271Z level=info msg="Path Home" path=/usr/share/grafana
netprobe-grafana       | logger=settings t=2024-05-02T15:08:59.122104395Z level=info msg="Path Data" path=/var/lib/grafana
netprobe-grafana       | logger=settings t=2024-05-02T15:08:59.122115087Z level=info msg="Path Logs" path=/var/log/grafana
netprobe-grafana       | logger=settings t=2024-05-02T15:08:59.122125003Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins
netprobe-grafana       | logger=settings t=2024-05-02T15:08:59.122134868Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning
netprobe-grafana       | logger=settings t=2024-05-02T15:08:59.12217373Z level=info msg="App mode production"
netprobe-grafana       | logger=sqlstore t=2024-05-02T15:08:59.122498901Z level=info msg="Connecting to DB" dbtype=sqlite3
netprobe-grafana       | logger=migrator t=2024-05-02T15:08:59.123275598Z level=info msg="Starting DB migrations"
netprobe-grafana       | logger=migrator t=2024-05-02T15:08:59.136804414Z level=info msg="migrations completed" performed=0 skipped=643 duration=553.464µs
netprobe-grafana       | logger=licensing t=2024-05-02T15:08:59.137704075Z level=info msg="Validated license token" appURL=http://localhost:3000/ source=disk status=NotFound
netprobe-grafana       | logger=secrets t=2024-05-02T15:08:59.138311247Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1
netprobe-grafana       | logger=caching.service t=2024-05-02T15:08:59.155085159Z level=info msg="Caching service is disabled"
netprobe-grafana       | logger=plugin.store t=2024-05-02T15:08:59.155880704Z level=info msg="Loading plugins..."
netprobe-grafana       | logger=local.finder t=2024-05-02T15:08:59.205960673Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled
netprobe-grafana       | logger=plugin.store t=2024-05-02T15:08:59.205974793Z level=info msg="Plugins loaded" count=55 duration=50.094597ms
netprobe-grafana       | logger=query_data t=2024-05-02T15:08:59.208386649Z level=info msg="Query Service initialization"
netprobe-grafana       | logger=live.push_http t=2024-05-02T15:08:59.211517881Z level=info msg="Live Push Gateway initialization"
netprobe-grafana       | logger=ngalert.migration t=2024-05-02T15:08:59.218295553Z level=info msg=Starting
netprobe-grafana       | logger=ngalert.state.manager t=2024-05-02T15:08:59.226472789Z level=info msg="Running in alternative execution of Error/NoData mode"
netprobe-grafana       | logger=infra.usagestats.collector t=2024-05-02T15:08:59.228250772Z level=info msg="registering usage stat providers" usageStatsProvidersLen=4
netprobe-grafana       | logger=provisioning t=2024-05-02T15:08:59.228489747Z level=error msg="Failed to provision data sources" error="Datasource provisioning error: read /etc/grafana/provisioning/datasources/automatic.yml: is a directory"
netprobe-grafana       | logger=provisioning t=2024-05-02T15:08:59.228497428Z level=error msg="Failed to provision data sources" error="Datasource provisioning error: read /etc/grafana/provisioning/datasources/automatic.yml: is a directory"
netprobe-grafana       | Error: ✗ Datasource provisioning error: read /etc/grafana/provisioning/datasources/automatic.yml: is a directory
Gracefully stopping... (press Ctrl+C again to force)
Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/home/user/docker/netprobe/config/prometheus/prometheus.yml" to rootfs at "/etc/prometheus/prometheus.yml": mount /home/user/docker/netprobe/config/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml (via /proc/self/fd/6), flags: 0x5400: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type
user@librenms ~/docker/netprobe

Packet Loss misspelled

In the Grafana interface - the Packet Loss (Avg) section is misspelled as "Pakcet Loss (Avg)"

Where is .env file?

I’d like to know where is the .env file to change the configuration of the hosts and the DNS.

regards
SdP

How can i see Packet loss per IP or domain

Hi

Now i am surprised that yesterday i had zero packet loss on chart, but today it is all the time around 1to 5% on netstat chart, which looks severe.

How do i know what test website isnt repllying, as it shows like internet is bad, but i suspect that one of the test websites is iffy as its saturday.

How can i confirm that there is no severe issue with my internet, as UptimeKuma says 100% to google and synology servers, and my internet experience today is fine.

Hwo can i display metrics per IP or domain of monitors for packet loss, so i can know the cause.

Thanks, Matej

Screenshot_20240427_121517_Brave
Screenshot_20240427_110934_Brave

Updating from Docker

When you have posted an update, is it simply a case of doing a docker compose down, docker compose pull and docker compose up -d or is there other actions needed?

Installation on QNAP

I'm trying to get this installed on my qnap server, does it happen to appear in the docker hub?

Usually I can installed docker apps by going through ssh with a command like:

docker run -d
kschovan/netprobelite

Any other tips on getting it installed, looks like the exact thing I have been looking for to keep an eye on my internet (that often seems to perform subpar)

speed test provider change

Hello, I ‘m currently uses this great app to monitor my network conditions, but found that there is a better speed test website I can use for my network environment, so anyway I can change a speed test service provider (different speed test website) in the speed test probe container? Thanks for the help

docker compose up error: externally-managed-environment

Facing an error during docker compose up. Excerpt of the log:

 => CACHED [presentation 2/4] COPY requirements.txt /netprobe_lite/requirements.txt                                       0.0s
 => ERROR [presentation 3/4] RUN apt update -y && apt install -y iputils-ping && apt install -y python3 && apt install  506.4s
------
 > [presentation 3/4] RUN apt update -y && apt install -y iputils-ping && apt install -y python3 && apt install -y python3-pip && pip install -r /netprobe_lite/requirements.txt:
...
...
522.7 Setting up libheif-plugin-aomenc:amd64 (1.17.6-1ubuntu4) ...
523.3 Processing triggers for libc-bin (2.39-0ubuntu8) ...
524.5 Reading package lists...
526.2 Building dependency tree...
526.3 Reading state information...
527.3 The following additional packages will be installed:
527.3   libcap2-bin libpam-cap
527.4 The following NEW packages will be installed:
527.4   iputils-ping libcap2-bin libpam-cap
528.0 0 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
528.0 Need to get 91.1 kB of archives.
528.0 After this operation, 319 kB of additional disk space will be used.
528.0 Get:1 http://archive.ubuntu.com/ubuntu noble/main amd64 libcap2-bin amd64 1:2.66-5ubuntu2 [34.5 kB]
528.4 Get:2 http://archive.ubuntu.com/ubuntu noble/main amd64 iputils-ping amd64 3:20240117-1build1 [44.3 kB]
528.5 Get:3 http://archive.ubuntu.com/ubuntu noble/main amd64 libpam-cap amd64 1:2.66-5ubuntu2 [12.4 kB]
529.3 debconf: delaying package configuration, since apt-utils is not installed
529.3 Fetched 91.1 kB in 1s (77.8 kB/s)
529.7 Selecting previously unselected package libcap2-bin.
(Reading database ... 17117 files and directories currently installed.)
529.8 Preparing to unpack .../libcap2-bin_1%3a2.66-5ubuntu2_amd64.deb ...
530.1 Unpacking libcap2-bin (1:2.66-5ubuntu2) ...
531.3 Selecting previously unselected package iputils-ping.
531.3 Preparing to unpack .../iputils-ping_3%3a20240117-1build1_amd64.deb ...
531.4 Unpacking iputils-ping (3:20240117-1build1) ...
532.7 Selecting previously unselected package libpam-cap:amd64.
532.7 Preparing to unpack .../libpam-cap_1%3a2.66-5ubuntu2_amd64.deb ...
532.8 Unpacking libpam-cap:amd64 (1:2.66-5ubuntu2) ...
533.9 Setting up libcap2-bin (1:2.66-5ubuntu2) ...
534.9 Setting up libpam-cap:amd64 (1:2.66-5ubuntu2) ...
535.7 debconf: unable to initialize frontend: Dialog
535.7 debconf: (TERM is not set, so the dialog frontend is not usable.)
535.7 debconf: falling back to frontend: Readline
535.7 debconf: unable to initialize frontend: Readline
535.7 debconf: (This frontend requires a controlling tty.)
535.7 debconf: falling back to frontend: Teletype
536.2 Setting up iputils-ping (3:20240117-1build1) ...
539.1 error: externally-managed-environment
539.1
539.1 × This environment is externally managed
539.1 ╰─> To install Python packages system-wide, try apt install
539.1     python3-xyz, where xyz is the package you are trying to
539.1     install.
539.1
539.1     If you wish to install a non-Debian-packaged Python package,
539.1     create a virtual environment using python3 -m venv path/to/venv.
539.1     Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
539.1     sure you have python3-full installed.
539.1
539.1     If you wish to install a non-Debian packaged Python application,
539.1     it may be easiest to use pipx install xyz, which will manage a
539.1     virtual environment for you. Make sure you have pipx installed.
539.1
539.1     See /usr/share/doc/python3.12/README.venv for more information.
...
...
------
failed to solve: process "/bin/sh -c apt update -y && apt install -y iputils-ping && apt install -y python3 && apt install -y python3-pip && pip install -r /netprobe_lite/requirements.txt" did not complete successfully: exit code: 1

I'm installing this in my home lab: an LXC container in Proxmox VE 7.4-17 on Dell PowerEdge R720.

In the latest image the package loss is reporting extremly high

I have stable Fibre connection, I have run tests from inside all the docker images, and they all come back with a very quick response (under 10ms for almost all) and no real changes or drop, however for some reason the system reports that it's losing 52% of items - I tried to look into the config but can't find anything that would point me to what is causing that issue. Any suggestions would be appreciated (all other values show correctly, so latency 6.4ms, jitter is 0.3ms, DNS response 1ms)

Package display container separatly

Is there a way to put the display (graphics) container in a separate compose bundle. That way I can run the probe in background without the overhead of the graphics. Then can start and stop the graphics to view results. I’m running this on a pi4/ubuntu server and need to conserve resources.

Netprobe uses same port as Uptime Kuma

Error response from daemon: driver failed programming external connectivity on endpoint netprobe-grafana (b4ac30551a4e633f7cd61067515636022e2351fad2ad2457ecb45e5c6a23996b): Bind for 0.0.0.0:3001 failed: port is already allocated

When speedtest gets a 404 it doesn't back off

2024-05-17 22:05:22 ERROR Error testing network
2024-05-17 22:05:22 ERROR HTTP Error 403: Forbidden
2024-05-17 22:05:22 ERROR Error testing network
2024-05-17 22:05:22 ERROR HTTP Error 403: Forbidden
2024-05-17 22:05:22 ERROR Error testing network
2024-05-17 22:05:22 ERROR HTTP Error 403: Forbidden
2024-05-17 22:05:22 ERROR Error testing network
2024-05-17 22:05:22 ERROR HTTP Error 403: Forbidden
2024-05-17 22:05:22 ERROR Error testing network
2024-05-17 22:05:22 ERROR HTTP Error 403: Forbidden
2024-05-17 22:05:22 ERROR Error testing network
2024-05-17 22:05:22 ERROR HTTP Error 403: Forbidden
2024-05-17 22:05:22 ERROR Error testing network
2024-05-17 22:05:22 ERROR HTTP Error 403: Forbidden
2024-05-17 22:05:22 ERROR Error testing network
2024-05-17 22:05:22 ERROR HTTP Error 403: Forbidden
2024-05-17 22:05:22 ERROR Error testing network
2024-05-17 22:05:22 ERROR HTTP Error 403: Forbidden

Speed test request / speed test not showing full speed

Great info here.

Any chance to have a speed test added into the graph as well?

External IP addresses would be nice too, to know if the router failed over for a period of time. Maybe show the current IP, and a link to the history? You can get that from ifconfig.me/ip

Division by 0 issue in presentation

2024-05-24 10:58:52 ZeroDivisionError: division by zero
2024-05-24 11:01:22 Traceback (most recent call last):
2024-05-24 11:01:22 File "/usr/local/lib/python3.11/wsgiref/handlers.py", line 137, in run
2024-05-24 11:01:22 self.result = application(self.environ, self.start_response)
2024-05-24 11:01:22 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-05-24 11:01:22 File "/usr/local/lib/python3.11/site-packages/prometheus_client/exposition.py", line 128, in prometheus_app
2024-05-24 11:01:22 status, headers, output = _bake_output(registry, accept_header, accept_encoding_header, params, disable_compression)
2024-05-24 11:01:22 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-05-24 11:01:22 File "/usr/local/lib/python3.11/site-packages/prometheus_client/exposition.py", line 104, in _bake_output
2024-05-24 11:01:22 output = encoder(registry)
2024-05-24 11:01:22 ^^^^^^^^^^^^^^^^^
2024-05-24 11:01:22 File "/usr/local/lib/python3.11/site-packages/prometheus_client/openmetrics/exposition.py", line 21, in generate_latest
2024-05-24 11:01:22 for metric in registry.collect():
2024-05-24 11:01:22 File "/usr/local/lib/python3.11/site-packages/prometheus_client/registry.py", line 97, in collect
2024-05-24 11:01:22 yield from collector.collect()
2024-05-24 11:01:22 File "/netprobe_lite/presentation.py", line 59, in collect
2024-05-24 11:01:22 average_latency = total_latency / len(stats_netprobe['stats'])
2024-05-24 11:01:22 ~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2024-05-24 11:01:22 ZeroDivisionError: division by zero

Support for the old Raspberry Pi Model B

Hey there,
I have two old raspberry pi model B laying around and I thought I could use one for your project.
I installed the newest version of Raspberry Pi OS (Legacy, 32-Bit) Lite and Docker Engine. After configuring the .env file I tried to “docker compose up” but it stopped with :

[+] Running 4/6
⠏ grafana Pulling 6.9s
✘ redis Error context canceled 6.9s
⠇ netprobe Pulling 6.9s
✘ speedtest Error context canceled 6.9s
✘ presentation Error context canceled 6.8s
✘ prometheus Error context canceled 6.8s
no matching manifest for linux/arm/v6 in the manifest list entries

Is there any way around this? Is there any way you could add support for such old devices or is it time to upgrade ? :D

Thanks a lot and have a great day!

netprobe speedtest not working

netprobe-speedtest (ver. 1.4.2) is not working and trying to restart all the time.
netprobe was running with default docker compose

Config changes stops Graphana displaying data

Hi, first of all great job !

I'm running Almalinux 9 and Python 3.9.18.

Installing this tool in a venv en start it, it runs perfectly. However when I change DNS_NAMESERVER_4_IP to my own address and change one of the sites, data is being collected (logging is telling me). But Graphana stops displaying data.

Any suggestions how to find the issue or how to fix this ?

Best regards,
Remco

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.