Support for NPMPlus?

[smilebasti]

Support for NPMPlus?

[tokar86a]

This is how the log locks like.

[12/Mar/2024:07:06:08 +0100] tokar86a.top ip address 0.192 "GET / HTTP/2.0" 200 36737 37435 - Uptime-Kuma/1.23.11

[tokar86a]

Thanks for the notice. As it will stay nginx as proxy for now, I don't see a problem.

Thats good to hear.

[smilebasti]

Sadly not. I have tried a few options of tail with -F or just -f or similar, but none of them worked. Also tried setting the log rotation to 0.

I currently don't have the time to test other options or rewrite the whole thing. If you want to try it, just download the scripts and test variations or if you have experience you could rewrite the code to help.

[smilebasti]

Hi,
i haven't had heard about project before. It definitely looks promising.

Have you tried running npmGrafStats as the project is a fork from the original NPM?

Are you able to provide a censored log entry from NPMPlus and the log location? Then i can have a look at the default output to logs.

[tokar86a]

Yes i have try and it is dealing with the log in a different way. Instead of different logs related to domain/proxy host it is all centralized to one access log. But it is not working as it is now as I have try.

The logs rotation is something that the user can change. Maybe try to spin it up and look for you self then you would easy get the date that you need. Or maybe ask the developer?

Hope this helps :)

[tokar86a]

So what do you think?

[smilebasti]

I haven't had to test it and haven't forgotten you.

Nit sure if i will get to it this weekend but probably over the Easter holidays.

[tokar86a]

Nice and thanks for the updated. No rush on this as i was more looking for on what you think about it.

It is your project after all and it don`t seems like allot of user use the NPMplus as it was new to you.

[gadget78]

whats NPMPlus like ? i am a little curious,
has it filled alot of NPM bugs, and shortfalls ? i did see its got a nice dark mode !
and talks more about the API, does it have more Stream options ?

[tokar86a]

whats NPMPlus like ? i am a little curious, has it filled alot of NPM bugs, and shortfalls ? i did see its got a nice dark mode ! and talks more about the API, does it have more Stream options ?

Have you check out there Github page? https://github.com/ZoeyVid/NPMplus

[smilebasti]

This is how the log locks like.

[12/Mar/2024:07:06:08 +0100] tokar86a.top ip address 0.192 "GET / HTTP/2.0" 200 36737 37435 - Uptime-Kuma/1.23.11

@tokar86a I just tried it to set it up. Maybe you can help me. I set LOGROTATE=true but i only see the NPMplus dashboard logs appear in the /opt/npm/ngin/access.log file. Am i missing something?

[tokar86a]

This is how the log locks like.
[12/Mar/2024:07:06:08 +0100] tokar86a.top ip address 0.192 "GET / HTTP/2.0" 200 36737 37435 - Uptime-Kuma/1.23.11

@tokar86a I just tried it to set it up. Maybe you can help me. I set LOGROTATE=true but i only see the NPMplus dashboard logs appear in the /opt/npm/ngin/access.log file. Am i missing something?

What you mean whit dashboards logs?

I use the following environment:
- TZ=Europe/Stockholm # set timezone, required
- NGINX_LOG_NOT_FOUND=true
- GOA=true
- LOGROTATE=true
- LOGROTATIONS=15

https://prnt.sc/2CJ9JBE80GN5

[tokar86a]

[10/Apr/2024:16:53:22 +0200] grafana.sysadmin222.top 172.68.243.31 0.006 "GET /api/ruler/grafana/api/v1/rules?subtype=cortex HTTP/2.0" 200 1826 2159 https://grafana.sysadmin222.top/alerting/grafana/f3936ead-3fee-45ef-a8db-d9bada2dfacb/view?returnTo=%2Falerting%2Flist Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:124.0) Gecko/20100101 Firefox/124.0 [10/Apr/2024:16:53:24 +0200] grafana.sysadmin222.top 172.68.243.31 0.004 "GET /api/search?type=dash-db&query=&limit=100 HTTP/2.0" 200 1235 1559 https://grafana.sysadmin222.top/alerting/list Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:124.0) Gecko/20100101 Firefox/124.0 [10/Apr/2024:16:53:24 +0200] grafana.sysadmin222.top 172.68.243.31 0.005 "GET /api/user HTTP/2.0" 200 295 619 https://grafana.sysadmin222.top/alerting/list Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:124.0) Gecko/20100101 Firefox/124.0 [10/Apr/2024:16:53:24 +0200] grafana.sysadmin222.top 172.68.243.31 0.007 "GET /api/prometheus/grafana/api/v1/rules?limit_alerts=16 HTTP/2.0" 200 788 1112 https://grafana.sysadmin222.top/alerting/list Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:124.0) Gecko/20100101 Firefox/124.0 [10/Apr/2024:16:53:24 +0200] grafana.sysadmin222.top 172.68.243.31 0.005 "GET /api/ruler/grafana/api/v1/rules?subtype=cortex HTTP/2.0" 200 1177 1501 https://grafana.sysadmin222.top/alerting/list Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:124.0) Gecko/20100101 Firefox/124.0 [10/Apr/2024:16:53:25 +0200] grafana.sysadmin222.top 172.68.243.31 0.003 "GET /api/prometheus/a111cd43-df3e-434c-a508-b9861bde17d4/api/v1/rules HTTP/2.0" 200 45 369 https://grafana.sysadmin222.top/alerting/list Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:124.0) Gecko/20100101 Firefox/124.0 [10/Apr/2024:16:53:27 +0200] grafana.sysadmin222.top 162.158.134.229 64.681 "GET /api/live/ws HTTP/1.1" 101 108 562 - Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:124.0) Gecko/20100101 Firefox/124.0

[tokar86a]

That should give you above information in the access log file "access.log"

[smilebasti]

Thank you very much. That helped :).
Supporting npmPlus will require a bit of rewriting but won't be too difficult.
As i am not able to create a fork of my own project i will develop npmPlusGrafStats inside a new branch.

I will keep you updated.

[tokar86a]

Thank you very much. That helped :). Supporting npmPlus will require a bit of rewriting but won't be too difficult. As i am not able to create a fork of my own project i will develop npmPlusGrafStats inside a new branch.

I will keep you updated.

Maybe ask the developer about what going to change when it comes to logging. ZoeyVid/NPMplus#731

Seems like she is going to rework it into php. So maybe best to wait a little so you don't need to redo the work again?

[smilebasti]

Thanks for the notice. As it will stay nginx as proxy for now, I don't see a problem.

[smilebasti]

I have yesterday pushed the plus version: https://hub.docker.com/r/smilebasti/npmgrafstats/tags.

Would be great if you could test it.

Here is the docker-compose.yml file. You can leave out npmplus as you already got a running instance.

version: '3.8'
services:
  npmplus:
    container_name: npmplus
    image: zoeyvid/npmplus
    restart: always
    network_mode: host
    volumes:
      - "/opt/npm:/data"
    environment:
      - "TZ=Europe/Berlin" # set timezone, required
      - "LOGROTATE=true" # required for npmgrafstats
      # for more see https://github.com/ZoeyVid/NPMplus/blob/develop/compose.yaml

  npmgraf:
    image: smilebasti/npmgrafstats:plus-latest # change to ghcr.io/smilebasti/npmgrafstats if you prefer Github Container Registry
    restart: unless-stopped
    environment:
      INTERNAL_LOGS: '<set TRUE or FALSE>' # see Github wiki for more information
      MONITORING_LOGS: '<set TRUE or FALSE>' # see Github wiki for more information
      INFLUX_HOST: '<replace>:8086' # use host IP
      INFLUX_BUCKET: 'npmgrafstats'
      INFLUX_ORG: 'npmgrafstats'
      INFLUX_TOKEN: '<replace>' # insert after first run and manual token creation
    volumes:
      - /opt/npm/nginx/access.log:/access.log
      - ./geolite:/geolite
      - ./monitoringips.txt:/monitoringips.txt # optional only mount if preexists and a wanted feature
    depends_on:
      - geoipupdate
      - npmplus

  influxdb:
    image: influxdb:2.7-alpine
    restart: unless-stopped
    environment:
      DOCKER_INFLUXDB_INIT_MODE: 'setup'
      DOCKER_INFLUXDB_INIT_USERNAME: '<replace>'
      DOCKER_INFLUXDB_INIT_PASSWORD: '<replace>' # atleast 8 characters
      DOCKER_INFLUXDB_INIT_ORG: 'npmgrafstats'
      DOCKER_INFLUXDB_INIT_BUCKET: 'npmgrafstats'
    volumes:
      # Mount for influxdb data directory and configuration
      - ./influxdbv2:/var/lib/influxdb2
    ports:
      - '8086:8086'

  geoipupdate:
    image: maxmindinc/geoipupdate
    restart: unless-stopped
    environment:
      GEOIPUPDATE_ACCOUNT_ID: '<replace>'
      GEOIPUPDATE_LICENSE_KEY: '<replace>'
      GEOIPUPDATE_EDITION_IDS: 'GeoLite2-City  GeoLite2-ASN'  #GeoLite2-ASN is optional
      GEOIPUPDATE_FREQUENCY: 24
    volumes:
      - ./geolite:/usr/share/GeoIP

  portainer:
    image: portainer/portainer-ce:latest
    restart: always
    ports: 
      - '8000:8000'
      - '9000:9000'
      - '9443:9443'
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./portainer_data:/data 

  grafana:
    image: grafana/grafana-oss
    restart: unless-stopped
    ports:
      - '3000:3000'
    volumes:
      - ./grafana-storage:/var/lib/grafana # after start: sudo chown 472:472 grafana-storage 

[tokar86a]

I am leaving out most of the part in this and only use following

version: "3.8"
services:
npmgraf:
image: smilebasti/npmgrafstats:plus-latest # set tag to 'latest-arm' for arm/v7 and arm64 image
container_name: npmstats
restart: unless-stopped
environment:
REDIRECTION_LOGS: "TRUE" # TRUE or FALSE or ONLY
INTERNAL_LOGS: "FALSE" # see Github wiki for more information
MONITORING_LOGS: "FALSE" # see Github wiki for more information
INFLUX_HOST: # use host IP
INFLUX_BUCKET: npmgrafstats
INFLUX_ORG: myorg
INFLUX_TOKEN: # insert after first run and manual token creation
volumes:
- /opt/stacks/npmplus/npm/nginx/access.log:/access.log
- /opt/stacks/npmplus/npm/etc/goaccess/geoip:/geolite
# - ./monitoringips.txt:/monitoringips.txt # optional only mount if preexists and a wanted feature

networks: {}

But it seems to be working good.

[tokar86a]

I have notes something now. When the log rotation happen it seems like i need to shutdown the container and start it backup again for it to work. So it seems to have trouble when the new file is being created. Something you have notes?

[smilebasti]

Ok interesting. It should continue to work as tail -F follows the filename and retries if not found.

I will have a look into it.
The branch is now available: https://github.com/smilebasti/npmGrafStats/tree/npmPlus-main

[tokar86a]

Ok interesting. It should continue to work as tail -F follows the filename and retries if not found.

I will have a look into it. The branch is now available: https://github.com/smilebasti/npmGrafStats/tree/npmPlus-main

yes it sure seems strange. But have you check that you end up whit the same problem?

[smilebasti]

Haven't but will do.

[tokar86a]

Try again today and ended up whit the same problem.

[smilebasti]

I noticed the same thing. Will take a few days to test different solutions as deleting the file makes it work :)

[gadget78]

are the log rotate issues you are having, due to them being disabled by default
Note: access.log/stream.log, logrotate and goaccess are NOT enabled by default because of GDPR, you can enable them in the compose.yaml.

[tokar86a]

are the log rotate issues you are having, due to them being disabled by default Note: access.log/stream.log, logrotate and goaccess are NOT enabled by default because of GDPR, you can enable them in the compose.yaml.

What you mean? Have you not check my compose file?

#41 (comment)

[tokar86a]

Any update about this?

[tokar86a]

Sadly not. I have tried a few options of tail with -F or just -f or similar, but none of them worked. Also tried setting the log rotation to 0.

I currently don't have the time to test other options or rewrite the whole thing. If you want to try it, just download the scripts and test variations or if you have experience you could rewrite the code to help.

Thanks for the update and sadly not i don`t have any experience when it comes to this. It sure is strange as it working whit vanila NPM so it should not be allot different then.