GoAccess dont get npm logs?

[xavier-hernandez]

GoAccess dont get npm logs?

[xavier-hernandez]

I recommend you do "docker container logs Goaccess" it should tell you if the docker container can read your logs files. It could be a permission issue..

[zeus1999]

it says

No archived logs found...
Checking active logs...
 [SETTING UP STORAGE /goaccess/access.log] {0} @ {0/s}
WebSocket server ready to accept new client connections

thats strange. there is an large default.log file

[xavier-hernandez]

It doesn't currently process the default.log. It's on the list to do since it's a different log format.

It should be listing all the files it found. Something like below. Are your logs stored at "/npm/data/logs" or "/root/npm/data/logs" ? I noticed that in your first comment. Please verify the location of the logs.

Loading (94) archived logs...
Checking active logs...
Filename: /opt/log/proxy-host-6_access.log | Read = yes
Filename: /opt/log/proxy-host-24_access.log | Read = yes
Filename: /opt/log/proxy-host-9_access.log | Read = yes
Filename: /opt/log/proxy-host-10_access.log | Read = yes
Filename: /opt/log/proxy-host-14_access.log | Read = yes
Filename: /opt/log/proxy-host-17_access.log | Read = yes
Filename: /opt/log/proxy-host-21_access.log | Read = yes
Filename: /opt/log/proxy-host-18_access.log | Read = yes
Filename: /opt/log/proxy-host-26_access.log | Read = yes
Filename: /opt/log/proxy-host-16_access.log | Read = yes
Filename: /opt/log/proxy-host-25_access.log | Read = yes
Filename: /opt/log/proxy-host-15_access.log | Read = yes
Filename: /opt/log/proxy-host-19_access.log | Read = yes
Filename: /opt/log/proxy-host-4_access.log | Read = yes
Filename: /opt/log/proxy-host-5_access.log | Read = yes
Filename: /opt/log/proxy-host-1_access.log | Read = yes
Filename: /opt/log/proxy-host-12_access.log | Read = yes
Filename: /opt/log/proxy-host-11_access.log | Read = yes
Filename: /opt/log/proxy-host-27_access.log | Read = yes
Filename: /opt/log/proxy-host-20_access.log | Read = yes
Filename: /opt/log/proxy-host-23_access.log | Read = yes
Filename: /opt/log/proxy-host-2_access.log | Read = yes
Filename: /opt/log/proxy-host-8_access.log | Read = yes
Filename: /opt/log/proxy-host-7_access.log | Read = yes
Filename: /opt/log/proxy-host-13_access.log | Read = yes
Filename: /opt/log/proxy-host-22_access.log | Read = yes
Filename: /opt/log/proxy-host-3_access.log | Read = yes
Loading proxy-host logs...
[PARSING /opt/log/proxy-host-26_access.log] {39} @ {0/s}0/s}
WebSocket server ready to accept new client connections

[zeus1999]

It doesn't currently process the default.log. It's on the list to do since it's a different log format.

It should be listing all the files it found. Something like below. Are your logs stored at "/npm/data/logs" or "/root/npm/data/logs" ? I noticed that in your first comment. Please verify the location of the logs.

Loading (94) archived logs... Checking active logs... Filename: /opt/log/proxy-host-6_access.log | Read = yes Filename: /opt/log/proxy-host-24_access.log | Read = yes Filename: /opt/log/proxy-host-9_access.log | Read = yes Filename: /opt/log/proxy-host-10_access.log | Read = yes Filename: /opt/log/proxy-host-14_access.log | Read = yes Filename: /opt/log/proxy-host-17_access.log | Read = yes Filename: /opt/log/proxy-host-21_access.log | Read = yes Filename: /opt/log/proxy-host-18_access.log | Read = yes Filename: /opt/log/proxy-host-26_access.log | Read = yes Filename: /opt/log/proxy-host-16_access.log | Read = yes Filename: /opt/log/proxy-host-25_access.log | Read = yes Filename: /opt/log/proxy-host-15_access.log | Read = yes Filename: /opt/log/proxy-host-19_access.log | Read = yes Filename: /opt/log/proxy-host-4_access.log | Read = yes Filename: /opt/log/proxy-host-5_access.log | Read = yes Filename: /opt/log/proxy-host-1_access.log | Read = yes Filename: /opt/log/proxy-host-12_access.log | Read = yes Filename: /opt/log/proxy-host-11_access.log | Read = yes Filename: /opt/log/proxy-host-27_access.log | Read = yes Filename: /opt/log/proxy-host-20_access.log | Read = yes Filename: /opt/log/proxy-host-23_access.log | Read = yes Filename: /opt/log/proxy-host-2_access.log | Read = yes Filename: /opt/log/proxy-host-8_access.log | Read = yes Filename: /opt/log/proxy-host-7_access.log | Read = yes Filename: /opt/log/proxy-host-13_access.log | Read = yes Filename: /opt/log/proxy-host-22_access.log | Read = yes Filename: /opt/log/proxy-host-3_access.log | Read = yes Loading proxy-host logs... [PARSING /opt/log/proxy-host-26_access.log] {39} @ {0/s}0/s} WebSocket server ready to accept new client connections

verified. pwd is /root/npm/data/logs

[xavier-hernandez]

It's not an ARM device I assume?
What OS?

[zeus1999]

It's not an ARM device I assume? What's OS?

Its an LXC (Ubuntu 21.04) on an Proxmox server

[rtorchia]

@zeus1999
I could be reading to much into your replies, but check your path to your log files.
You replied that your logs are at /npm/data/logs, but your script has /root/npm/data/logs
If so, then you should have:

-v /npm/data/logs:/opt/log

[zeus1999]

i triple checked it now. the logs are here: /root/npm/data/logs

[rtorchia]

That good to know.
I had problems with goaccess not readying my log files until I deleted everything in the logs directory and started fresh.
Looking at your directory, there is a strange file

'og-format=%h %^[%d:%t %^] %v "%r" "%s" %b "%R" "%u" %^'

Wonder if you are having something similar to what I experienced

[zeus1999]

That good to know. I had problems with goaccess not readying my log files until I deleted everything in the logs directory and started fresh. Looking at your directory, there is a strange file

'og-format=%h %^[%d:%t %^] %v "%r" "%s" %b "%R" "%u" %^'

Wonder if you are having something similar to what I experienced

Very Strange, i removed all log files restartet the system, created the container, and same problem....

[xavier-hernandez]

@zeus1999 I see the issue. I didn't realize it before but your filenames are different. The app reads "proxy-host-*_access.log" files. Your file is called proxy_host-37.log.

For example...
proxy-host-8_access.log
proxy-host-8_error.log

What version of NPM are you running?

[zeus1999]

@zeus1999 I see the issue. I didn't realize it before but your filenames are different. The app reads "proxy-host-*_access.log" files. Your file is called proxy_host-37.log.

For example... proxy-host-8_access.log proxy-host-8_error.log

What version of NPM are you running?

according to the webui> v2.8.1

[xavier-hernandez]

The latest is 2.9.16. Before we have you updating NPM or anything like that. Would it be possible to send me a sample log file? I'd like to check the log file and see if it matches the current log file structure.

[zeus1999]

The latest is 2.9.16. Before we have you updating NPM or anything like that. Would it be possible to send me a sample log file? I'd like to check the log file and see if it matches the current log file structure.

Sure, i would delete the link, after you have downloaded it.

https://pastebin.com/xxxxxxxxxxxxxxxxxxx

[xavier-hernandez]

Got it, feel free to delete.

[xavier-hernandez]

@zeus1999 I think you sent me a redirection log is that correct? I need a proxy_host-*.log. It just needs to be a few lines as well.

[zeus1999]

@zeus1999 I think you sent me a redirection log is that correct? I need a proxy_host-*.log. It just needs to be a few lines as well.

Sorry! My fault. Here you go: https://pastebin.com/yyyyyyyyyyyyyyyyy

[xavier-hernandez]

Got it!

[xavier-hernandez]

@zeus1999

I've made a mod to the script that looks for the proxy files.

"proxy-host-.log" except anything containing "_error.log" which will exclude the error logs for now.

Can you pull down the dev version and see if it works for you? xavierh/goaccess-for-nginxproxymanager:develop

[xavier-hernandez]

I still can't find documentation of when this changed. I've searched the NPM repo. I do see examples of log files names in comments from 2019 that differ from comments made in 2021/2022.

¯_(ツ)_/¯

[zeus1999]

@zeus1999

I've made a mod to the script that looks for the proxy files.

"proxy-host-_.log" except anything containing "__error.log" which will exclude the error logs for now.

Can you pull down the dev version and see if it works for you? xavierh/goaccess-for-nginxproxymanager:develop

Thanks!

But it doesnt work either. Same problem, same output.

I created the container

docker create -it \
--name=Goaccess \
--restart=always \
-p 7880:7880 \
-v /root/npm/data/logs:/opt/log \
-e SKIP_ARCHIVED_LOGS=False \
xavierh/goaccess-for-nginxproxymanager:develop

[xavier-hernandez]

can you send over the logs again? docker container logs Goaccess

then can you go into the docker container itself?

• docker exec -it Goaccess /bin/bash
• navigate to /opt/log
• do a ls command and send over the output
• then "exit"

[zeus1999]

can you send over the logs again? docker container logs Goaccess

then can you go into the docker container itself?

• docker exec -it Goaccess /bin/bash
• navigate to /opt/log
• do a ls command and send over the output
• then "exit"

docker container logs Goaccess

No archived logs found...
Checking active logs...
 [SETTING UP STORAGE /goaccess/access.log] {0} @ {0/s}
WebSocket server ready to accept new client connections

ls

default.log                                             proxy_host-23.log                                       proxy_host-37.log                                       proxy_host-51.log
default_host.log                                        proxy_host-24.log                                       proxy_host-38.log                                       proxy_host-52.log
error.log                                               proxy_host-25.log                                       proxy_host-39.log                                       proxy_host-53.log
letsencrypt-requests.log                                proxy_host-26.log                                       proxy_host-40.log                                       proxy_host-54.log
og-format=%h %^[%d:%t %^] %v "%r" "%s" %b "%R" "%u" %^  proxy_host-27.log                                       proxy_host-41.log                                       proxy_host-55.log
proxy_host-1.log                                        proxy_host-28.log                                       proxy_host-42.log                                       proxy_host-56.log
proxy_host-12.log                                       proxy_host-29.log                                       proxy_host-43.log                                       proxy_host-57.log
proxy_host-16.log                                       proxy_host-3.log                                        proxy_host-44.log                                       proxy_host-58.log
proxy_host-17.log                                       proxy_host-30.log                                       proxy_host-45.log                                       proxy_host-59.log
proxy_host-18.log                                       proxy_host-31.log                                       proxy_host-46.log                                       proxy_host-6.log
proxy_host-19.log                                       proxy_host-32.log                                       proxy_host-47.log                                       proxy_host-60.log
proxy_host-2.log                                        proxy_host-33.log                                       proxy_host-48.log                                       proxy_host-7.log
proxy_host-20.log                                       proxy_host-34.log                                       proxy_host-49.log                                       redirection_host-1.log
proxy_host-21.log                                       proxy_host-35.log                                       proxy_host-5.log                                        redirection_host-3.log
proxy_host-22.log                                       proxy_host-36.log                                       proxy_host-50.log                                       redirection_host-4.log

[xavier-hernandez]

@zeus1999

I can confirm it works on my end using a file you sent me yesterday with the filename you are using.

If you used the develop tag in the past maybe do "docker pull xavierh/goaccess-for-nginxproxymanager:develop" to get the latest.

docker run --name=Goaccess --restart=always -p 7880:7880 -v /opt/sample-logs-develop:/opt/log -e SKIP_ARCHIVED_LOGS=False xavierh/goaccess-for-nginxproxymanager:develop

/opt/sample-logs-develop contains one log file called proxy-host-2.log

No archived logs found...
Checking active logs...
Filename: /opt/log/proxy-host-2.log | Read = yes
Loading proxy-host logs...
[SETTING UP STORAGE /opt/log/proxy-host-2.log] {0} @ {0/s}
WebSocket server ready to accept new client connections

[xavier-hernandez]

Other than a couple of file names I don't believe there is any personal information in the screenshot. Let me know if you want me to remove it.

Also, can you try going back in the container and do a "cat proxy_host-1.log" and check if any data is displayed. It should detect file permissions but maybe its not. Just want to verify that and cover all the basics.

Thanks.

[zeus1999]

ry going back in the container and do a "cat proxy_host-1.log" and check if a

Okay, i created an test folder with an proxy file (proxy_host-1.log) and gave them chmod 777. when i go in the docker shell for the container i can see the file content (cat proxy_host-1.log).

but same result.

i will try some things.

(i repulled the image)

[buzzke]

Seems like a mismatch between the Arm and Amd version.
Had the same issue on the Arm docker image (jc21/nginx-proxy-manager:latest)
As a test i deployed the same image on a Amd machine. And the logs are indeed in different format.

-rwxrwxrwx+ 1 root root   3304 Mar 25 11:26 fallback_access.log
-rwxrwxrwx+ 1 root root    713 Mar 25 11:04 fallback_error.log
-rwxrwxrwx+ 1 root root   5592 Mar 25 11:23 letsencrypt-requests_access.log
-rwxrwxrwx+ 1 root root      0 Mar 25 10:41 letsencrypt-requests_error.log
-rwxrwxrwx+ 1 root root 143055 Mar 25 10:57 proxy-host-1_access.log
-rwxrwxrwx+ 1 root root   4289 Mar 25 10:45 proxy-host-1_error.log
-rwxrwxrwx+ 1 root root  80728 Mar 25 11:12 proxy-host-2_access.log
-rwxrwxrwx+ 1 root root      0 Mar 25 10:44 proxy-host-2_error.log
-rwxrwxrwx+ 1 root root  98921 Mar 25 11:29 proxy-host-3_access.log
-rwxrwxrwx+ 1 root root   1427 Mar 25 11:28 proxy-host-3_error.log
-rwxrwxrwx+ 1 root root  92033 Mar 25 11:24 proxy-host-4_access.log
-rwxrwxrwx+ 1 root root   2716 Mar 25 10:50 proxy-host-4_error.log
-rwxrwxrwx+ 1 root root   5560 Mar 25 11:31 proxy-host-5_access.log
-rwxrwxrwx+ 1 root root    215 Mar 25 11:24 proxy-host-5_error.log

After this the Docker image reads the log files

Setting up basic auth in NGINX...
Adding password for user docker
No archived logs found...
Checking active logs...
Filename: /opt/log/proxy-host-1_access.log | Read = yes
Filename: /opt/log/proxy-host-2_access.log | Read = yes
Filename: /opt/log/proxy-host-3_access.log | Read = yes
Filename: /opt/log/proxy-host-4_access.log | Read = yes
Loading proxy-host logs...
 [PARSING /opt/log/proxy-host-3_access.log] {55} @ {0/s}0} @ {0/s}
WebSocket server ready to accept new client connections

[zeus1999]

Yeah I see, i could get this to run. Always stays by No archived logs found

[xavier-hernandez]

@zeus1999 it started working for you overall, just no archives found?

Archived/old data has an extension of .gz at least in my case. Do you see anything like that on your NPM version?

[zeus1999]

@zeus1999 it started working for you overall, just no archives found?

Archived/old data has an extension of .gz at least in my case. Do you see anything like that on your NPM version?

No not working. There is any compressed file. Only proxy_host-x.log and some other log files. See here: https://pastebin.com/ugLzuMxn

[xavier-hernandez]

@zeus1999 since there are no .gz files then it'll always tell you it didn't find any archive files.

Have you tried adding the following, looks your log files are created another root?

environment:
    - PUID=0
    - PGID=0

[xavier-hernandez]

@zeus1999 also please pull the latest from the develop image. This is what your docker-compose should look like. I've tried plugging everything as part of your post. You'll need to set the right timezone.

goaccess:
    image: xavierh/goaccess-for-nginxproxymanager:develop
    container_name: goaccess
    restart: always
    volumes:
        - /root/npm/data/logs:/opt/log
    ports:
        - '7880:7880'
    environment:
        - PUID=0
        - PGID=0
        - TZ=America/New_York        
        - SKIP_ARCHIVED_LOGS=False #optional
        - BASIC_AUTH=False #optional
        - BASIC_AUTH_USERNAME=user #optional
        - BASIC_AUTH_PASSWORD=pass #optional      

[zeus1999]

@zeus1999 also please pull the latest from the develop image. This is what your docker-compose should look like. I've tried plugging everything as part of your post. You'll need to set the right timezone.

goaccess:
    image: xavierh/goaccess-for-nginxproxymanager:develop
    container_name: goaccess
    restart: always
    volumes:
        - /root/npm/data/logs:/opt/log
    ports:
        - '7880:7880'
    environment:
        - PUID=0
        - PGID=0
        - TZ=America/New_York        
        - SKIP_ARCHIVED_LOGS=False #optional
        - BASIC_AUTH=False #optional
        - BASIC_AUTH_USERNAME=user #optional
        - BASIC_AUTH_PASSWORD=pass #optional      

i created the docker compose exactly from your paste:

this output:

root@Service-nginx:~/test# docker-compose up
Creating goaccess ... done
Attaching to goaccess
goaccess    | No archived logs found...
goaccess    | Checking active logs...
goaccess    |  [SETTING UP STORAGE /goaccess/access.log] {0} @ {0/s}
WebSocket server ready to accept new client connections

i entered the shell from the docker container and can see the logs and also read them.

but same result? what am i doing wrong?

[xavier-hernandez]

@zeus1999

That looks like an old image, I added a version number recently.

run "docker pull image xavierh/goaccess-for-nginxproxymanager:develop" then run "docker-compose up" again

[zeus1999]

true

new log

Recreating goaccess ... done
Attaching to goaccess
goaccess    | 
goaccess    | GOAN v1.0.3
goaccess    | 
goaccess    | No archived logs found at /opt/log...
goaccess    | Checking active logs...
goaccess    | 
goaccess    | Processing (0) total log(s)...
goaccess    | 
goaccess    | 
WebSocket server ready to accept new client connections

[zeus1999]

but it doesnt work.... i am clueless

[xavier-hernandez]

A new version has been pushed out. Try that one.

docker pull xavierh/goaccess-for-nginxproxymanager:develop