kfirfer/helm

Deprecation notice !!!!

Moved to: https://github.com/kfirfer/charts

helm's Issues

PMM Agents will not connect

hi @kfirfer I'm using your PMM helm chart and it installs properly, I get the GUI to load but I can't seem to get agents to connect. Have you been successful at getting agents to connect properly?

I'm using these commands:

pmm-admin config --server pmm.url.com --server-password admin --server-user admin --server-ssl
pmm-admin config --server pmm.url.com --server-password admin --server-user admin --server-insecure-ssl

All or any variation is telling me that it can't resolve but loads properly in my browser. any ideas?

Failed to start PMM server

Hi @kfirfer , I'm facing an error when spawning this pmm-server chart, even with default values,, pod logs:

WARN[2022-05-24T09:07:16.130+00:00] Configuration warning: unknown environment variable "MY_PMM_SERVER_PORT_80_TCP=tcp://172.20.90.86:80". 
WARN[2022-05-24T09:07:16.130+00:00] Configuration warning: unknown environment variable "MY_PMM_SERVER_SERVICE_PORT=4431". 
WARN[2022-05-24T09:07:16.130+00:00] Configuration warning: unknown environment variable "MY_PMM_SERVER_PORT=tcp://172.20.90.86:4431". 
WARN[2022-05-24T09:07:16.130+00:00] Configuration warning: unknown environment variable "SERVER_USER=admin". 
WARN[2022-05-24T09:07:16.130+00:00] Configuration warning: unknown environment variable "KUBERNETES_PORT=tcp://172.20.0.1:443". 
WARN[2022-05-24T09:07:16.130+00:00] Configuration warning: unknown environment variable "KUBERNETES_PORT_443_TCP_PORT=443". 
WARN[2022-05-24T09:07:16.131+00:00] Configuration warning: unknown environment variable "MY_PMM_SERVER_PORT_80_TCP_PORT=80". 
WARN[2022-05-24T09:07:16.131+00:00] Configuration warning: unknown environment variable "KUBERNETES_SERVICE_PORT=443". 
WARN[2022-05-24T09:07:16.131+00:00] Configuration warning: unknown environment variable "KUBERNETES_SERVICE_HOST=172.20.0.1". 
WARN[2022-05-24T09:07:16.131+00:00] Configuration warning: unknown environment variable "MY_PMM_SERVER_PORT_4431_TCP_PORT=4431". 
WARN[2022-05-24T09:07:16.131+00:00] Configuration warning: unknown environment variable "MY_PMM_SERVER_PORT_80_TCP_PROTO=tcp". 
WARN[2022-05-24T09:07:16.131+00:00] Configuration warning: unknown environment variable "MY_PMM_SERVER_PORT_4431_TCP_ADDR=172.20.90.86". 
WARN[2022-05-24T09:07:16.131+00:00] Configuration warning: unknown environment variable "MY_PMM_SERVER_SERVICE_PORT_HTTPS=4431". 
WARN[2022-05-24T09:07:16.131+00:00] Configuration warning: unknown environment variable "KUBERNETES_PORT_443_TCP_PROTO=tcp". 
WARN[2022-05-24T09:07:16.131+00:00] Configuration warning: unknown environment variable "KUBERNETES_SERVICE_PORT_HTTPS=443". 
WARN[2022-05-24T09:07:16.131+00:00] Configuration warning: unknown environment variable "MY_PMM_SERVER_PORT_4431_TCP=tcp://172.20.90.86:4431". 
WARN[2022-05-24T09:07:16.131+00:00] Configuration warning: unknown environment variable "KUBERNETES_PORT_443_TCP_ADDR=172.20.0.1". 
WARN[2022-05-24T09:07:16.131+00:00] Configuration warning: unknown environment variable "MY_PMM_SERVER_PORT_4431_TCP_PROTO=tcp". 
WARN[2022-05-24T09:07:16.132+00:00] Configuration warning: unknown environment variable "MY_PMM_SERVER_SERVICE_PORT_HTTP=80". 
WARN[2022-05-24T09:07:16.132+00:00] Configuration warning: unknown environment variable "KUBERNETES_PORT_443_TCP=tcp://172.20.0.1:443". 
WARN[2022-05-24T09:07:16.132+00:00] Configuration warning: unknown environment variable "MY_PMM_SERVER_PORT_80_TCP_ADDR=172.20.90.86". 
WARN[2022-05-24T09:07:16.132+00:00] Configuration warning: unknown environment variable "MY_PMM_SERVER_SERVICE_HOST=172.20.90.86". 
WARN[2022-05-24T09:07:16.132+00:00] Configuration warning: unknown environment variable "SERVER_PASSWORD=a123123". 
2022-05-24 09:07:16,228 INFO Included extra file "/etc/supervisord.d/grafana.ini" during parsing
2022-05-24 09:07:16,228 INFO Included extra file "/etc/supervisord.d/pmm.ini" during parsing
2022-05-24 09:07:16,228 INFO Set uid to user 0 succeeded
2022-05-24 09:07:16,236 INFO RPC interface 'supervisor' initialized
2022-05-24 09:07:16,236 INFO supervisord started with pid 1
2022-05-24 09:07:17,239 INFO spawned: 'pmm-update-perform-init' with pid 13
2022-05-24 09:07:17,241 INFO spawned: 'postgresql' with pid 14
2022-05-24 09:07:17,243 INFO spawned: 'clickhouse' with pid 15
2022-05-24 09:07:17,245 INFO spawned: 'grafana' with pid 17
2022-05-24 09:07:17,247 INFO spawned: 'nginx' with pid 20
2022-05-24 09:07:17,249 INFO spawned: 'cron' with pid 21
2022-05-24 09:07:17,251 INFO spawned: 'pmm-managed' with pid 22
2022-05-24 09:07:17,269 INFO spawned: 'pmm-agent' with pid 25
2022-05-24 09:07:17,330 INFO exited: postgresql (exit status 1; not expected)
2022-05-24 09:07:17,445 INFO exited: grafana (exit status 1; not expected)
2022-05-24 09:07:18,407 INFO success: pmm-update-perform-init entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-05-24 09:07:18,408 INFO spawned: 'postgresql' with pid 109
2022-05-24 09:07:18,409 INFO success: clickhouse entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-05-24 09:07:18,409 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-05-24 09:07:18,409 INFO success: cron entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-05-24 09:07:18,409 INFO success: pmm-managed entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-05-24 09:07:18,409 INFO success: pmm-agent entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-05-24 09:07:18,437 INFO exited: postgresql (exit status 1; not expected)
2022-05-24 09:07:18,523 INFO spawned: 'grafana' with pid 110
2022-05-24 09:07:18,648 INFO exited: grafana (exit status 1; not expected)
2022-05-24 09:07:20,472 INFO spawned: 'postgresql' with pid 128
2022-05-24 09:07:20,622 INFO exited: postgresql (exit status 1; not expected)
2022-05-24 09:07:21,490 INFO spawned: 'grafana' with pid 136
2022-05-24 09:07:21,770 INFO exited: grafana (exit status 1; not expected)
2022-05-24 09:07:24,496 INFO spawned: 'postgresql' with pid 146
2022-05-24 09:07:24,587 INFO exited: postgresql (exit status 1; not expected)
2022-05-24 09:07:25,493 INFO spawned: 'grafana' with pid 151
2022-05-24 09:07:25,730 INFO exited: grafana (exit status 1; not expected)
2022-05-24 09:07:29,496 INFO spawned: 'postgresql' with pid 165
2022-05-24 09:07:29,539 INFO exited: postgresql (exit status 1; not expected)
2022-05-24 09:07:30,500 INFO spawned: 'grafana' with pid 167
2022-05-24 09:07:30,746 INFO exited: grafana (exit status 1; not expected)
2022-05-24 09:07:34,605 INFO spawned: 'postgresql' with pid 341
2022-05-24 09:07:34,632 INFO exited: postgresql (exit status 1; not expected)
2022-05-24 09:07:35,821 INFO spawned: 'grafana' with pid 475
2022-05-24 09:07:35,902 INFO exited: grafana (exit status 1; not expected)
2022-05-24 09:07:40,652 INFO spawned: 'postgresql' with pid 552
2022-05-24 09:07:40,699 INFO exited: postgresql (exit status 1; not expected)
2022-05-24 09:07:42,065 INFO spawned: 'grafana' with pid 578
2022-05-24 09:07:42,326 INFO exited: grafana (exit status 1; not expected)
2022-05-24 09:07:43,711 INFO spawned: 'grafana' with pid 590
2022-05-24 09:07:43,780 INFO exited: grafana (exit status 1; not expected)
2022-05-24 09:07:44,609 INFO exited: pmm-update-perform-init (exit status 1; not expected)
2022-05-24 09:07:45,521 INFO spawned: 'pmm-update-perform-init' with pid 602
2022-05-24 09:07:46,521 INFO success: pmm-update-perform-init entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-05-24 09:07:48,300 INFO spawned: 'postgresql' with pid 640
2022-05-24 09:07:48,334 INFO exited: postgresql (exit status 1; not expected)

Post more logs below from the pod.
pmm-managed.log

INFO[2022-05-24T09:59:22.639+00:00] Migrating database...                         component=migration
WARN[2022-05-24T09:59:22.639+00:00] Failed to migrate database: dial tcp 127.0.0.1:5432: connect: connection refused.  component=migration
INFO[2022-05-24T09:59:23.640+00:00] Migrating database...                         component=migration
WARN[2022-05-24T09:59:23.640+00:00] Failed to migrate database: dial tcp 127.0.0.1:5432: connect: connection refused.  component=migration
INFO[2022-05-24T09:59:24.640+00:00] Migrating database...                         component=migration
WARN[2022-05-24T09:59:24.641+00:00] Failed to migrate database: dial tcp 127.0.0.1:5432: connect: connection refused.  component=migration
INFO[2022-05-24T09:59:25.641+00:00] Migrating database...                         component=migration
WARN[2022-05-24T09:59:25.641+00:00] Failed to migrate database: dial tcp 127.0.0.1:5432: connect: connection refused.  component=migration
INFO[2022-05-24T09:59:26.642+00:00] Migrating database...                         component=migration
WARN[2022-05-24T09:59:26.643+00:00] Failed to migrate database: dial tcp 127.0.0.1:5432: connect: connection refused.  component=migration

pmm-update-perform-init.log

PLAY RECAP *********************************************************************
localhost                  : ok=11   changed=2    unreachable=0    failed=1    skipped=10   rescued=0    ignored=1   

time="2022-05-24T10:06:08Z" level=fatal msg="RunPlaybook failed: exit status 2"
ProjectName: pmm-update
Version: 2.27.0
PMMVersion: 2.27.0
Timestamp: 2022-04-12 11:11:07 (UTC)
FullCommit: 3112681ff16267dad7b883601869a8ea7a6381be
Starting "ansible-playbook --flush-cache /usr/share/pmm-update/ansible/playbook/tasks/init.yml" ...
[root@my-pmm-server-0 logs]# tail pmm-update-perform-init.log -n 100

TASK [Gathering Facts] *********************************************************
ok: [localhost]

TASK [Run initializaion role] **************************************************

TASK [initialization : Get current version] ************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "file not found: /srv/grafana/PERCONA_DASHBOARDS_VERSION"}
...ignoring

TASK [initialization : Get image version] **************************************
ok: [localhost]

TASK [initialization : Set current version if VERSION fail doen't exist] *******
ok: [localhost]

TASK [initialization : Setting current and image version] **********************
skipping: [localhost]

TASK [initialization : Setting current and image version] **********************
ok: [localhost]

TASK [initialization : Print current version] **********************************
ok: [localhost] => {
    "msg": "Current version: 0.0 Image Version: 2.27.0"
}

TASK [Checking if we need an update or not] ************************************

TASK [dashboards_upgrade : Check that old plugin dir exists] *******************
ok: [localhost]

TASK [dashboards_upgrade : Find custom plugin in old plugin dir] ***************
skipping: [localhost]

TASK [dashboards_upgrade : Synchronization plugins (old version before 2.22.0)] ***
skipping: [localhost]

TASK [dashboards_upgrade : Find custom plugin in old plugin dir] ***************
skipping: [localhost]

TASK [dashboards_upgrade : Get plugin list] ************************************
ok: [localhost]

TASK [dashboards_upgrade : Delete existing dist folder] ************************
changed: [localhost] => (item={u'rusr': True, u'uid': 0, u'rgrp': True, u'xoth': True, u'islnk': False, u'woth': False, u'nlink': 3, u'issock': False, u'mtime': 1649869342.0, u'gr_name': u'root', u'path': u'/usr/share/percona-dashboards/panels/camptocamp-prometheus-alertmanager-datasource', u'xusr': True, u'atime': 0.0, u'inode': 20405714, u'isgid': False, u'size': 146, u'isdir': True, u'ctime': 1653378131.693982, u'isblk': False, u'wgrp': False, u'xgrp': True, u'isuid': False, u'dev': 1048731, u'roth': True, u'isreg': False, u'isfifo': False, u'mode': u'0755', u'pw_name': u'root', u'gid': 0, u'ischr': False, u'wusr': True})
changed: [localhost] => (item={u'rusr': True, u'uid': 0, u'rgrp': True, u'xoth': True, u'islnk': False, u'woth': False, u'nlink': 3, u'issock': False, u'mtime': 1649869342.0, u'gr_name': u'root', u'path': u'/usr/share/percona-dashboards/panels/digiapulssi-breadcrumb-panel', u'xusr': True, u'atime': 0.0, u'inode': 22338802, u'isgid': False, u'size': 270, u'isdir': True, u'ctime': 1653378131.693982, u'isblk': False, u'wgrp': False, u'xgrp': True, u'isuid': False, u'dev': 1048731, u'roth': True, u'isreg': False, u'isfifo': False, u'mode': u'0755', u'pw_name': u'root', u'gid': 0, u'ischr': False, u'wusr': True})
changed: [localhost] => (item={u'rusr': True, u'uid': 0, u'rgrp': True, u'xoth': True, u'islnk': False, u'woth': False, u'nlink': 4, u'issock': False, u'mtime': 1649869342.0, u'gr_name': u'root', u'path': u'/usr/share/percona-dashboards/panels/grafana-piechart-panel', u'xusr': True, u'atime': 0.0, u'inode': 25521415, u'isgid': False, u'size': 277, u'isdir': True, u'ctime': 1653378131.693982, u'isblk': False, u'wgrp': False, u'xgrp': True, u'isuid': False, u'dev': 1048731, u'roth': True, u'isreg': False, u'isfifo': False, u'mode': u'0755', u'pw_name': u'root', u'gid': 0, u'ischr': False, u'wusr': True})
changed: [localhost] => (item={u'rusr': True, u'uid': 0, u'rgrp': True, u'xoth': True, u'islnk': False, u'woth': False, u'nlink': 5, u'issock': False, u'mtime': 1649869342.0, u'gr_name': u'root', u'path': u'/usr/share/percona-dashboards/panels/grafana-polystat-panel', u'xusr': True, u'atime': 0.0, u'inode': 29296395, u'isgid': False, u'size': 275, u'isdir': True, u'ctime': 1653378131.693982, u'isblk': False, u'wgrp': False, u'xgrp': True, u'isuid': False, u'dev': 1048731, u'roth': True, u'isreg': False, u'isfifo': False, u'mode': u'0755', u'pw_name': u'root', u'gid': 0, u'ischr': False, u'wusr': True})
changed: [localhost] => (item={u'rusr': True, u'uid': 0, u'rgrp': True, u'xoth': True, u'islnk': False, u'woth': False, u'nlink': 6, u'issock': False, u'mtime': 1649869342.0, u'gr_name': u'root', u'path': u'/usr/share/percona-dashboards/panels/grafana-worldmap-panel', u'xusr': True, u'atime': 0.0, u'inode': 33826614, u'isgid': False, u'size': 188, u'isdir': True, u'ctime': 1653378131.693982, u'isblk': False, u'wgrp': False, u'xgrp': True, u'isuid': False, u'dev': 1048731, u'roth': True, u'isreg': False, u'isfifo': False, u'mode': u'0755', u'pw_name': u'root', u'gid': 0, u'ischr': False, u'wusr': True})
changed: [localhost] => (item={u'rusr': True, u'uid': 0, u'rgrp': True, u'xoth': True, u'islnk': False, u'woth': False, u'nlink': 3, u'issock': False, u'mtime': 1649869342.0, u'gr_name': u'root', u'path': u'/usr/share/percona-dashboards/panels/jdbranham-diagram-panel', u'xusr': True, u'atime': 0.0, u'inode': 39284456, u'isgid': False, u'size': 175, u'isdir': True, u'ctime': 1653378131.693982, u'isblk': False, u'wgrp': False, u'xgrp': True, u'isuid': False, u'dev': 1048731, u'roth': True, u'isreg': False, u'isfifo': False, u'mode': u'0755', u'pw_name': u'root', u'gid': 0, u'ischr': False, u'wusr': True})
changed: [localhost] => (item={u'rusr': True, u'uid': 0, u'rgrp': True, u'xoth': True, u'islnk': False, u'woth': False, u'nlink': 4, u'issock': False, u'mtime': 1649869342.0, u'gr_name': u'root', u'path': u'/usr/share/percona-dashboards/panels/natel-discrete-panel', u'xusr': True, u'atime': 0.0, u'inode': 41210251, u'isgid': False, u'size': 191, u'isdir': True, u'ctime': 1653378131.693982, u'isblk': False, u'wgrp': False, u'xgrp': True, u'isuid': False, u'dev': 1048731, u'roth': True, u'isreg': False, u'isfifo': False, u'mode': u'0755', u'pw_name': u'root', u'gid': 0, u'ischr': False, u'wusr': True})
changed: [localhost] => (item={u'rusr': True, u'uid': 0, u'rgrp': True, u'xoth': True, u'islnk': False, u'woth': False, u'nlink': 7, u'issock': False, u'mtime': 1649869342.0, u'gr_name': u'root', u'path': u'/usr/share/percona-dashboards/panels/petrslavotinek-carpetplot-panel', u'xusr': True, u'atime': 0.0, u'inode': 9397258, u'isgid': False, u'size': 4096, u'isdir': True, u'ctime': 1653378131.693982, u'isblk': False, u'wgrp': False, u'xgrp': True, u'isuid': False, u'dev': 1048731, u'roth': True, u'isreg': False, u'isfifo': False, u'mode': u'0755', u'pw_name': u'root', u'gid': 0, u'ischr': False, u'wusr': True})
changed: [localhost] => (item={u'rusr': True, u'uid': 0, u'rgrp': True, u'xoth': True, u'islnk': False, u'woth': False, u'nlink': 3, u'issock': False, u'mtime': 1649869342.0, u'gr_name': u'root', u'path': u'/usr/share/percona-dashboards/panels/pmm-app', u'xusr': True, u'atime': 0.0, u'inode': 13981435, u'isgid': False, u'size': 18, u'isdir': True, u'ctime': 1653378131.693982, u'isblk': False, u'wgrp': False, u'xgrp': True, u'isuid': False, u'dev': 1048731, u'roth': True, u'isreg': False, u'isfifo': False, u'mode': u'0755', u'pw_name': u'root', u'gid': 0, u'ischr': False, u'wusr': True})
changed: [localhost] => (item={u'rusr': True, u'uid': 0, u'rgrp': True, u'xoth': True, u'islnk': False, u'woth': False, u'nlink': 4, u'issock': False, u'mtime': 1649869343.0, u'gr_name': u'root', u'path': u'/usr/share/percona-dashboards/panels/vertamedia-clickhouse-datasource', u'xusr': True, u'atime': 0.0, u'inode': 35109184, u'isgid': False, u'size': 170, u'isdir': True, u'ctime': 1653378131.693982, u'isblk': False, u'wgrp': False, u'xgrp': True, u'isuid': False, u'dev': 1048731, u'roth': True, u'isreg': False, u'isfifo': False, u'mode': u'0755', u'pw_name': u'root', u'gid': 0, u'ischr': False, u'wusr': True})
changed: [localhost] => (item={u'rusr': True, u'uid': 0, u'rgrp': True, u'xoth': True, u'islnk': False, u'woth': False, u'nlink': 7, u'issock': False, u'mtime': 1649869344.0, u'gr_name': u'root', u'path': u'/usr/share/percona-dashboards/panels/yesoreyeram-boomtable-panel', u'xusr': True, u'atime': 0.0, u'inode': 39284465, u'isgid': False, u'size': 181, u'isdir': True, u'ctime': 1653378131.693982, u'isblk': False, u'wgrp': False, u'xgrp': True, u'isuid': False, u'dev': 1048731, u'roth': True, u'isreg': False, u'isfifo': False, u'mode': u'0755', u'pw_name': u'root', u'gid': 0, u'ischr': False, u'wusr': True})

TASK [dashboards_upgrade : Synchronization plugin] *****************************
changed: [localhost]

TASK [dashboards_upgrade : Check that the SQLite grafana database exists] ******
ok: [localhost]

TASK [dashboards_upgrade : Restart grafana before delete data] *****************
skipping: [localhost]

TASK [dashboards_upgrade : Remove old DBaaS dashboard (Before provisioning times)] ***
skipping: [localhost]

TASK [dashboards_upgrade : Remove old PMM Inventory (Before provisioning times)] ***
skipping: [localhost]

TASK [dashboards_upgrade : Remove old PMM Add Instance dashboard (Before provisioning times)] ***
skipping: [localhost]

TASK [dashboards_upgrade : Remove old PMM Database Checks dashboard (Before provisioning times)] ***
skipping: [localhost]

TASK [dashboards_upgrade : Remove old PMM Settings dashboard (Before provisioning times)] ***
skipping: [localhost]

TASK [dashboards_upgrade : Restart grafana with new plugins] *******************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "grafana: stopped\ngrafana: ERROR (spawn error)\n"}

PLAY RECAP *********************************************************************
localhost                  : ok=11   changed=2    unreachable=0    failed=1    skipped=10   rescued=0    ignored=1   

time="2022-05-24T10:06:08Z" level=fatal msg="RunPlaybook failed: exit status 2"

postgresql.log

2022-05-24 09:49:10.558 UTC [31] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2022-05-24 09:49:10.558 UTC [31] LOG:  could not bind IPv6 address "::1": Cannot assign requested address
2022-05-24 09:49:10.558 UTC [31] HINT:  Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
2022-05-24 09:49:10.562 UTC [31] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-05-24 09:49:10.564 UTC [31] FATAL:  could not create lock file "/tmp/.s.PGSQL.5432.lock": Permission denied
2022-05-24 09:49:10.564 UTC [31] LOG:  database system is shut down
2022-05-24 09:49:11.743 UTC [128] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2022-05-24 09:49:11.743 UTC [128] LOG:  could not bind IPv6 address "::1": Cannot assign requested address
2022-05-24 09:49:11.743 UTC [128] HINT:  Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
2022-05-24 09:49:11.753 UTC [128] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-05-24 09:49:11.758 UTC [128] FATAL:  could not create lock file "/tmp/.s.PGSQL.5432.lock": Permission denied
2022-05-24 09:49:11.758 UTC [128] LOG:  database system is shut down
2022-05-24 09:49:14.823 UTC [151] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2022-05-24 09:49:14.824 UTC [151] LOG:  could not bind IPv6 address "::1": Cannot assign requested address
2022-05-24 09:49:14.824 UTC [151] HINT:  Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
2022-05-24 09:49:14.828 UTC [151] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-05-24 09:49:14.830 UTC [151] FATAL:  could not create lock file "/tmp/.s.PGSQL.5432.lock": Permission denied
2022-05-24 09:49:14.831 UTC [151] LOG:  database system is shut down
2022-05-24 09:49:18.751 UTC [169] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2022-05-24 09:49:18.753 UTC [169] LOG:  could not bind IPv6 address "::1": Cannot assign requested address
2022-05-24 09:49:18.753 UTC [169] HINT:  Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
2022-05-24 09:49:18.756 UTC [169] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-05-24 09:49:18.758 UTC [169] FATAL:  could not create lock file "/tmp/.s.PGSQL.5432.lock": Permission denied
2022-05-24 09:49:18.758 UTC [169] LOG:  database system is shut down
2022-05-24 09:49:22.938 UTC [191] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2022-05-24 09:49:22.939 UTC [191] LOG:  could not bind IPv6 address "::1": Cannot assign requested address
2022-05-24 09:49:22.939 UTC [191] HINT:  Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
2022-05-24 09:49:22.941 UTC [191] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-05-24 09:49:22.943 UTC [191] FATAL:  could not create lock file "/tmp/.s.PGSQL.5432.lock": Permission denied
2022-05-24 09:49:22.943 UTC [191] LOG:  database system is shut down
2022-05-24 09:49:28.753 UTC [529] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2022-05-24 09:49:28.753 UTC [529] LOG:  could not bind IPv6 address "::1": Cannot assign requested address
2022-05-24 09:49:28.753 UTC [529] HINT:  Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
2022-05-24 09:49:28.755 UTC [529] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-05-24 09:49:28.757 UTC [529] FATAL:  could not create lock file "/tmp/.s.PGSQL.5432.lock": Permission denied
2022-05-24 09:49:28.757 UTC [529] LOG:  database system is shut down
2022-05-24 09:49:35.064 UTC [624] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2022-05-24 09:49:35.064 UTC [624] LOG:  could not bind IPv6 address "::1": Cannot assign requested address
2022-05-24 09:49:35.064 UTC [624] HINT:  Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
2022-05-24 09:49:35.066 UTC [624] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-05-24 09:49:35.070 UTC [624] FATAL:  could not create lock file "/tmp/.s.PGSQL.5432.lock": Permission denied
2022-05-24 09:49:35.071 UTC [624] LOG:  database system is shut down
2022-05-24 09:49:42.774 UTC [673] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2022-05-24 09:49:42.775 UTC [673] LOG:  could not bind IPv6 address "::1": Cannot assign requested address
2022-05-24 09:49:42.775 UTC [673] HINT:  Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
2022-05-24 09:49:42.778 UTC [673] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-05-24 09:49:42.782 UTC [673] FATAL:  could not create lock file "/tmp/.s.PGSQL.5432.lock": Permission denied
2022-05-24 09:49:42.783 UTC [673] LOG:  database system is shut down
2022-05-24 09:49:51.653 UTC [1026] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2022-05-24 09:49:51.653 UTC [1026] LOG:  could not bind IPv6 address "::1": Cannot assign requested address
2022-05-24 09:49:51.653 UTC [1026] HINT:  Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
2022-05-24 09:49:51.655 UTC [1026] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-05-24 09:49:51.656 UTC [1026] FATAL:  could not create lock file "/tmp/.s.PGSQL.5432.lock": Permission denied
2022-05-24 09:49:51.656 UTC [1026] LOG:  database system is shut down
2022-05-24 09:50:00.852 UTC [1142] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2022-05-24 09:50:00.852 UTC [1142] LOG:  could not bind IPv6 address "::1": Cannot assign requested address
2022-05-24 09:50:00.852 UTC [1142] HINT:  Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
2022-05-24 09:50:00.854 UTC [1142] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-05-24 09:50:00.855 UTC [1142] FATAL:  could not create lock file "/tmp/.s.PGSQL.5432.lock": Permission denied
2022-05-24 09:50:00.855 UTC [1142] LOG:  database system is shut down
2022-05-24 09:50:11.196 UTC [1251] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2022-05-24 09:50:11.196 UTC [1251] LOG:  could not bind IPv6 address "::1": Cannot assign requested address
2022-05-24 09:50:11.196 UTC [1251] HINT:  Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
2022-05-24 09:50:11.200 UTC [1251] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-05-24 09:50:11.202 UTC [1251] FATAL:  could not create lock file "/tmp/.s.PGSQL.5432.lock": Permission denied
2022-05-24 09:50:11.202 UTC [1251] LOG:  database system is shut down

grafana.log

t=2022-05-24T10:10:18+0000 lvl=info msg="Creating SQLite database file" logger=sqlstore path=/srv/grafana/grafana.db
Failed to start grafana. error: failed to connect to database: failed to create SQLite database file "/srv/grafana/grafana.db": open /srv/grafana/grafana.db: permission denied
failed to connect to database: failed to create SQLite database file "/srv/grafana/grafana.db": open /srv/grafana/grafana.db: permission denied
t=2022-05-24T10:10:22+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings
t=2022-05-24T10:10:22+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings
t=2022-05-24T10:10:22+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini
t=2022-05-24T10:10:22+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini
t=2022-05-24T10:10:22+0000 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.data=/srv/grafana"
t=2022-05-24T10:10:22+0000 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.logs=/srv/logs"
t=2022-05-24T10:10:22+0000 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.plugins=/srv/grafana/plugins"
t=2022-05-24T10:10:22+0000 lvl=info msg="Config overridden from command line" logger=settings arg="default.server.root_url=https://%(domain)s/graph"
t=2022-05-24T10:10:22+0000 lvl=info msg="Config overridden from command line" logger=settings arg="default.log.mode=console"
t=2022-05-24T10:10:22+0000 lvl=info msg="Config overridden from command line" logger=settings arg="default.log.console.format=console"
t=2022-05-24T10:10:22+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana
t=2022-05-24T10:10:22+0000 lvl=info msg="Path Data" logger=settings path=/srv/grafana
t=2022-05-24T10:10:22+0000 lvl=info msg="Path Logs" logger=settings path=/srv/logs
t=2022-05-24T10:10:22+0000 lvl=info msg="Path Plugins" logger=settings path=/srv/grafana/plugins
t=2022-05-24T10:10:22+0000 lvl=info msg="Path Provisioning" logger=settings path=/usr/share/grafana/conf/provisioning
t=2022-05-24T10:10:22+0000 lvl=info msg="App mode production" logger=settings
t=2022-05-24T10:10:22+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3
t=2022-05-24T10:10:22+0000 lvl=info msg="Creating SQLite database file" logger=sqlstore path=/srv/grafana/grafana.db
Failed to start grafana. error: failed to connect to database: failed to create SQLite database file "/srv/grafana/grafana.db": open /srv/grafana/grafana.db: permission denied
failed to connect to database: failed to create SQLite database file "/srv/grafana/grafana.db": open /srv/grafana/grafana.db: permission denied

pmm-agent.log

ERRO[2022-05-24T10:11:39.579+00:00] Failed to establish two-way communication channel: Internal Server Error: HTTP status code 500; transport: received the unexpected content-type "text/html".  component=client
INFO[2022-05-24T10:11:52.625+00:00] Connecting to https://127.0.0.1:443/ ...      component=client
INFO[2022-05-24T10:11:52.628+00:00] Connected to 127.0.0.1:443.                   component=client
INFO[2022-05-24T10:11:52.628+00:00] Establishing two-way communication channel ...  component=client
ERRO[2022-05-24T10:11:52.631+00:00] Failed to establish two-way communication channel: Internal Server Error: HTTP status code 500; transport: received the unexpected content-type "text/html".  component=client
INFO[2022-05-24T10:12:08.592+00:00] Connecting to https://127.0.0.1:443/ ...      component=client
INFO[2022-05-24T10:12:08.599+00:00] Connected to 127.0.0.1:443.                   component=client
INFO[2022-05-24T10:12:08.599+00:00] Establishing two-way communication channel ...  component=client
ERRO[2022-05-24T10:12:08.601+00:00] Failed to establish two-way communication channel: Internal Server Error: HTTP status code 500; transport: received the unexpected content-type "text/html".  component=client

I see some unknown environment variable hint, is that related? Or any thoughts? Thank you!