CrashLoopBackOff on awx-postgres-0 pod

[kuchynkap]

CrashLoopBackOff on awx-postgres-0 pod

[kurokobo]

@kuchynkap
Thanks for reporting, If your PV is based on K3s' Local Path Provisioner instead of hostPath as in my repository, a similar issue has been reported on ansible/awx-operator#483.
Try using the latest 0.13.0 of AWX Operator.

If it doesn't helps, we need more information to investigate. The following commands may help you find the cause of CrashLoopBackOff.

# You can get detailed information in "Events" section
kubectl -n awx describe pod awx-postgres-0

# You can get the logs from PostgreSQL
kubectl -n awx logs awx-postgres-0

[kuchynkap]

@kurokobo
Thank you for your reply! I am using hostpath, but I am hitting the issue mentioned.

$ kubectl -n awx logs awx-postgres-0
mkdir: cannot create directory ‘/var/lib/postgresql/data’: Permission denied

[kurokobo]

@kuchynkap
Okay, so you should focus on the permissions, as described in that error message.
Could you check the permission and owner/group of the actual directories you are using in hostPath?

# Replace `/data/postgres` with your directory path.
$ ls -ld /data/postgres /data/postgres/data
drwxr-xr-x. 2 root root 18 Aug 20 10:09 /data/postgres
drwxr-xr-x. 3 root root 20 Aug 20 10:09 /data/postgres/data

If your hostPath is /data/postgres, Kubernetes will create /data/postgres/data as subPath for awx-postgres-0 pod. The problem is the permissions of this subPath directory.

In my environment, 755 and root:root (0:0) works correctly. So you can try;

# Replace `/data/postgres` with your directory path.
sudo chmod 755 /data/postgres /data/postgres/data
sudo chown 0:0 /data/postgres /data/postgres/data

Or, you can also try 999:999 as owner/group for the directory.

sudo chmod 755 /data/postgres /data/postgres/data
sudo chown 999:999 /data/postgres /data/postgres/data

999 is the UID/GID of the postgres user which used in the container.

[kuchynkap]

Hi @kurokobo,

thank you!

@kuchynkap
Okay, so you should focus on the permissions, as described in that error message.
Could you check the permission and owner/group of the actual directories you are using in hostPath?

# Replace `/data/postgres` with your directory path.
$ ls -ld /data/postgres /data/postgres/data
drwxr-xr-x. 2 root root 18 Aug 20 10:09 /data/postgres
drwxr-xr-x. 3 root root 20 Aug 20 10:09 /data/postgres/data

If your hostPath is /data/postgres, Kubernetes will create /data/postgres/data as subPath for awx-postgres-0 pod. The problem is the permissions of this subPath directory.

In my environment, 755 and root:root (0:0) works correctly. So you can try;

# Replace `/data/postgres` with your directory path.
sudo chmod 755 /data/postgres /data/postgres/data
sudo chown 0:0 /data/postgres /data/postgres/data

Or, you can also try 999:999 as owner/group for the directory.

sudo chmod 755 /data/postgres /data/postgres/data
sudo chown 999:999 /data/postgres /data/postgres/data

999 is the UID/GID of the postgres user which used in the container.

Hi,

thank you! chmodding /data/postgres /data/postgres/data solved the issue!

[kurokobo]

good to hear that, have fun 🚀