Bad Gateway upon fresh deployment

[kurokobo]

Bad Gateway upon fresh deployment

[kurokobo]

Just F.Y.I., as of my past experience, there are several possibilities:

• DB issue
  • Something has occured on DB side.
  • Check the logs of PostgreSQL pod.
• Firewall issue
  • Firewall might block communication between AWX and PSQL.
  • Disable any kind of firewall on your K3s host, like firewalld or SuSEfirewall2.
• K3s issue
  • Very rarely, internal network in K3s might be broken and block something.
  • Try uninstall and reinstall K3s.

[nathanlin24]

Thank you for your tips @kurokobo. It's super helpful. Turned out it's the K3S host firewall that's giving me grief. Once firewallD disabled, I do not get the Bad Gateway error anymore.

On a separate note, can I pick your brain a bit on the following error from the postgres pod logs?

2022-03-02 17:53:51.376 UTC [1356] ERROR:  relation "conf_setting" does not exist at character 158
2022-03-02 17:53:51.376 UTC [1356] STATEMENT:  SELECT "conf_setting"."id", "conf_setting"."created", "conf_setting"."modified", "conf_setting"."key", "conf_setting"."value", "conf_setting"."user_id" FROM "conf_setting" WHERE ("conf_setting"."key" = 'OAUTH2_PROVIDER' AND "conf_setting"."user_id" IS NULL) ORDER BY "conf_setting"."id" ASC  LIMIT 1

It doesn't seem to be giving me any issue. My postgres pod has been up for 20+ hours with no restarts. I'm reading through the issue threads in the AWX repo but have not yet gotten to any conclusion yet. I promise will close this issue after this. Thanks a bunch!

[kurokobo]

@nathanlin24
I've double checked that the same errors on my environment. I haven't digged into this error in details, but this seems can be safely ignored at this time if it works well.

[kurokobo]

@nathanlin24
Seems the pod awx-b85cd74b6-l4zn4 restarts over 20 times. So please check the logs to get more information.
Refer my troubleshooting guide.

kubectl -n awx logs deployment/awx-operator-controller-manager -c manager
kubectl -n awx logs deployment/awx -c awx-web
kubectl -n awx logs deployment/awx -c awx-task
kubectl -n awx logs deployment/awx -c awx-ee
kubectl -n awx logs deployment/awx -c redis
kubectl -n awx logs statefulset/awx-postgres

[nathanlin24]

Thanks @kurokobo. It seems to be due to database migration. I will continue to look into this. Thanks.

awx-nathan:~ # kubectl -n awx logs deployments/awx -c awx-web -f
[wait-for-migrations] Waiting for database migrations...
[wait-for-migrations] Attempt 1 of 30
[wait-for-migrations] Waiting 0.5 seconds before next attempt
[wait-for-migrations] Attempt 2 of 30
[wait-for-migrations] Waiting 1 seconds before next attempt
[wait-for-migrations] Attempt 3 of 30
[wait-for-migrations] Waiting 2 seconds before next attempt
[wait-for-migrations] Attempt 4 of 30
[wait-for-migrations] Waiting 4 seconds before next attempt
[wait-for-migrations] Attempt 5 of 30
[wait-for-migrations] Waiting 8 seconds before next attempt
[wait-for-migrations] Attempt 6 of 30
[wait-for-migrations] Waiting 16 seconds before next attempt
[wait-for-migrations] Attempt 7 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 8 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 9 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 10 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 11 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 12 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 13 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 14 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 15 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 16 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 17 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 18 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 19 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 20 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 21 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 22 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 23 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 24 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 25 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 26 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 27 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 28 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 29 of 30
[wait-for-migrations] Waiting 30 seconds before next attempt
[wait-for-migrations] Attempt 30 of 30
[wait-for-migrations] ERROR: Database migrations not applied

[kurokobo]

So you should check the logs of PostgreSQL too.

kubectl -n awx logs statefulset/awx-postgres

[LeoSpyke]

Thank you for your tips @kurokobo. It's super helpful. Turned out it's the K3S host firewall that's giving me grief. Once firewallD disabled, I do not get the Bad Gateway error anymore.

On a separate note, can I pick your brain a bit on the following error from the postgres pod logs?

2022-03-02 17:53:51.376 UTC [1356] ERROR:  relation "conf_setting" does not exist at character 158
2022-03-02 17:53:51.376 UTC [1356] STATEMENT:  SELECT "conf_setting"."id", "conf_setting"."created", "conf_setting"."modified", "conf_setting"."key", "conf_setting"."value", "conf_setting"."user_id" FROM "conf_setting" WHERE ("conf_setting"."key" = 'OAUTH2_PROVIDER' AND "conf_setting"."user_id" IS NULL) ORDER BY "conf_setting"."id" ASC  LIMIT 1

It doesn't seem to be giving me any issue. My postgres pod has been up for 20+ hours with no restarts. I'm reading through the issue threads in the AWX repo but have not yet gotten to any conclusion yet. I promise will close this issue after this. Thanks a bunch!

Faced the same issue, thanks for the advice!

[nathanlin24]

Thanks for your feedback @kurokobo!