Nothing is up on port 443 (or 80) ? - Why ?

[kurokobo]

Nothing is up on port 443 (or 80) ? - Why ?

[nashford77]

ok wow - success - it was a prior failed deployment from something other that k3s - anyways i cleaned up FW rules, the lot - took some hassle but its up ! thanks man!

[nashford77]

On a NC, Telnet, CURL etc - nothing there - also nothing in 'netstat -an' ? nothing else is on 443 or 80 ....

[Randy29800]

Hi same problem here.

[kurokobo]

@nashford77 @Randy29800
Thanks for reporting,

The K3s (to be accurate, its Traefik) forwards packets using iptables' rules, so it doesn't listen directly on the host's port. It is expected behavior that netstat -an doesn't show any process which listenning on :80 or :443.

If curl <Your External IP, in @nashford77's case 192.168.5.104> returns 404 page not found, it's working fine. You can use your AWX at https://awx.tunninet.com/

$ curl 192.168.0.219
404 page not found

If this returns Connection refused even if you stopped your firewalld, it may be the same as this issue in K3s. You can try the workaround in this comment.

[Randy29800]

@kurokobo
Thanks for you reply.
I checked that the static files are not in the folder projects.
After the deployment the /data/folder in empty.
The folder postgres well contained the database.
To check if the folder is correct, i put it in 777.
After redeploy come back to 755 but is 0:1000 (instead of 1000:0) so something happen on it.
Don't understand why no files are in projects.
Any idea?
(Sorry for my english)

[kurokobo]

@Randy29800
Seems you're commenting about the issue #2 instead of this issue (#3) ?
Could you please move on to #2 or create new one.

[nashford77]

I tried those random ports as well - they do not respond either. If it relies on IP tables, would forwarding have to be enabled? That's not default of course on Ubuntu. I tried on local house to connect to those random ports without luck

…

On Thu, Aug 26, 2021, 8:31 AM kurokobo ***@***.***> wrote: @nashford77 <https://github.com/nashford77> @Randy29800 <https://github.com/Randy29800> Thanks for reporting, The K3s (to be accurate, its Traefik) forwards packets using iptables' rules, so it doesn't listen directly on the host's port. It is expected behavior that netstat -an doesn't show any process which listenning on :80 or :443. If curl <Your External IP, in @nashford77's case 192.168.5.104> returns 404 page not found, it's working fine. You can use your AWX at https://awx.tunninet.com/ $ curl 192.168.0.219 404 page not found If this returns Connection refused even if you stopped your firewalld, it may be the same as this issue <k3s-io/k3s#1414> in K3s. You can try the workaround in this comment <k3s-io/k3s#1414 (comment)>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFSSJU7TL3GJTQZSUUSKINDT6YX2NANCNFSM5C2MTTNQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .

[kurokobo]

@nashford77
Nope, I mean you should try this workaround, which applies and patches a spec for traefik.
Can you tell me the OS and platform? I can't reproduce it in my environment.

[nashford77]

Ubuntu 20.04 vanilla - nothing fancy on it, was a new build totally

…

On Thu, Aug 26, 2021, 10:53 AM kurokobo ***@***.***> wrote: @nashford77 <https://github.com/nashford77> Nope, I mean you should try this workaround <k3s-io/k3s#1414 (comment)>, which applies and patches a spec for traefik. Can you tell me the OS and platform? I can't reproduce it in my environment. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFSSJU6V5UYK3E55XYLHNE3T6ZIQBANCNFSM5C2MTTNQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .

[nashford77]

apiVersion: v1
kind: Service
metadata:
annotations:
meta.helm.sh/release-name: traefik
meta.helm.sh/release-namespace: kube-system
creationTimestamp: "2021-08-26T02:34:45Z"
labels:
app.kubernetes.io/instance: traefik
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: traefik
helm.sh/chart: traefik-9.18.2
name: traefik
namespace: kube-system
resourceVersion: "33339"
uid: 5512d2e3-e339-4821-8de8-5236b73cba6d
spec:
clusterIP: 10.43.185.225
clusterIPs:

• 10.43.185.225
  externalIPs:
• 192.168.5.104
  externalTrafficPolicy: Cluster
  ipFamilies:
• IPv4
  ipFamilyPolicy: SingleStack
  ports:
• name: web
  nodePort: 32080
  port: 80
  protocol: TCP
  targetPort: web
• name: websecure
  nodePort: 32443
  port: 443
  protocol: TCP
  targetPort: websecure
  selector:
  app.kubernetes.io/instance: traefik
  app.kubernetes.io/name: traefik
  sessionAffinity: None
  type: LoadBalancer <----- HERE
  status:
  loadBalancer:
  ingress:
  • ip: 192.168.5.104

(I have this in the hosts file just to test...)

root@u500-cube-server:~/awx-on-k3s# curl https://awx.tunninet.com --insecure
<no response, hitting enter lots>

^C
root@u500-cube-server:~/awx-on-k3s# curl http://awx.tunninet.com --insecure
<no response, hitting enter lots>

[kurokobo]

@nashford77
I've deployed new Ubuntu 20.04, K3s, AWX just now, everything works as expected 🤔

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.1 LTS
Release:        20.04
Codename:       focal

$ kubectl -n awx get awx,all,ingress
NAME                      AGE
awx.awx.ansible.com/awx   5m9s

NAME                      READY   STATUS    RESTARTS   AGE
pod/awx-postgres-0        1/1     Running   0          4m53s
pod/awx-59ff55b5b-8hp99   4/4     Running   0          4m45s

NAME                   TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
service/awx-postgres   ClusterIP   None         <none>        5432/TCP   4m53s
service/awx-service    ClusterIP   10.43.21.0   <none>        80/TCP     4m46s

NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/awx   1/1     1            1           4m45s

NAME                            DESIRED   CURRENT   READY   AGE
replicaset.apps/awx-59ff55b5b   1         1         1       4m45s

NAME                            READY   AGE
statefulset.apps/awx-postgres   1/1     4m53s

NAME                                    CLASS    HOSTS             ADDRESS         PORTS     AGE
ingress.networking.k8s.io/awx-ingress   <none>   awx.example.com   192.168.0.218   80, 443   4m46s

curl also respond as expected.

$ curl 192.168.0.219
404 page not found

$ curl -k https://awx.example.com/
<!doctype html><html lang="en"><head><script nonce="xSOSPXwun2bwLidkIAIBMLegcEGQxs6CQ6X+gI5M71A=" typ...

[nashford77]

Mine deploys fine, but it did not respond to a named call. I have wiped it many times, retried it etc. I have to be missing something. Did you not enable ip forwarding or anything special ?

…

On Thu, Aug 26, 2021, 11:30 AM kurokobo ***@***.***> wrote: @nashford77 <https://github.com/nashford77> I've deployed new Ubuntu 20.04, K3s, AWX just now, everything works as expected 🤔 $ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.1 LTS Release: 20.04 Codename: focal $ kubectl -n awx get awx,all,ingress NAME AGE awx.awx.ansible.com/awx 5m9s NAME READY STATUS RESTARTS AGE pod/awx-postgres-0 1/1 Running 0 4m53s pod/awx-59ff55b5b-8hp99 4/4 Running 0 4m45s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/awx-postgres ClusterIP None <none> 5432/TCP 4m53s service/awx-service ClusterIP 10.43.21.0 <none> 80/TCP 4m46s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/awx 1/1 1 1 4m45s NAME DESIRED CURRENT READY AGE replicaset.apps/awx-59ff55b5b 1 1 1 4m45s NAME READY AGE statefulset.apps/awx-postgres 1/1 4m53s NAME CLASS HOSTS ADDRESS PORTS AGE ingress.networking.k8s.io/awx-ingress <none> awx.example.com 192.168.0.218 80, 443 4m46s curl also respond as expected. $ curl 192.168.0.219 404 page not found $ curl -k https://awx.example.com/ <!doctype html><html lang="en"><head><script nonce="xSOSPXwun2bwLidkIAIBMLegcEGQxs6CQ6X+gI5M71A=" typ... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFSSJUYZBO4XQ5FMWHQC7JTT6ZMZXANCNFSM5C2MTTNQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .

[kurokobo]

@nashford77
Try curl http://192.168.5.104/ -H "host: awx.tunninet.com"

[kurokobo]

Did you not enable ip forwarding or anything special

I haven't changed any OS settings except for /etc/hosts since the clean install of Ubuntu 20.04 Server. Current configuration around IP forwarding looks like this.

$ sudo sysctl -a | grep ip_forward
net.ipv4.ip_forward = 1
net.ipv4.ip_forward_update_priority = 1
net.ipv4.ip_forward_use_pmtu = 0

$ sudo systemctl status ufw
● ufw.service - Uncomplicated firewall
     Loaded: loaded (/lib/systemd/system/ufw.service; enabled; vendor preset: enabled)
     Active: active (exited) since Thu 2021-08-26 15:16:21 UTC; 23min ago
       Docs: man:ufw(8)
   Main PID: 502 (code=exited, status=0/SUCCESS)
      Tasks: 0 (limit: 9448)
     Memory: 0B
     CGroup: /system.slice/ufw.service

$ grep awx /etc/hosts
192.168.0.218 awx.example.com

[nashford77]

root@u500-cube-server:~/awx-on-k3s# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 21.04
Release: 21.04
Codename: hirsute

root@u500-cube-server:~/awx-on-k3s# kubectl -n awx get awx,all,ingress
NAME AGE
awx.awx.ansible.com/awx 12h

NAME READY STATUS RESTARTS AGE
pod/awx-postgres-0 1/1 Running 0 12h
pod/awx-59ff55b5b-2czpx 4/4 Running 68 12h

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/awx-postgres ClusterIP None 5432/TCP 12h
service/awx-service ClusterIP 10.43.251.54 80/TCP 12h

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/awx 1/1 1 1 12h

NAME DESIRED CURRENT READY AGE
replicaset.apps/awx-59ff55b5b 1 1 1 12h

NAME READY AGE
statefulset.apps/awx-postgres 1/1 12h

NAME CLASS HOSTS ADDRESS PORTS AGE
ingress.networking.k8s.io/awx-ingress awx.tunninet.com 192.168.5.104 80, 443 12h

root@u500-cube-server:~/awx-on-k3s# curl 192.168.5.104
404 page not found

root@u500-cube-server:~/awx-on-k3s# ping awx.tunninet.com (ENABLED FOR TESTING ONLY)
PING awx.tunninet.com (192.168.5.104) 56(84) bytes of data.
64 bytes from awx.tunninet.com (192.168.5.104): icmp_seq=1 ttl=64 time=0.121 ms
64 bytes from awx.tunninet.com (192.168.5.104): icmp_seq=2 ttl=64 time=0.083 ms

root@u500-cube-server:~/awx-on-k3s# curl -k https://awx.tunninet.com/

^C

root@u500-cube-server:~/awx-on-k3s# curl http://192.168.5.104/ -H "host: awx.tunninet.com"
<Same, it hung....>

^C

root@u500-cube-server:~/awx-on-k3s# sudo systemctl status ufw
● ufw.service - Uncomplicated firewall
Loaded: loaded (/lib/systemd/system/ufw.service; disabled; vendor preset: enabled)
Active: active (exited) since Thu 2021-08-26 15:41:16 UTC; 22s ago
Docs: man:ufw(8)
Process: 478512 ExecStart=/lib/ufw/ufw-init start quiet (code=exited, status=0/SUCCESS)
Main PID: 478512 (code=exited, status=0/SUCCESS)

root@u500-cube-server:~/awx-on-k3s# sudo sysctl -a | grep ip_forward
net.ipv4.ip_forward = 1
net.ipv4.ip_forward_update_priority = 1
net.ipv4.ip_forward_use_pmtu = 0

[nashford77]

i also tried w UFW down as well to test - no dice

[nashford77]

oh - i am on 21.04 - i am guessing that's an issue :( Can you try on that code line ?

[kurokobo]

Now installing 21.04 on my lab 🚀

[kurokobo]

Wired, works good for me 🤔

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 21.04
Release:        21.04
Codename:       hirsute

$ grep awx /etc/hosts
192.168.0.218 awx.tunninet.com

$ curl -k https://awx.tunninet.com/
<!doctype html><html lang="en"><head><script nonce="ZAShxq63O1EUgPzaBUxvuqDX4Cqdpb/91ow/0fIxbSg=" type="t...

Could you share the output of these commands?
And just to confirm, is there any proxies?

k3s --version
kubectl -n kube-system get po
kubectl -n awx get ingress awx-ingress -o yaml
curl 192.168.5.104
curl http://192.168.5.104/
dig awx.tunninet.com

[nashford77]

root@u500-cube-server:/awx-on-k3s# k3s --version
k3s version v1.21.3+k3s1 (1d1f220f)
go version go1.16.6
root@u500-cube-server:/awx-on-k3s# kubectl -n kube-system get po
NAME READY STATUS RESTARTS AGE
metrics-server-86cbb8457f-l9mnh 1/1 Running 0 13h
local-path-provisioner-5ff76fc89d-v6tv2 1/1 Running 0 13h
coredns-7448499f4d-kgnms 1/1 Running 0 13h
helm-install-traefik-crd-wrhc7 0/1 Completed 0 13h
helm-install-traefik-nbvwh 0/1 Completed 1 13h
svclb-traefik-9s8vg 2/2 Running 0 13h
traefik-97b44b794-n5j4t 1/1 Running 0 13h
root@u500-cube-server:~/awx-on-k3s# kubectl -n awx get ingress awx-ingress -o yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: '{"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"labels":{"app.kubernetes.io/component":"awx","app.kubernetes.io/managed-by":"awx-operator","app.kubernetes.io/name":"awx","app.kubernetes.io/operator-version":"0.13.0","app.kubernetes.io/part-of":"awx"},"name":"awx-ingress","namespace":"awx"},"spec":{"rules":[{"host":"awx.tunninet.com","http":{"paths":[{"backend":{"serviceName":"awx-service","servicePort":80},"path":"/"}]}}],"tls":[{"hosts":["awx.tunninet.com"],"secretName":"awx-secret-tls"}]}}'
creationTimestamp: "2021-08-26T02:38:00Z"
generation: 1
labels:
app.kubernetes.io/component: awx
app.kubernetes.io/managed-by: awx-operator
app.kubernetes.io/name: awx
app.kubernetes.io/operator-version: 0.13.0
app.kubernetes.io/part-of: awx
name: awx-ingress
namespace: awx
ownerReferences:

• apiVersion: awx.ansible.com/v1beta1
  kind: AWX
  name: awx
  uid: 6cc1d8cd-45a3-46a0-8d8c-38974e7657d3
  resourceVersion: "913"
  uid: c3f5d150-fdf0-4a16-b555-d3e78bf66dd8
  spec:
  rules:
• host: awx.tunninet.com
  http:
  paths:
  • backend:
    service:
    name: awx-service
    port:
    number: 80
    path: /
    pathType: ImplementationSpecific
    tls:
• hosts:
  • awx.tunninet.com
    secretName: awx-secret-tls
    status:
    loadBalancer:
    ingress:
  • ip: 192.168.5.104
    root@u500-cube-server:/awx-on-k3s# curl 192.168.5.104
    404 page not found
    root@u500-cube-server:/awx-on-k3s# curl http://192.168.5.104/
    404 page not found
    root@u500-cube-server:~/awx-on-k3s# dig awx.tunninet.com

; <<>> DiG 9.16.8-Ubuntu <<>> awx.tunninet.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 52069
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;awx.tunninet.com. IN A

;; ANSWER SECTION:
awx.tunninet.com. 0 IN A 192.168.5.104

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Thu Aug 26 16:29:51 UTC 2021
;; MSG SIZE rcvd: 61

``

[nashford77]

can you show me a snap list as well ? wondering which snaps you choose or did not choose

[kurokobo]

$ sudo snap list
Name    Version   Rev    Tracking         Publisher   Notes
core18  20210309  1997   latest/stable    canonical?  base
lxd     4.13      20037  latest/stable/…  canonical?  -
snapd   2.49.2    11588  latest/stable    canonical?  snapd

[kurokobo]

F.Y.I, it's good way to wrap your output with triple back-quotes.

```
$ sudo snap list
Name    Version   Rev    Tracking         Publisher   Notes
core18  20210309  1997   latest/stable    canonical?  base
lxd     4.13      20037  latest/stable/…  canonical?  -
snapd   2.49.2    11588  latest/stable    canonical?  snapd
```

This will show your output as formatted strings.

[kurokobo]

I don't know if it will be helpful, can you try restarting treafik?

# Once deleted, new pods will be created and started again automatically
kubectl -n kube-system delete pod -l app.kubernetes.io/name=traefik
kubectl -n kube-system delete pod -l app=svclb-traefik

And just wait few minutes, check again; -v is the option to get verbose output.

curl -v http://192.168.5.104/
curl -v http://192.168.5.104/ -H "Host: example.com"
curl -v http://192.168.5.104/ -H "Host: awx.tunninet.com"
curl -v http://awx.tunninet.com/

[nashford77]

root@u500-cube-server:~/awx-on-k3s# snap list
Name Version Rev Tracking Publisher Notes
core 16-2.51.4 11606 latest/stable canonical✓ core
core18 20210722 2128 latest/stable canonical✓ base
core20 20210702 1081 latest/stable canonical✓ base
lxd 4.17 21260 latest/stable/… canonical✓ -
snapd 2.51.4 12883 latest/stable canonical✓ snapd

[nashford77]

deleted pods - waited 5-10....

root@u500-cube-server:~/awx-on-k3s# curl -v http://192.168.5.104/

• Trying 192.168.5.104:80...
• Connected to 192.168.5.104 (192.168.5.104) port 80 (#0)

GET / HTTP/1.1
Host: 192.168.5.104
User-Agent: curl/7.74.0
Accept: /

• Mark bundle as not supporting multiuse
  < HTTP/1.1 404 Not Found
  < Content-Type: text/plain; charset=utf-8
  < X-Content-Type-Options: nosniff
  < Date: Thu, 26 Aug 2021 17:05:14 GMT
  < Content-Length: 19
  <
  404 page not found
• Connection #0 to host 192.168.5.104 left intact
  root@u500-cube-server:~/awx-on-k3s# curl -v http://192.168.5.104/
• Trying 192.168.5.104:80...
• Connected to 192.168.5.104 (192.168.5.104) port 80 (#0)

GET / HTTP/1.1
Host: 192.168.5.104
User-Agent: curl/7.74.0
Accept: /

• Mark bundle as not supporting multiuse
  < HTTP/1.1 404 Not Found
  < Content-Type: text/plain; charset=utf-8
  < X-Content-Type-Options: nosniff
  < Date: Thu, 26 Aug 2021 17:11:50 GMT
  < Content-Length: 19
  <
  404 page not found
• Connection #0 to host 192.168.5.104 left intact
  root@u500-cube-server:~/awx-on-k3s# curl -v http://192.168.5.104/ -H "Host: example.com"
• Trying 192.168.5.104:80...
• Connected to 192.168.5.104 (192.168.5.104) port 80 (#0)

GET / HTTP/1.1
Host: example.com
User-Agent: curl/7.74.0
Accept: /

• Mark bundle as not supporting multiuse
  < HTTP/1.1 404 Not Found
  < Content-Type: text/plain; charset=utf-8
  < X-Content-Type-Options: nosniff
  < Date: Thu, 26 Aug 2021 17:12:07 GMT
  < Content-Length: 19
  <
  404 page not found
• Connection #0 to host 192.168.5.104 left intact
  root@u500-cube-server:~/awx-on-k3s# curl -v http://192.168.5.104/ -H "Host: awx.tunninet.com"
• Trying 192.168.5.104:80...
• Connected to 192.168.5.104 (192.168.5.104) port 80 (#0)

GET / HTTP/1.1
Host: awx.tunninet.com
User-Agent: curl/7.74.0
Accept: /

^C
root@u500-cube-server:~/awx-on-k3s# curl -v http://awx.tunninet.com/

• Trying 192.168.5.104:80...
• Connected to awx.tunninet.com (192.168.5.104) port 80 (#0)

GET / HTTP/1.1
Host: awx.tunninet.com
User-Agent: curl/7.74.0
Accept: /

still hangs but responds ? so odd

[nashford77]

can you show the output of::

kubectl get svc --all-namespaces

[kurokobo]

$ kubectl get svc --all-namespaces
NAMESPACE     NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                      AGE
default       kubernetes             ClusterIP      10.43.0.1       <none>          443/TCP                      99m
kube-system   kube-dns               ClusterIP      10.43.0.10      <none>          53/UDP,53/TCP,9153/TCP       99m
kube-system   metrics-server         ClusterIP      10.43.132.143   <none>          443/TCP                      99m
default       awx-operator-metrics   ClusterIP      10.43.77.208    <none>          8383/TCP,8686/TCP            95m
kube-system   traefik                LoadBalancer   10.43.147.173   192.168.0.218   80:30107/TCP,443:31629/TCP   98m
awx           awx-postgres           ClusterIP      None            <none>          5432/TCP                     2m32s
awx           awx-service            ClusterIP      10.43.215.150   <none>          80/TCP                       2m26s

[nashford77]

should the LB not point to the AWX-service ?

--> kubectl edit svc -n kube-system traefik <-- this shows me the below (i wanted to edit the ports as these are just how the system maps to the svc right ? can you should me this output ? should the IP not match up to AWX-SERVICE ?

apiVersion: v1
kind: Service
metadata:
annotations:
meta.helm.sh/release-name: traefik
meta.helm.sh/release-namespace: kube-system
creationTimestamp: "2021-08-26T02:34:45Z"
labels:
app.kubernetes.io/instance: traefik
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: traefik
helm.sh/chart: traefik-9.18.2
name: traefik
namespace: kube-system
resourceVersion: "37977"
uid: 5512d2e3-e339-4821-8de8-5236b73cba6d
spec:
clusterIP: 10.43.185.225
clusterIPs:

• 10.43.185.225
  externalIPs:
• 192.168.5.104
  externalTrafficPolicy: Cluster
  ipFamilies:
• IPv4
  ipFamilyPolicy: SingleStack
  ports:
• name: web
  nodePort: 32080
  port: 80
  protocol: TCP
  targetPort: web
• name: websecure
  nodePort: 32443
  port: 443
  protocol: TCP
  targetPort: websecure
  selector:
  app.kubernetes.io/instance: traefik
  app.kubernetes.io/name: traefik
  sessionAffinity: None
  type: LoadBalancer
  status:
  loadBalancer:
  ingress:
  • ip: 192.168.5.104

[kurokobo]

There is no problem if the treafik doesn't have a spec pointing to the AWX's IP address.
Traefik is used commonly across the cluster and is not specific to AWX.

$ kubectl -n kube-system get svc traefik -o yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: traefik
    meta.helm.sh/release-namespace: kube-system
  creationTimestamp: "2021-08-26T15:59:15Z"
  labels:
    app.kubernetes.io/instance: traefik
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: traefik
    helm.sh/chart: traefik-9.18.2
  name: traefik
  namespace: kube-system
  resourceVersion: "3124"
  uid: 51895f37-1076-4ac5-a854-4d9d3df13b12
spec:
  clusterIP: 10.43.147.173
  clusterIPs:
  - 10.43.147.173
  externalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: web
    nodePort: 30107
    port: 80
    protocol: TCP
    targetPort: web
  - name: websecure
    nodePort: 31629
    port: 443
    protocol: TCP
    targetPort: websecure
  selector:
    app.kubernetes.io/instance: traefik
    app.kubernetes.io/name: traefik
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: 192.168.0.218

[kurokobo]

Looking at the results of curl -v, it seems that Traefik is working, the requests was acceptted but no response...?
Can you check the logs of awx-web container?

# Start watching
kubectl -n awx logs -f deployment.apps/awx -c awx-web

# Open another terminal and invoke curl
curl -v http://awx.tunninet.com/

Expected logs when accessing from curl is:

2021-08-26 17:48:52,271 DEBUG    [20dde60880e84f5da9907cb3414c2263] awx.analytics.performance request: <WSGIRequest: GET '/'>, response_time: 0.211s
10.42.0.16 - - [26/Aug/2021:17:48:52 +0000] "GET / HTTP/1.1" 200 1190 "-" "curl/7.74.0" "10.42.0.1"
[pid: 46|app: 0|req: 1/2] 10.42.0.16 () {46 vars in 582 bytes} [Thu Aug 26 17:48:52 2021] GET / => generated 1190 bytes in 212 msecs (HTTP/1.1 200) 7 headers in 269 bytes (1 switches on core 0)

[kurokobo]

Sorry, I'm out of time today. Please write it down and I'll look at it again later.

[nashford77]

Nothing at all is logged - i just tried. So it would seem its talking to nothing somehow

[nashford77]

root@u500-cube-server:~/awx-on-k3s# kubectl -n awx logs -f deployment.apps/awx -c awx-web
[wait-for-migrations] Waiting for database migrations...
[wait-for-migrations] Attempt 1 of 30
[wait-for-migrations] Waiting 0.5 seconds before next attempt
[wait-for-migrations] Attempt 2 of 30
[wait-for-migrations] Waiting 1 seconds before next attempt
[wait-for-migrations] Attempt 3 of 30
[wait-for-migrations] Waiting 2 seconds before next attempt
[wait-for-migrations] Attempt 4 of 30
[wait-for-migrations] Waiting 4 seconds before next attempt

[kurokobo]

oops, that logs mean simply your awx is not working...

[nashford77]

i read that perhaps its stuck in a migrating state ? how would we manually force it ?

awx-manage migrate --noinput

how do i pass this to the container or log in to it with a shell of /bin/bash and do that ?

[kurokobo]

i'm out now so can't type exact command, but you can open shell by kubectl -n awx exec -it <pod name> -c <container name> -- bash. awx-manage is in awx-task container i think.

typically that log is caused by the connection issue between awx and postgres.
i recommend wipe out everything by uninstalling k3s and manually delete data file under /data/postgres which used by PV and deploy again.
The postgres credentials may be inconsistent with the data files, or it may be a problem with iptables or ufw.

[nashford77]

No go ....

root@u500-cube-server:~/awx-on-k3s# kubectl -n awx exec -it awx-59ff55b5b-2czpx -c awx-task --stdin --tty awx-task -- /bin/bash
bash-4.4$ awx-manage migrate --noinput
Traceback (most recent call last):
File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/psycopg2/init.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not translate host name "awx-postgres" to address: Name or service not known

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/bin/awx-manage", line 8, in
sys.exit(manage())
File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/awx/init.py", line 155, in manage
if (connection.pg_version // 10000) < 12:
File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/init.py", line 28, in getattr
return getattr(connections[DEFAULT_DB_ALIAS], item)
File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/utils/functional.py", line 80, in get
res = instance.dict[self.name] = self.func(instance)
File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/postgresql/base.py", line 282, in pg_version
with self.temporary_connection():
File "/usr/lib64/python3.8/contextlib.py", line 113, in enter
return next(self.gen)
File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/base/base.py", line 593, in temporary_connection
with self.cursor() as cursor:
File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/utils.py", line 89, in exit
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
connection = Database.connect(**conn_params)
File "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/psycopg2/init.py", line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not translate host name "awx-postgres" to address: Name or service not known

bash-4.4$ command terminated with exit code 137
root@u500-cube-server:~/awx-on-k3s#

[nashford77]

btw i also tried on aws-web - no go

[nashford77]

how the heck to i delete this :) theres no deployment attached.... i think something stale is causing FW issues.

root@u500-cube-server:~# kubectl get all --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system pod/helm-install-traefik-crd-r94hs 0/1 Pending 0 2m
kube-system pod/helm-install-traefik-hdhq4 0/1 Pending 0 112s

NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/kubernetes ClusterIP 10.43.0.1 443/TCP 31m

NAMESPACE NAME COMPLETIONS DURATION AGE
kube-system job.batch/helm-install-traefik-crd 0/1 2m 2m
kube-system job.batch/helm-install-traefik 0/1 112s 112s

[nashford77]

they just return (cant set replica to 0 ... ) a delete comes right back

[kurokobo]

Sorry I didn't get a chance to reply whole day. Thanks for letting me know it was resolved 😃