Few containers fail to start: iptables v1.4.21: can't initialize iptables table `filter'

[metal3-io]

Few containers fail to start: iptables v1.4.21: can't initialize iptables table `filter'

[yprokule]

/cc @dtantsur @derekhiggins

[juliakreger]

@yprokule Are they being launched by the BMO as privileged containers with host networking?

[yprokule]

@juliakreger not sure how to check it, but here is containers' description from pod:

  baremetal-operator:
    Container ID:  cri-o://ba633c4366c33bb0b4a8b457e891b88a78c93fb0f4d27b18ed35262bbfb2a1c9
    Image:         quay.io/metal3-io/baremetal-operator:master
    Image ID:      quay.io/metal3-io/baremetal-operator@sha256:9723f8bc650e83e135e1be3387c906ec4c5fc5213030a043d8f955b2f8f22638
    Port:          60000/TCP
    Host Port:     60000/TCP
    Command:
      /baremetal-operator
    State:          Running
      Started:      Mon, 29 Jul 2019 12:17:38 +0000
    Ready:          True
    Restart Count:  0
    Environment:
      WATCH_NAMESPACE:            openshift-machine-api (v1:metadata.namespace)
      POD_NAME:                   metal3-baremetal-operator-74fdb86688-pw6c4 (v1:metadata.name)
      OPERATOR_NAME:              baremetal-operator
      DEPLOY_KERNEL_URL:          <set to the key 'deploy_kernel_url' of config map 'ironic-bmo-configmap'>          Optional: false
      DEPLOY_RAMDISK_URL:         <set to the key 'deploy_ramdisk_url' of config map 'ironic-bmo-configmap'>         Optional: false
      IRONIC_ENDPOINT:            <set to the key 'ironic_endpoint' of config map 'ironic-bmo-configmap'>            Optional: false
      IRONIC_INSPECTOR_ENDPOINT:  <set to the key 'ironic_inspector_endpoint' of config map 'ironic-bmo-configmap'>  Optional: false
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from metal3-baremetal-operator-token-9s57b (ro)
  ironic-dnsmasq:
    Container ID:  cri-o://9cbc212cc120330dae1c053a04f7ef9593222642bb89446c63874efcf80c33e9
    Image:         quay.io/metal3-io/ironic:master
    Image ID:      quay.io/metal3-io/ironic@sha256:187b0a2918d4ec3c79f2aa2ffd8cfbae19603c75266a675789e5b014936088bb
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/rundnsmasq
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    3
      Started:      Mon, 29 Jul 2019 16:09:59 +0000
      Finished:     Mon, 29 Jul 2019 16:09:59 +0000
    Ready:          False
    Restart Count:  49
    Environment:
      HTTP_PORT:               <set to the key 'http_port' of config map 'ironic-bmo-configmap'>               Optional: false
      PROVISIONING_INTERFACE:  <set to the key 'provisioning_interface' of config map 'ironic-bmo-configmap'>  Optional: false
      DHCP_RANGE:              <set to the key 'dhcp_range' of config map 'ironic-bmo-configmap'>              Optional: false
    Mounts:
      /shared from ironic-data-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from metal3-baremetal-operator-token-9s57b (ro)
  ironic-httpd:
    Container ID:  cri-o://8d567cb6e0382b1d48b9db032a04ec9585ba8aae86011ac5da0f439f2c6d3ab2
    Image:         quay.io/metal3-io/ironic:master
    Image ID:      quay.io/metal3-io/ironic@sha256:187b0a2918d4ec3c79f2aa2ffd8cfbae19603c75266a675789e5b014936088bb
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/runhttpd
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    3
      Started:      Mon, 29 Jul 2019 16:10:03 +0000
      Finished:     Mon, 29 Jul 2019 16:10:03 +0000
    Ready:          False
    Restart Count:  49
    Environment:
      HTTP_PORT:               <set to the key 'http_port' of config map 'ironic-bmo-configmap'>               Optional: false
      PROVISIONING_INTERFACE:  <set to the key 'provisioning_interface' of config map 'ironic-bmo-configmap'>  Optional: false
    Mounts:
      /shared from ironic-data-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from metal3-baremetal-operator-token-9s57b (ro)
  ironic-api:
    Container ID:  cri-o://b1b3e604f303bf93fedd0d5b43f8116917a3392568598443ff1ac56ad1c78d33
    Image:         quay.io/metal3-io/ironic:master
    Image ID:      quay.io/metal3-io/ironic@sha256:187b0a2918d4ec3c79f2aa2ffd8cfbae19603c75266a675789e5b014936088bb
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/runironic-api
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    3
      Started:      Mon, 29 Jul 2019 16:11:05 +0000
      Finished:     Mon, 29 Jul 2019 16:11:05 +0000
    Ready:          False
    Restart Count:  49
    Environment:
      MARIADB_PASSWORD:        <set to the key 'password' in secret 'mariadb-password'>                        Optional: false
      HTTP_PORT:               <set to the key 'http_port' of config map 'ironic-bmo-configmap'>               Optional: false
      PROVISIONING_INTERFACE:  <set to the key 'provisioning_interface' of config map 'ironic-bmo-configmap'>  Optional: false
    Mounts:
      /shared from ironic-data-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from metal3-baremetal-operator-token-9s57b (ro)

[sadasu]

@imain @dhellmann All containers except the baremetal-operator have privileged set to True. https://github.com/openshift-metal3/dev-scripts/blob/master/operator_ironic.yaml

[juliakreger]

@yprokule Are your steps to reproduce purely fire-up an install with openshift-metal3/dev-scripts? Or are you taking other steps?

[juliakreger]

I suspect the only option here is to try ingress policy filters for the pod....

In other words, remove the ip tables statements and we update the yaml loading the pod. Thoughts? @dtantsur @derekhiggins

see: https://kubernetes.io/docs/concepts/services-networking/network-policies/

[dtantsur]

As long as it works - I'm good with it :) I iptables might have been a temporary solution while we were launching ironic directly via podman.

[derekhiggins]

I thought this error could be ignored but maybe i'm wrong,

hmm, in the pod description(above) i don't see mention of privileged (as in https://github.com/metal3-io/baremetal-operator/blob/8169897aeb6580c27671f1b78d6217a541c8e079/deploy/operator_ironic.yaml#L60)
securityContext:
privileged: true

not sure if its relevant tbh..

[derekhiggins]

#83

[dhellmann]

I think #83 fixes this.

[imain]

What environment are you running this on? Is it a baremetal deployment? I'm wondering if 'provisioning interface' is set correctly?

[yprokule]

What environment are you running this on? Is it a baremetal deployment? I'm wondering if 'provisioning interface' is set correctly?

Virtual, deployed with make script from dev-scripts