Node containers can't connect to ETCD

[storageos]

Node containers can't connect to ETCD

[coneking]

Hi @nsoubelet,
I know that your ETCD cluster are insecure but I had the same issue whit tls, in my case the problem was the secret resource storageos-tls-etcd.
When I created storageoscluster for the frst time, a secret was created (storageos-tls-etcd), this secret is used by Daemonset storageos-daemonset to comunicate Pods with the cluster ETCD, this secret contains the tls certs.
Then I uninstalled storageoscluster and ETCD cluster, and install again (by some errors), but this secret still existed and my pods could not connect to the cluster ETCD (same error "{"error":"failed to instantiate ETCD: context deadline exceeded","level":"error","msg":"failed to initialise store client","time":"2020-12-08T21:44:41.780958599Z"}").
To solve this just delete that secret en reinstall storageos and storageoscluster. I hope this helps.

[nsoubelet]

Hi @coneking
I was not able to see that tls secret in my deployment.
However, I found the problem. Basically etcd cluster could not be resolved by service name, only by its IP. To me seems like a problem with the image, since all other containers I started referencing etcd cluster by service name in the same namespace worked ok.

[angelos-p]

Hi @nsoubelet ,

Could you send us over the StorageOS Cluster CR please? You can do that with kubectl describe stos -A

Could you also make sure that you have all the relevant firewall ports open? (https://docs.storageos.com/docs/prerequisites/firewalls/)

Also, is there a reason you are using v2.1? The newest version is v2.3.2.

[nsoubelet]

Hi @aeroniero33
Thanks for replying.
Unfortunately I have uninstalled StorageOS but I will give it another try and send it over.

Could you also make sure that you have all the relevant firewall ports open?

Yes, firewalls open, that's why I clarified points 2 and 3 in my issue.

Also, is there a reason you are using v2.1? The newest version is v2.3.2.

I first tried with latest, then I tried other versions just is case there was a particular issue in that version, but it is failing in all 2.x versions.

[nsoubelet]

Hi @coneking
Many thanks for sharing. Really interesting, I will give it a try for sure!

[nsoubelet]

Hi @coneking, I tried what you said with no luck. I don't see any secret being created by storage os apart from tokens. Maybe you are using a different version?.

@aeroniero33, please find bellow the output of the command. Some sensitive fields were replaced.

Name:         example-storageos
Namespace:    storageos-operator
Labels:       <none>
Annotations:  API Version:  storageos.com/v1
Kind:         StorageOSCluster
Metadata:
  Creation Timestamp:  2021-01-03T19:16:41Z
  Finalizers:
    finalizer.storageoscluster.storageos.com
  Generation:  3
  Managed Fields:
    API Version:  storageos.com/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:debug:
        f:images:
          .:
          f:nodeContainer:
        f:k8sDistro:
        f:kvBackend:
          .:
          f:address:
          f:backend:
        f:nodeSelectorTerms:
        f:resources:
          .:
          f:requests:
            .:
            f:memory:
        f:secretRefName:
        f:secretRefNamespace:
    Manager:      kubectl
    Operation:    Update
    Time:         2021-01-03T19:16:41Z
    API Version:  storageos.com/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
      f:spec:
        f:csi:
          .:
          f:deploymentStrategy:
          f:enable:
          f:enableControllerExpandCreds:
          f:enableControllerPublishCreds:
          f:enableNodePublishCreds:
          f:enableProvisionCreds:
        f:images:
          f:apiManagerContainer:
          f:csiClusterDriverRegistrarContainer:
          f:csiExternalAttacherContainer:
          f:csiExternalProvisionerContainer:
          f:csiExternalResizerContainer:
          f:csiLivenessProbeContainer:
          f:csiNodeDriverRegistrarContainer:
          f:initContainer:
          f:kubeSchedulerContainer:
        f:ingress:
        f:join:
        f:namespace:
        f:resources:
          f:requests:
            f:cpu:
        f:service:
          .:
          f:externalPort:
          f:internalPort:
          f:name:
          f:type:
      f:status:
        .:
        f:members:
          .:
          f:unready:
        f:nodes:
        f:phase:
        f:ready:
    Manager:         cluster-operator
    Operation:       Update
    Time:            2021-01-03T19:16:47Z
  Resource Version:  3106986
  Self Link:         /apis/storageos.com/v1/namespaces/storageos-operator/storageosclusters/example-storageos
  UID:               67436d62-b8f4-4d2b-819b-ed02c2073419
Spec:
  Csi:
    Deployment Strategy:              deployment
    Enable:                           true
    Enable Controller Expand Creds:   true
    Enable Controller Publish Creds:  true
    Enable Node Publish Creds:        true
    Enable Provision Creds:           true
  Debug:                              true
  Images:
    API Manager Container:                   storageos/api-manager:v1.0.0
    Csi Cluster Driver Registrar Container:  quay.io/k8scsi/csi-cluster-driver-registrar:v1.0.1
    Csi External Attacher Container:         quay.io/k8scsi/csi-attacher:v2.2.0
    Csi External Provisioner Container:      storageos/csi-provisioner:v1.6.0-patched
    Csi External Resizer Container:          quay.io/k8scsi/csi-resizer:v0.5.0
    Csi Liveness Probe Container:            quay.io/k8scsi/livenessprobe:v1.1.0
    Csi Node Driver Registrar Container:     quay.io/k8scsi/csi-node-driver-registrar:v1.2.0
    Init Container:                          storageos/init:v2.1.0
    Kube Scheduler Container:                k8s.gcr.io/kube-scheduler:v1.18.4
    Node Container:                          storageos/node:v2.3.1
  Ingress:
  Join:       [some-ip]
  k8sDistro:  upstream
  Kv Backend:
    Address:  [service]:2379
    Backend:  etcd
  Namespace:  kube-system
  Node Selector Terms:
    Match Expressions:
      Key:       [some-key]
      Operator:  In
      Values:
        [some-val]
  Resources:
    Requests:
      Cpu:               1
      Memory:            256Mi
  Secret Ref Name:       storageos-api
  Secret Ref Namespace:  storageos-operator
  Service:
    External Port:  5705
    Internal Port:  5705
    Name:           storageos
    Type:           ClusterIP
Status:
  Members:
    Unready:
      [some-ip]
  Nodes:
    [some-ip]
  Phase:  Creating
  Ready:  0/1
Events:
  Type     Reason         Age    From                       Message
  ----     ------         ----   ----                       -------
  Warning  ChangedStatus  6m10s  storageoscluster-operator  0/1 StorageOS nodes are functional

[coneking]

Hi @nsoubelet,

The secret was created automatically in kube-system

kubectl -n kube-system get secret storageos-tls-etcd

Am using v2.3.1 and v2.3.2 on different clusters

apiVersion: storageos.com/v1
kind: StorageOSCluster
metadata:
  finalizers:
  - finalizer.storageoscluster.storageos.com
  name: example-storageos
  namespace: storageos-operator
spec:
  csi:
    deploymentStrategy: deployment
    enable: true
    enableControllerExpandCreds: true
    enableControllerPublishCreds: true
    enableNodePublishCreds: true
    enableProvisionCreds: true
  images:
    apiManagerContainer: storageos/api-manager:v1.0.0
    csiClusterDriverRegistrarContainer: quay.io/k8scsi/csi-cluster-driver-registrar:v1.0.1
    csiExternalAttacherContainer: quay.io/k8scsi/csi-attacher:v2.2.0
    csiExternalProvisionerContainer: storageos/csi-provisioner:v1.6.0-patched
    csiExternalResizerContainer: quay.io/k8scsi/csi-resizer:v0.5.0
    csiLivenessProbeContainer: quay.io/k8scsi/livenessprobe:v1.1.0
    csiNodeDriverRegistrarContainer: quay.io/k8scsi/csi-node-driver-registrar:v1.2.0
    initContainer: storageos/init:v2.1.0
    kubeSchedulerContainer: k8s.gcr.io/kube-scheduler:v1.16.7
    nodeContainer: storageos/node:v2.3.2
  ingress: {}
  join: ALL-MY-NODES
  k8sDistro: upstream
  kvBackend:
    address: MY-SERVERS-ETCD:2379
    backend: etcd
  namespace: kube-system
  resources:
    requests:
      cpu: "1"
      memory: 512Mi
  secretRefName: storageos-api
  secretRefNamespace: storageos-operator
  service:
    externalPort: 5705
    internalPort: 5705
    name: storageos
    type: ClusterIP
  tlsEtcdSecretRefName: MY-SECRET-TLS
  tlsEtcdSecretRefNamespace: kube-system