Helm Chart installation on AWS EKS: Unable to attach or mount volumes

[DarkoDevops]

Helm Chart installation on AWS EKS: Unable to attach or mount volumes

[klesh]

@IronCore864 @JorgeGar would you mind taking a look at this?

[IronCore864]

@DarkoDevops can you run kubectl get/describe pv/pvc and see what you can get from the output?

A wild guess: do you have the aws-ebs-csi-driver addon enabled in your EKS?

[DarkoDevops]

@IronCore864 I will run it and post it.

Yes, I have aws-ebs-csi-driver addon.

[DarkoDevops]

pv/pvc describe:

pvc-09b75f09-f0e4-4700-b42c-e9f901a13bf2   1Gi        RWO            Delete           Bound    default/devlake-lake-config-devlake-lake-0   gp2                     65d
pvc-16438fe5-4163-441b-be46-2a9d8920b99f   50Gi       RWO            Delete           Bound    default/devlake-mysql-data-devlake-mysql-0   gp2                     65d

Name:              pvc-09b75f09-f0e4-4700-b42c-e9f901a13bf2
Labels:            topology.kubernetes.io/region=eu-central-1
                   topology.kubernetes.io/zone=eu-central-1a
Annotations:       kubernetes.io/createdby: aws-ebs-dynamic-provisioner
                   pv.kubernetes.io/bound-by-controller: yes
                   pv.kubernetes.io/migrated-to: ebs.csi.aws.com
                   pv.kubernetes.io/provisioned-by: kubernetes.io/aws-ebs
Finalizers:        [kubernetes.io/pv-protection external-attacher/ebs-csi-aws-com]
StorageClass:      gp2
Status:            Bound
Claim:             default/devlake-lake-config-devlake-lake-0
Reclaim Policy:    Delete
Access Modes:      RWO
VolumeMode:        Filesystem
Capacity:          1Gi
Node Affinity:     
  Required Terms:  
    Term 0:        topology.kubernetes.io/zone in [eu-central-1a]
                   topology.kubernetes.io/region in [eu-central-1]
Message:           
Source:
    Type:       AWSElasticBlockStore (a Persistent Disk resource in AWS)
    VolumeID:   aws://eu-central-1a/vol-0b203af215cc352d9
    FSType:     ext4
    Partition:  0
    ReadOnly:   false
Events:         <none>

Name:              pvc-16438fe5-4163-441b-be46-2a9d8920b99f
Labels:            topology.kubernetes.io/region=eu-central-1
                   topology.kubernetes.io/zone=eu-central-1b
Annotations:       kubernetes.io/createdby: aws-ebs-dynamic-provisioner
                   pv.kubernetes.io/bound-by-controller: yes
                   pv.kubernetes.io/migrated-to: ebs.csi.aws.com
                   pv.kubernetes.io/provisioned-by: kubernetes.io/aws-ebs
Finalizers:        [kubernetes.io/pv-protection external-attacher/ebs-csi-aws-com]
StorageClass:      gp2
Status:            Bound
Claim:             default/devlake-mysql-data-devlake-mysql-0
Reclaim Policy:    Delete
Access Modes:      RWO
VolumeMode:        Filesystem
Capacity:          50Gi
Node Affinity:     
  Required Terms:  
    Term 0:        topology.kubernetes.io/zone in [eu-central-1b]
                   topology.kubernetes.io/region in [eu-central-1]
Message:           
Source:
    Type:       AWSElasticBlockStore (a Persistent Disk resource in AWS)
    VolumeID:   aws://eu-central-1b/vol-07a37b8cc48ea301f
    FSType:     ext4
    Partition:  0
    ReadOnly:   false
Events:         <none>

[JorgeGar]

I'm running similar values without any issues. I have mysql in an external database, but that's not the one giving you issues, as per what I see in your comment @DarkoDevops. The key difference I guess is that I have aws ebs csi driver installed in the cluster, rather than it being an add-on. Could you try and get logs from it?
Also, which EKS version are you using?

[DarkoDevops]

@JorgeGar
It's 1.25

It's installed, in cluster, also, added policy, not the add-on.

[JorgeGar]

@JorgeGar It's 1.25

It's installed, in cluster, also, added policy, not the add-on.

Have you checked the logs of the attacher?

[DarkoDevops]

This is what I have of the pods for ebs:

ebs-csi-controller-78bdbf98ff-574px   5/5     Running   0            2d5h
ebs-csi-controller-78bdbf98ff-hwktx   5/5     Running   2 (8d ago)   8d
ebs-csi-node-46nh7                    3/3     Running   0            8d
ebs-csi-node-dthdq                    3/3     Running   0            2d5h
ebs-csi-node-fpz98                    3/3     Running   0            8d

[JorgeGar]

Honestly, at this point, I'm following the troubleshooting page from AWS, but...
On the ebs-csi-controller, you should see a container called "csi-attacher". Try getting logs from there.

[BrookeKatalon]

Seeing the same issue with imageTag: v0.17.0-beta5

containers have errors like:
MountVolume.SetUp failed for volume "devlake-mysql-localtime" : hostPath type check failed: /etc/localtime is not a file