[Bug]: Suddenly stopped creating pvc after it started working. PVC stays in pending state!

[deba10106]

[Bug]: Suddenly stopped creating pvc after it started working. PVC stays in pending state!

[vatsa287]

@deba10106 Is the external volume connected/reachable from kadalu-operator, and external volume is up. Please also check if storage-class is deployed and pvc has the same storage-class name on it.

[deba10106]

Yes I checked everything. Volume was up, hosts are reachable via ssh and ping. I checked kds,sc and PVC. No issue with names but still PVC was pending. Then I redeployed the kadalu and it worked. Now, I see an issue with stability. In every few hours, PODs are crashing due to TCP transport endpoint disconnection! Not sure why!

[deba10106]

I guess, I was hit by this gluster/glusterfs#1028 (comment) bug and a fix to this bug is not released yet by glusterfs. A temporary solution to this is gluster/glusterfs#1028 (comment) . I cannot confirm yet. I'll wait another day or two to see if my pods are stable now. I hope, this helps someone.

[deba10106]

ok I can confirm, the issue still persists.

[vatsa287]

@deba10106 Will look into this again.

[leelavg]

Then I deleted the kids, sc and recreated it after changing certain configurations.

• You don't generally delete the storage class
• You delete the CR, operator checks for any in-use PVCs and then only it cleans up the storage, so I believe this is an in-correct action

kubectl-kadalu plugin: devel

• not related to this issue, raised #987

• it seems the logs are from sidecar container not one of kadalu containers

I see an issue with stability. In every few hours, PODs are crashing due to TCP transport endpoint

• I don't remember hearing this upto now, on a fresh cluster with PVCs from external gluster, is this reproducible?

[deba10106]

I've removed all the kadalu artifacts from the cluster one by one. I will try again sometime and will try to reproduce the scenario.

Yes I checked everything. Volume was up, hosts are reachable via ssh and ping. I checked kds,sc and PVC. No issue with names but still PVC was pending. Then I redeployed the kadalu and it worked. Now, I see an issue with stability. In every few hours, PODs are crashing due to TCP transport endpoint disconnection! Not sure why!

This is the main issue. Because pods were getting disconnected every now and then.

[leelavg]

I've removed all the kadalu artifacts from the cluster one by one.

• if you were using external gluster, you need to remove the old metadata that kadalu stores in gluster volume before re-using the vol