rook-ceph cluster osd pod in permanent crash loop CrashLoopBackOff with steps on how to fix without deleting the entire cluster

[algotraders]

rook-ceph cluster osd pod in permanent crash loop CrashLoopBackOff with steps on how to fix without deleting the entire cluster

[jay-johnson]

looks like the cluster actually had a problem back on 5-13-2019 and the redis cluster never reloaded the data afterwards. then the osd pod crashed and took it all down last night (5-28-2019).

[jay-johnson]

The rook-ceph cluster dashboard is showing the cluster looks stable even though the monitor pod restarted and there are multiple osd-prepare pods on here now:

k get po -n rook-ceph
NAME                                          READY   STATUS      RESTARTS   AGE
rook-ceph-mgr-a-68cb58b456-6m8df              1/1     Running     0          58d
rook-ceph-mon-a-855bbddfd4-sxq9m              1/1     Running     0          58d
rook-ceph-mon-b-f949d66dd-rr9lk               1/1     Running     1          58d
rook-ceph-mon-d-5cb4b65d84-tktc7              1/1     Running     0          35m
rook-ceph-osd-0-5dc6b6686f-hmtkd              1/1     Running     1          58d
rook-ceph-osd-1-5fd56d7798-crtbf              1/1     Running     0          39m
rook-ceph-osd-2-5d5965fcc8-l6qz5              1/1     Running     0          58d
rook-ceph-osd-prepare-m10.example.com-pj9p7   0/2     Completed   0          15d
rook-ceph-osd-prepare-m11.example.com-tq8zr   0/2     Completed   0          15d
rook-ceph-osd-prepare-m12.example.com-26p9x   0/2     Completed   2          15d
rook-ceph-tools-bffbf4d8f-znj7q               1/1     Running     0          58d

k get po -n rook-ceph-system
NAME                                 READY   STATUS    RESTARTS   AGE
rook-ceph-agent-d9cdp                1/1     Running   0          58d
rook-ceph-agent-hz6mj                1/1     Running   0          58d
rook-ceph-agent-l9ns4                1/1     Running   2          58d
rook-ceph-operator-d97564799-hzglp   1/1     Running   1          58d
rook-discover-fhg4h                  1/1     Running   0          58d
rook-discover-g7hwd                  1/1     Running   0          58d
rook-discover-mwkcv                  1/1     Running   1          58d

[jay-johnson]

logs from the osd pod that was crashing before the delete:

k logs -n rook-ceph rook-ceph-osd-1-5fd56d7798-fttdh
2019-05-29 18:35:27.969 7f25adf371c0  0 ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) mimic (stable), process ceph-osd, pid 8864
2019-05-29 18:35:27.970 7f25adf371c0  0 pidfile_write: ignore empty --pid-file
starting osd.1 at - osd_data /var/lib/rook/osd1 /var/lib/rook/osd1/journal
2019-05-29 18:35:28.085 7f25adf371c0 -1  Processor -- bind unable to bind to 10.244.2.33:7300/0 on any port in range 6800-7300: (99) Cannot assign requested address
2019-05-29 18:35:28.085 7f25adf371c0 -1  Processor -- bind unable to bind to 10.244.2.33:7300/0 on any port in range 6800-7300: (99) Cannot assign requested address
2019-05-29 18:35:28.086 7f25adf371c0 -1  Processor -- bind was unable to bind. Trying again in 5 seconds 
2019-05-29 18:35:28.086 7f25adf371c0 -1  Processor -- bind was unable to bind. Trying again in 5 seconds 
2019-05-29 18:35:33.116 7f25adf371c0 -1  Processor -- bind unable to bind to 10.244.2.33:7300/0 on any port in range 6800-7300: (99) Cannot assign requested address
2019-05-29 18:35:33.116 7f25adf371c0 -1  Processor -- bind unable to bind to 10.244.2.33:7300/0 on any port in range 6800-7300: (99) Cannot assign requested address
2019-05-29 18:35:33.119 7f25adf371c0 -1  Processor -- bind was unable to bind. Trying again in 5 seconds 
2019-05-29 18:35:33.119 7f25adf371c0 -1  Processor -- bind was unable to bind. Trying again in 5 seconds 
2019-05-29 18:35:38.147 7f25adf371c0 -1  Processor -- bind unable to bind to 10.244.2.33:7300/0 on any port in range 6800-7300: (99) Cannot assign requested address
2019-05-29 18:35:38.147 7f25adf371c0 -1  Processor -- bind unable to bind to 10.244.2.33:7300/0 on any port in range 6800-7300: (99) Cannot assign requested address
2019-05-29 18:35:38.150 7f25adf371c0 -1  Processor -- bind was unable to bind after 3 attempts: (99) Cannot assign requested address
2019-05-29 18:35:38.150 7f25adf371c0 -1  Processor -- bind was unable to bind after 3 attempts: (99) Cannot assign requested address

[jay-johnson]

I never figured out these rook-related crashes for this issue. I ended up moving over to openebs on kubernetes and I never looked back. I do not know if these old issues were the k8 version, the redis version, my kernel (at the time it was 3 something), or something else fun and exciting.

here's the repo:
https://github.com/openebs/openebs