petset-replicas - unclear shutdown

[sclorg]

petset-replicas - unclear shutdown

[omron93]

Deployment of application was done using this template https://github.com/sclorg/mongodb-container/blob/master/examples/petset/mongodb-petset-persistent.yaml

@dongboyan77 Could you provide oc commands how to reproduce this please?

[dongboyan77]

create template oc create -f https://github.com/sclorg/mongodb-container/blob/master/examples/petset/mongodb-petset-persistent.yaml
Then modify template instead of your expected image oc edit template
create app oc new-app mongodb-petset-replication

[omron93]

@dongboyan77 Thanks. And then?

oc scale --replicas=0 statefulsets/mongodb
oc scale --replicas=3 statefulsets/mongodb

is working for me. So I am not possible to reproduce this issue with current docker.io/mongodb-XY-centos7 :-(
Could you provide more information please?

[dongboyan77]

delete all pods
oc delete pod --all

[omron93]

Thanks.

After this I can also see "Detected unclean shutdown - /var/lib/mongodb/data/mongod.lock is not empty." error message in logs. But mongod recovers automatically so deployment does not fail. (so after oc delete pod --all 3 pods starts again)
@dongboyan77 Does oc delete pod --all fail every time?

It seems that even if all cleanup() is run, mongod.lock isn't empty. Also upstream suggests to resync member when recover is needed. So we should think about "more intelligent" way of starting run-mongod-replication when data directory isn't empty.