percona / percona-server-mongodb-operator Goto Github PK

View Code? Open in Web Editor NEW

318.0 44.0 138.0 56.94 MB

Percona Operator for MongoDB

Home Page: https://www.percona.com/doc/kubernetes-operator-for-psmongodb/

License: Apache License 2.0

Go 68.12% Dockerfile 0.68% Shell 30.51% JavaScript 0.07% Makefile 0.63%

kubernetes-operator mongodb-operator k8spsmdb

percona-server-mongodb-operator's Issues

psmdb-operator crashes when psmdb-db is deployed

Report

I'm using both the psmdb-operator and psmdb-db helm charts. I have deployed the operator (without the db deployment) and it was working fine, without errors/crashes.

However, now that I have deployed the db, the operator enters a crashloop. When the operator starts crashlooping, it causes a complete restart of all the pods from psmdb-db as well.

2024-06-12T21:42:34.549Z        INFO    setup   Manager starting up     {"gitCommit": "54e1b18dd9dac8e0ed5929bb2c91318cd6829a48", "gitBranch": "release-1-16-0", "goVersion": "go1.22.3", "os": "linux", "arch": "amd64"}
2024-06-12T21:42:34.565Z        INFO    server version  {"platform": "kubernetes", "version": "v1.28.7+k3s1"}
2024-06-12T21:42:34.570Z        INFO    controller-runtime.metrics      Starting metrics server
2024-06-12T21:42:34.570Z        INFO    starting server {"name": "health probe", "addr": "[::]:8081"}
2024-06-12T21:42:34.570Z        INFO    controller-runtime.metrics      Serving metrics server  {"bindAddress": ":8080", "secure": false}
I0612 21:42:34.570960       1 leaderelection.go:250] attempting to acquire leader lease mongodb/08db0feb.percona.com...
I0612 21:42:53.320941       1 leaderelection.go:260] successfully acquired lease mongodb/08db0feb.percona.com
2024-06-12T21:42:53.321Z        INFO    Starting EventSource    {"controller": "psmdb-controller", "source": "kind source: *v1.PerconaServerMongoDB"}
2024-06-12T21:42:53.321Z        INFO    Starting Controller     {"controller": "psmdb-controller"}
2024-06-12T21:42:53.321Z        INFO    Starting EventSource    {"controller": "psmdbrestore-controller", "source": "kind source: *v1.PerconaServerMongoDBRestore"}
2024-06-12T21:42:53.321Z        INFO    Starting EventSource    {"controller": "psmdbbackup-controller", "source": "kind source: *v1.PerconaServerMongoDBBackup"}
2024-06-12T21:42:53.321Z        INFO    Starting EventSource    {"controller": "psmdbrestore-controller", "source": "kind source: *v1.Pod"}
2024-06-12T21:42:53.321Z        INFO    Starting Controller     {"controller": "psmdbrestore-controller"}
2024-06-12T21:42:53.321Z        INFO    Starting EventSource    {"controller": "psmdbbackup-controller", "source": "kind source: *v1.Pod"}
2024-06-12T21:42:53.321Z        INFO    Starting Controller     {"controller": "psmdbbackup-controller"}
2024-06-12T21:42:53.444Z        INFO    Starting workers        {"controller": "psmdbbackup-controller", "worker count": 1}
2024-06-12T21:42:53.445Z        INFO    Starting workers        {"controller": "psmdb-controller", "worker count": 1}
2024-06-12T21:42:53.445Z        INFO    Starting workers        {"controller": "psmdbrestore-controller", "worker count": 1}
E0612 21:42:53.685207       1 runtime.go:79] Observed a panic: "assignment to entry in nil map" (assignment to entry in nil map)
goroutine 313 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x1f11320, 0x298b1f0})
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:75 +0x85
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000802fc0?})
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:49 +0x6b
panic({0x1f11320?, 0x298b1f0?})
        /usr/local/go/src/runtime/panic.go:770 +0x132
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).setUpdateMongosFirst.func1()
        /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/smart.go:226 +0xd0
k8s.io/client-go/util/retry.OnError.func1()
        /go/pkg/mod/k8s.io/[email protected]/util/retry/util.go:51 +0x30
k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection(0x411b9b?)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:145 +0x3e
k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff({0x989680, 0x4014000000000000, 0x3fb999999999999a, 0x4, 0x0}, 0xc000baaa18)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:461 +0x5a
k8s.io/client-go/util/retry.OnError({0x989680, 0x4014000000000000, 0x3fb999999999999a, 0x4, 0x0}, 0x4171ba?, 0x0?)
        /go/pkg/mod/k8s.io/[email protected]/util/retry/util.go:50 +0xa5
k8s.io/client-go/util/retry.RetryOnConflict(...)
        /go/pkg/mod/k8s.io/[email protected]/util/retry/util.go:104
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).setUpdateMongosFirst(0x1ef45e0?, {0x29affe0?, 0xc0011ad140?}, 0x6?)
        /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/smart.go:220 +0xbc
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).createSSLByCertManager(0xc000b882d0, {0x29affe0, 0xc0011ad140}, 0xc000dcaf08)
        /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/ssl.go:187 +0x794
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileSSL(0xc000b882d0, {0x29affe0, 0xc0011ad140}, 0xc000dcaf08)
        /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/ssl.go:66 +0x30d
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile(0xc000b882d0, {0x29affe0, 0xc0011ad140}, {{{0xc0006dade8?, 0x5?}, {0xc0006dade0?, 0xc000d25d10?}}})
        /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:368 +0x16d0
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x29b4dc8?, {0x29affe0?, 0xc0011ad140?}, {{{0xc0006dade8?, 0xb?}, {0xc0006dade0?, 0x0?}}})
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114 +0xb7
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000b820b0, {0x29b0018, 0xc0009c03c0}, {0x1fdf1a0, 0xc000dd27a0})
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311 +0x3bc
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000b820b0, {0x29b0018, 0xc0009c03c0})
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261 +0x1be
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222 +0x79
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 141
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:218 +0x486
2024-06-12T21:42:53.730Z        INFO    Observed a panic in reconciler: assignment to entry in nil map  {"controller": "psmdb-controller", "object": {"name":"psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "psmdb-db", "reconcileID": "7676acba-b62f-4d00-a4dc-51c0e17bc27c"}
panic: assignment to entry in nil map [recovered]
        panic: assignment to entry in nil map [recovered]
        panic: assignment to entry in nil map

goroutine 313 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:111 +0x1e5
panic({0x1f11320?, 0x298b1f0?})
        /usr/local/go/src/runtime/panic.go:770 +0x132
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000802fc0?})
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:56 +0xcd
panic({0x1f11320?, 0x298b1f0?})
        /usr/local/go/src/runtime/panic.go:770 +0x132
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).setUpdateMongosFirst.func1()
        /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/smart.go:226 +0xd0
k8s.io/client-go/util/retry.OnError.func1()
        /go/pkg/mod/k8s.io/[email protected]/util/retry/util.go:51 +0x30
k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection(0x411b9b?)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:145 +0x3e
k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff({0x989680, 0x4014000000000000, 0x3fb999999999999a, 0x4, 0x0}, 0xc000efea18)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:461 +0x5a
k8s.io/client-go/util/retry.OnError({0x989680, 0x4014000000000000, 0x3fb999999999999a, 0x4, 0x0}, 0x4171ba?, 0x0?)
        /go/pkg/mod/k8s.io/[email protected]/util/retry/util.go:50 +0xa5
k8s.io/client-go/util/retry.RetryOnConflict(...)
        /go/pkg/mod/k8s.io/[email protected]/util/retry/util.go:104
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).setUpdateMongosFirst(0x1ef45e0?, {0x29affe0?, 0xc0011ad140?}, 0x6?)
        /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/smart.go:220 +0xbc
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).createSSLByCertManager(0xc000b882d0, {0x29affe0, 0xc0011ad140}, 0xc000dcaf08)
        /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/ssl.go:187 +0x794
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileSSL(0xc000b882d0, {0x29affe0, 0xc0011ad140}, 0xc000dcaf08)
        /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/ssl.go:66 +0x30d
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile(0xc000b882d0, {0x29affe0, 0xc0011ad140}, {{{0xc0006dade8?, 0x5?}, {0xc0006dade0?, 0xc000d25d10?}}})
        /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:368 +0x16d0
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x29b4dc8?, {0x29affe0?, 0xc0011ad140?}, {{{0xc0006dade8?, 0xb?}, {0xc0006dade0?, 0x0?}}})
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114 +0xb7
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000b820b0, {0x29b0018, 0xc0009c03c0}, {0x1fdf1a0, 0xc000dd27a0})
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311 +0x3bc
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000b820b0, {0x29b0018, 0xc0009c03c0})
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261 +0x1be
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222 +0x79
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 141
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:218 +0x486

Steps to reproduce

Deploy psmdb-operator via helm
Deploy psmdb-db via helm
Wait until deployment finishes, operator will start crashlooping shortly

Versions

Kubernetes: v1.28.7
Operator: 1.16.0
Database: 7.0.8-5

Anything else?

Deployment is handled via ArgoCD.

Proposal

test

Use-Case

test

Is this a feature you are interested in implementing yourself?

Yes

Anything else?

test

Report

When update secret for backup storage in kind PerconaServerMongoDB updates are not applied to PBM backup configuration
When update endpointUrl backup storages s3 updates are not applied to PBM backup configuration###

Steps to reproduce

Create cluster using PerconaServerMongoDB with s3 backup and SecretV1 and endpointUrlV1
1. Check configutaion in backup-agent using pbm config
Update PerconaServerMongoDB and use for s3 backup SecretV2 and endpointUrlV2
Check configutaion in backup-agent using pbm config

Versions

Kubernetes - v1.27.7
Operator - percona/percona-server-mongodb-operator:1.15.0
Database - percona/percona-backup-mongodb:2.3.0

Anything else?

No response

Failure to start cluster with `unmanaged: true` on latest versions

I opened this issue when I attempted with previous version, but even latest versions do not work, so updated title.

Report

Creation of cluster with unmanaged: true fails, mongodb nodes bootloop.

Operator continues to log the following repeatedly as the mongo node bootloop.
INFO Replset is not exposed. Make sure each pod in the replset can reach each other. {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "4071c15b-9595-443f-bb20-5705204cbd3d", "replset": "rs0"}

Steps to reproduce

Deploy yaml as described, wait for failure.

Versions

Kubernetes: v1.26.13 +rke2r1
Operator: First tried with 1.16.1, upgraded to 1.16.3, no change

Anything else?

The DB I am attempting to migrate requires a target of 4.4, hence the cr version selection.

The deployment works without unmanaged: true

Two pod logs from start up to bootloop
example-mongodb-rs0-0_mongod.log
example-mongodb-rs0-1_mongod.log

Operator logs:

2024-07-24T18:21:17.244954170-04:00 2024-07-24T22:21:17.244Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "de793d73-e501-4bac-93cd-351957acaa4a", "replset": "rs0"}
2024-07-24T18:21:17.519518649-04:00 2024-07-24T22:21:17.518Z	INFO	Created a new mongo key	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "de793d73-e501-4bac-93cd-351957acaa4a", "KeyName": "example-mongodb-mongodb-keyfile"}
2024-07-24T22:21:17.535Z	INFO	Created a new mongo key	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "de793d73-e501-4bac-93cd-351957acaa4a", "KeyName": "example-mongodb-mongodb-encryption-key"}
2024-07-24T18:21:17.699208811-04:00 2024-07-24T22:21:17.698Z	INFO	Waiting for the pods	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "de793d73-e501-4bac-93cd-351957acaa4a", "replset": "rs0", "size": 3, "pods": 1}
2024-07-24T18:21:17.751281423-04:00 2024-07-24T22:21:17.750Z	INFO	add new job	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "de793d73-e501-4bac-93cd-351957acaa4a", "name": "ensure-version/mongodb/example-mongodb", "schedule": "0 2 * * *"}
2024-07-24T18:21:17.779567104-04:00 2024-07-24T22:21:17.779Z	INFO	Cluster state changed	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "de793d73-e501-4bac-93cd-351957acaa4a", "previous": "", "current": "initializing"}
2024-07-24T18:21:17.816437308-04:00 2024-07-24T22:21:17.815Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "a48c61c2-ac90-4f17-b581-cf8d1a3c5dbe", "replset": "rs0"}
2024-07-24T18:21:18.096219954-04:00 2024-07-24T22:21:18.095Z	INFO	Waiting for the pods	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "a48c61c2-ac90-4f17-b581-cf8d1a3c5dbe", "replset": "rs0", "size": 3, "pods": 
1}
2024-07-24T18:21:22.815267111-04:00 2024-07-24T22:21:22.814Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "97ee1bca-d292-44c0-ae9b-4adf1dc2570d", "replset": "rs0"}
2024-07-24T22:21:23.071Z	INFO	Waiting for the pods	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "97ee1bca-d292-44c0-ae9b-4adf1dc2570d", "replset": "rs0", "size": 3, "pods": 1}
2024-07-24T18:21:28.187110713-04:00 2024-07-24T22:21:28.186Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "9be35d66-6137-4d9d-aa13-2d98d1658fc2", "replset": "rs0"}
2024-07-24T18:21:28.489985716-04:00 2024-07-24T22:21:28.489Z	INFO	Waiting for the pods	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "9be35d66-6137-4d9d-aa13-2d98d1658fc2", "replset": "rs0", "size": 3, "pods": 1}
2024-07-24T18:21:33.639456067-04:00 2024-07-24T22:21:33.639Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "10c49def-f0d8-4766-96e6-adf1c185f949", "replset": "rs0"}
2024-07-24T22:21:33.943Z	INFO	Waiting for the pods	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "10c49def-f0d8-4766-96e6-adf1c185f949", "replset": "rs0", "size": 3, "pods": 1}
2024-07-24T22:21:39.363Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "21bb72e2-b804-4246-9ce7-45b87bce823d", "replset": "rs0"}
2024-07-24T22:21:39.630Z	INFO	Waiting for the pods	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "21bb72e2-b804-4246-9ce7-45b87bce823d", "replset": "rs0", "size": 3, "pods": 1}
2024-07-24T22:21:45.632Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "e8703952-b61d-47a0-9193-27b57146df16", "replset": "rs0"}
2024-07-24T22:21:45.919Z	INFO	Waiting for the pods	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "e8703952-b61d-47a0-9193-27b57146df16", "replset": "rs0", "size": 3, "pods": 2}
2024-07-24T22:21:46.084Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "14129a56-3d0e-47fc-981a-7c86b951d285", "replset": "rs0"}
2024-07-24T22:21:46.405Z	INFO	Waiting for the pods	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "14129a56-3d0e-47fc-981a-7c86b951d285", "replset": "rs0", "size": 3, "pods": 2}
2024-07-24T22:21:51.312Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "c667ba1e-1d82-4850-b17f-759d61e0e4dd", "replset": "rs0"}
2024-07-24T22:21:51.619Z	INFO	Waiting for the pods	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "c667ba1e-1d82-4850-b17f-759d61e0e4dd", "replset": "rs0", "size": 3, "pods": 2}
2024-07-24T22:21:57.004Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "fce820a1-c739-44e9-b578-47c9e3defcee", "replset": "rs0"}
2024-07-24T22:21:57.282Z	INFO	Waiting for the pods	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "fce820a1-c739-44e9-b578-47c9e3defcee", "replset": "rs0", "size": 3, "pods": 2}
2024-07-24T22:22:02.817Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "79d72aa6-f450-41f9-8997-e8869282f337", "replset": "rs0"}
2024-07-24T22:22:03.090Z	INFO	Waiting for the pods	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "79d72aa6-f450-41f9-8997-e8869282f337", "replset": "rs0", "size": 3, "pods": 2}
2024-07-24T22:22:08.509Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "fbcd6837-1e99-49c7-a801-37a6ab0d143d", "replset": "rs0"}
2024-07-24T22:22:08.795Z	INFO	Waiting for the pods	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "fbcd6837-1e99-49c7-a801-37a6ab0d143d", "replset": "rs0", "size": 3, "pods": 2}
2024-07-24T22:22:14.213Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "10fdccca-4165-452b-8b56-043fdf63f8db", "replset": "rs0"}
2024-07-24T22:22:14.511Z	INFO	Waiting for the pods	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "10fdccca-4165-452b-8b56-043fdf63f8db", "replset": "rs0", "size": 3, "pods": 2}
2024-07-24T22:22:19.927Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "8a52d2f9-57a4-4a20-94a9-507980a8a66b", "replset": "rs0"}
2024-07-24T22:22:31.851Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "6e9ad316-6787-4cb8-8146-ed34accbea83", "replset": "rs0"}
2024-07-24T22:22:43.739Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "41260c8c-bd30-4b66-a827-5fbb1a2b7081", "replset": "rs0"}
2024-07-24T22:22:55.687Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "7c3eb48f-59d5-4892-a2d6-f75118109e5c", "replset": "rs0"}
2024-07-24T22:23:07.532Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "727a406f-4f22-4e05-b1f3-7d4856077402", "replset": "rs0"}
2024-07-24T22:23:19.398Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "18cf442f-00ab-4605-82e3-4eec2c330abd", "replset": "rs0"}
2024-07-24T22:23:31.221Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "d018f4e4-efa9-4749-95b4-fdd6e9929538", "replset": "rs0"}
2024-07-24T22:23:43.030Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "5ebc4aed-b855-4483-b404-58580864283d", "replset": "rs0"}
2024-07-24T22:23:54.899Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "8b2c2559-2804-4388-a798-732bd5986536", "replset": "rs0"}
2024-07-24T22:24:06.718Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "1680e0de-9a6f-4669-9b8c-8a88e98c5397", "replset": "rs0"}
2024-07-24T22:24:18.539Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "81a4d31a-1abc-4eaa-8b22-823200d9c57f", "replset": "rs0"}
2024-07-24T22:24:30.317Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "58e30cba-39a6-4d2a-88af-2cb736dce724", "replset": "rs0"}
2024-07-24T22:24:43.680Z	INFO	Replset is not exposed. Make sure each pod in the replset can reach each other.	{"controller": "psmdb-controller",

PVC remains in Pending state when deploying MongoDB with Percona Server for MongoDB Operator it does not automatically set the default storageClassName

Report

Hello,

I am experiencing an issue where the Persistent Volume Claim (PVC) remains in the Pending state when deploying MongoDB using the Percona Server for MongoDB Operator. The PVC does not automatically get bound to a Persistent Volume (PV), despite having a default storage class configured in the Kubernetes cluster.

Steps to Reproduce

Deploy MongoDB using the Percona Server for MongoDB Operator with the following configuration:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: p-pm-mongodbtest
spec:
  project: default
  destination:
    namespace: p-pm-mongodbtest
    server: https://kubernetes.default.svc
  syncPolicy:
    automated: {}
    syncOptions:
      - CreateNamespace=true
  source:
    repoURL: https://charts.rock8s.com
    chart: mongodb
    targetRevision: "1.15.0"
    helm:
      values: |
        config:
          debug: false
          istio: false
          mongodb:
            replicas: 1
            resources:
              requests:
                cpu: 300m
                memory: 0.5G
              limits:
                cpu: 300m
                memory: 0.5G
          pmm:
            enabled: false
        service:
          mongodb:
            type: ClusterIP
            tls:
              enabled: false
        persistence:
          accessMode: ReadWriteOnce
          size: 1Gi
          storageClassName: ""
          kanister:
            enabled: false
            schedule: '0 0 * * *'

Check the status of the PVC:

kubectl get pvc -n p-pm-mongodbtest

Expected Results

The PVC should be bound to a PV and not remain in the Pending state.

Actual Results

The PVC remains in the Pending state and is not bound to any PV. Here are the details of the PVC:

kubectl describe pvc mongod-data-mongodb-rs0-0 -n p-pm-mongodbtest

Name:          mongod-data-mongodb-rs0-0
Namespace:     p-pm-mongodbtest
StorageClass:  
Status:        Pending
Volume:        
Labels:        app.kubernetes.io/component=mongod
               app.kubernetes.io/instance=mongodb
               app.kubernetes.io/managed-by=percona-server-mongodb-operator
               app.kubernetes.io/name=percona-server-mongodb
               app.kubernetes.io/part-of=percona-server-mongodb
               app.kubernetes.io/replset=rs0
Annotations:   <none>
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Used By:       mongodbtest-rs0-0
Events:        <none>

Additional Information

kubectl get storageclass

NAME                      PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
default                   kubernetes.io/aws-ebs   Delete          Immediate              false                  190d
efs-sc                    efs.csi.aws.com         Delete          Immediate              false                  142d
gp2                       kubernetes.io/aws-ebs   Delete          Immediate              false                  190d
kops-csi-1-21 (default)   ebs.csi.aws.com         Delete          WaitForFirstConsumer   true                   190d
kops-ssd-1-17             kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   true                   190d

Workaround

As a temporary workaround, I manually specified the storage class in the PVC definition, which allowed it to get bound:

spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 4Gi
  storageClassName: "kops-csi-1-21"
  volumeMode: Filesystem

I would appreciate any guidance on how to resolve this issue and ensure the PVC gets bound automatically without needing to specify the storage class explicitly.

Thank you!

Versions

Kubernetes: v1.22.0
Operator: Percona Server for MongoDB Operator 1.15.0
Database: Percona Server for MongoDB 4.4.6-8

Client Version: v1.30.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.26.11

storageClassName not ending up in PVC

Hi! I want to make sure my pvcs use a storageclass with allowVolumeExpansion, so I've added these helm values to make sure the storageclass is getting set:

 replsets:
- name: rs0
  volumeSpec:
    pvc:
      storageClassName: custom
sharding:
  configrs:
    volumeSpec:
      pvc:
        storageClassName: custom

And I'm seeing the pvcTemplate ending up in the corresponding statefulsets:

  volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      creationTimestamp: null
      name: mongod-data
      namespace: percona
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 3Gi
      storageClassName: custom
      volumeMode: Filesystem
    status:
      phase: Pending

But unfortunately, the cfg pvcs seem to be created w the EKS default storage class:

% k get pvc
NAME                        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
mongod-data-mongodb-cfg-0   Bound    pvc-5c40c40a-d1ce-4636-b3e2-1e9c36755047   3Gi        RWO            gp2            64m
mongod-data-mongodb-cfg-1   Bound    pvc-d3b39739-780a-4358-8487-1db81dd9eb5e   3Gi        RWO            gp2            63m
mongod-data-mongodb-cfg-2   Bound    pvc-115a1fd6-bd83-4235-beea-f366aa18bb3d   3Gi        RWO            gp2            63m
mongod-data-mongodb-rs0-0   Bound    pvc-e966d021-f864-4122-9c3a-deb85118a252   3Gi        RWO            custom         64m
mongod-data-mongodb-rs0-1   Bound    pvc-dc2846c4-4bf1-437d-b061-2e2feb372820   3Gi        RWO            custom         63m
mongod-data-mongodb-rs0-2   Bound    pvc-6b3868e3-d880-47b6-923f-b635278f0cea   3Gi        RWO            custom         63m

Any idea why that would be? I've deleted and redeployed a few times. Any ideas?

requireTLS is ignored in "unsafe" mode

Report

setting

spec:
  allowUnsafeConfigurations: true
  replsets:
    configuration: |
      net:
        tls:
          mode: requireTLS

means that requireTLS is (silently) ignored. From the code this appears to be because "unsafe" means both "less than 3 replicas" (I would like to use a PSA config) and also "don't use TLS certificates for mongo replica authentication"

I would suggest either splitting this flag into two to allow for a PSA config that requires tls, or leaving it and considering a PSA configuration safe.

Steps to reproduce

apply a config as above
check mongo parameters in the container, observe that requireTLS is not set

Versions

1.28.3
1.15
mongo 6.0.9-7

Anything else?

No response

Issue with AWS IRSA Configuration for Restore: S3 Object Header Forbidden (403)

Report

When configuring AWS IAM Roles for Service Accounts (IRSA) for backups in the Percona MongoDB Operator, the backup process works correctly. However, restoring from the backup fails with a "Forbidden" error when trying to access S3 objects.

Steps to reproduce

Configure AWS IRSA:
Set up IAM roles and policies for the Percona MongoDB Operator.
Annotate the Kubernetes service account with the IAM role ARN.
refer to this doumentation: https://docs.percona.com/percona-backup-mongodb/manage/automate-s3-access.html#iam-roles-for-service-accounts-irsa
Configure Backup:
Set up the Percona MongoDB helm chart to use IRSA for S3 backups.
backups are created and listed successfully.
backups are configured with psmdb-db helm chart

backup:
  enabled: true
  pitr:
    enabled: false
  storages:
    s3-us-west:
      type: s3
      s3:
        bucket: S3-BACKUP-BUCKET-NAME-HERE
        credentialsSecret: "" # empty and it works, the backup use the aws IRSA
        prefix: "someprefix"
        region: us-west-3

Initiate a restore process using the PerconaServerMongoDBRestore CRD.

apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDBRestore
metadata:
  name: restore1
spec:
  clusterName: my-cluster-name
  backupName: backup1
  storageName: s3-us-west
 backupSource:
   type: logical
   storageName: s3-us-west
   destination: s3://S3-BACKUP-BUCKET-NAME-HERE/BACKUP-DESTINATION
   s3:
     credentialsSecret: "" # empty, i want to use the aws IRSA
     region: us-west-2
     bucket: S3-BACKUP-BUCKET-NAME-HERE
     prefix: "some prefix"

Versions

Kubernetes: 1.29
Operator: 1.16.1
Helm chart psmdb-db: 1.16.2
Database: mongodb 7.0.8-5

Anything else?

No response

Node restarting during Index Build

Report

Hello,

We are deploying a sharded cluster in EKS containing 3 node. Two of the nodes are working as expected, however, the third is not. It stopped working after a restart.
It's building the Indexes when it suddenly restart and goes into crash loop.

I believe that it might have something to do with liveness probe.

I've altered the liveness probe directly on the operator, but it has not propagated.

Steps to reproduce

Versions

Kubernetes = 1.28
Operator = 1.15
Database = MongoDB

Anything else?

No response

PerconaServerMongoDBRestore: ERROR oplog no such file

Report

I am trying to configure MongoDB backup/restore using Percona Operator. Backups are being successfully uploaded to GCS:

Backups:
========
S3 us s3://https://storage.googleapis.com/mongodb-macpaw-staging-backup/test
  Snapshots:
    2024-08-01T10:23:21Z 26.75KB <logical> [restore_to_time: 2024-08-01T10:23:32Z]

When I try to create PerconaServerMongoDBRestore resource to restore DB from previously created backup I get an error:

Backups:
========
S3 us s3://https://storage.googleapis.com/mongodb-macpaw-staging-backup/test
  Snapshots:
    2024-08-01T10:23:21Z 26.75KB <logical> [ERROR: file "2024-08-01T10:23:21Z/test-rs1/oplog": no such file] [2024-08-01T10:23:36Z]

Steps to reproduce

Cluster backup config:

backup:
    enabled: true
    image: perconalab/percona-server-mongodb-operator:main-backup
    serviceAccountName: <serviceAccountName>
    storages:
      gcs:
        type: s3
        s3:
          region: us
          credentialsSecret: <credentialsSecret>
          bucket: <bucket>
          endpointUrl: https://storage.googleapis.com
          prefix: <prefix>
    pitr:
      enabled: true
    tasks:
      - name: daily
        enabled: true
        schedule: "0 11 * * *"
        keep: 3
        storageName: gcs

PerconaServerMongoDBRestore manifest

apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDBRestore
metadata:
  name: restore1
  namespace: mongodb
spec:
  clusterName: test
  backupName: cron-test-20240802124900-fjk52
  backupSource:
    type: physical
    destination: s3://<bucket>/<prefix>
    s3:
      region: us
      credentialsSecret: <credentialsSecret>
      bucket: <bucket>
      endpointUrl: https://storage.googleapis.com
      prefix: <prefix>

Versions

Kubernetes - 1.29 GKE
Operator - 1.5.0
Database - percona/percona-server-mongodb:6.0.5-4

Anything else?

No response

Cannot scale up mongodb instances storage from the CR definition of the psmdb cluster

Proposal

If we try to increase the size of the replset mongodb instances storage defined in the spec.replsets[*].volumeSpec.persistentVolumeClaim.resources.requests.storage or spec.sharding.configsvrReplSet.volumeSpec.persistentVolumeClaim.resources.requests.storage CR attributes, the operator is not able to update the related replset statefulsets and pvc as we can see from its logs :

"Forbidden: updates to statefulset spec for fields other than …"

As far as I know, the only way to scale up the storage size of these instances is to directly update the related pvc definition, assuming that these pvc are provided by storageclasses allowing the volume expansion of course. But this is not very convenient as the CR definition and related statefulsets are not updated behind.

It would be a great improvement to allow volume expansion from the CR definition by creating external pvc attached to the pod using Claims As Volumes method as implemented with the PostgreSQL operator instead of the dynamic pvc creation from statefulset definition with the Volume Claim Templates method.

Use-Case

Steps to reproduce this issue :
1 - Create a psmdb cluster with one replset configure with its storage size
2 - Wait for the cluster to be ready and see the newly created pvc with the desired storage size
3 - Try to scale up the instances by increasing spec.replsets[0].volumeSpec.persistentVolumeClaim.resources.requests.storage attribute in the CR definition
4 - See that the pvc storage size is not updated because the operator is unable to update the related statefulset definition as mentioned in its logs

Is this a feature you are interested in implementing yourself?

Anything else?

No response

Support for accessing syslog and mounting additional volumes for audit collection

Proposal

Add the ability to configure syslog and add option to mount additional volumes, so i can use a sidecar like fluentbit to collect audit logs

Use-Case

Currently i cannot access audit logs in syslog or file format as i cannot access these directories from a sidecar. Having the option to modify rsyslog.conf or mounting volumes on the mongod pod will allow me to collect the logs using fluentbit

In the existing operator, we can add volumes and mounts to sidecars, but there is no extraVolumes or additionalVolumes property available for the mongod replicaset itself, so i cannot expose directories for fluentbit to collect from

For our SIEM monitoring, we have a requirement to collect audit events from MongoDB

Is this a feature you are interested in implementing yourself?

Maybe

Anything else?

No response

secrets are deleted when delete-psmdb-pvc has been set

percona-server-mongodb-operator/pkg/controller/perconaservermongodb/finalizers.go

Line 204 in 7b1610a

err = r.deleteSecrets(ctx, cr)

secrets are deleted when delete-psmdb-pvc has been set.
the problem is sometimes the percona secrets (cr.Spec.Secrets.Users,
"internal-" + cr.Name + "-users") are respectively being recreated right after the deletion in reconcileUsersSecret, reconcileUsers func calls. Sometimes not.

This logic of deleting secrets causes issues/side effects in tha case that the psmdb CR might be re-created after to recover a deleted cluster for example.

To work around this logic we try to overwrite/create the user secret with the previous data (leaving to the operator to sync the secrets, etc), but sometimes recreating psmdb lateron ends up creating the mongodb pods successfully with some auth errors on the mongo pods though, sometimes not. the psmdb does not go in initializing status even. (there are logs on the operator: not found the internal users secret.)

Operator not using IRSA Service Account IAM Role for restore and backup reconcile

Report

Hi,
We use the operator to do automatic backups to S3 buckets on EKS, which works fine, as the backups are done by the agent container running in the mongodb pods.

We use IRSA, so the EKS pods are using the AWS IAM roles to access resources like S3 buckets, which is working fine on the agent container, but it seems that it's not working on the operator.

Steps to reproduce

Create EKS cluster
Create role/profile for the nodes running mongodb
Install percona opertor via helm
Add the IRSA annotation to the mongodb serviceaccount (or to default)
Add the IRSA annotation to the operator's serviceaccount
Create a backup -> works fine
Restore backup -> fails
Reconcile backup -> fails

Versions

Kubernetes: 1.26
Operator: 1.14.3, 1.15.0
Database: 5.0.15-13

Anything else?

No response

Backups/Restores are in Waiting Status after Kubernetes scheduler restarted the backup-agent container

Report

MongoDB Backup is stuck on Status:Waiting and backup-agent container is not doing anything after Kubernetes scheduler restarted the backup-agent container during the execution of a restore:

Steps to reproduce

Start a MongoDB cluster in unsafe mode with only 1 replica (this is usefull for development environments) and fill it with some data (let's say about 600MB of gzipped data);

Do a MongoDB backup and wait for the completion (Status = Ready) with the following yml (this will upload the backup to our AWS S3 bucket):

apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDBBackup
metadata:
  finalizers:
    - delete-backup
  name: backup1
spec:
  clusterName: mongodb-percona-cluster
  storageName: eu-central-1
  type: logical

Drop collections on MongoDB replicaset (just to avoid the _id clashes at next point);

Now ask for a restore of the above backup with the following yml (this works as intended since I saw the logs and the data inside MongoDB ReplicaSet):

apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDBRestore
metadata:
  name: restore1
spec:
  clusterName: mongodb-percona-cluster
  backupName: backup1

Ask for another backup with the following yml (keep in mind that at this point the previous restore process is still in progress)

apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDBBackup
metadata:
  finalizers:
    - delete-backup
  name: backup2
spec:
  clusterName: mongodb-percona-cluster
  storageName: eu-central-1
  type: logical

The backup2 will be put on Status=Waiting;

At this point Kubernetes scheduler should kill the backup-agent container from the MongoDB replica pod because of memory issues and restart it;

Now if you do a kubectl get psmdb-backup, you'll see that backup2 is in Error status and if you do a kubectl get psmdb-restore, you'll see that restore1 is also in Error status (OK, I can take that);

From this point onwards, no backup/restore will be possible through any yml, because they'll be appended as Status=Waiting.

The new backup-agent container logs state that it is waiting for incoming requests:

2024/03/05 16:36:01 [entrypoint] starting `pbm-agent`
2024-03-05T16:36:05.000+0000 I pbm-agent:
Version:   2.3.0
Platform:  linux/amd64
GitCommit: 3b1c2e263901cf041c6b83547f6f28ac2879911f
GitBranch: release-2.3.0
BuildTime: 2023-09-20_14:42_UTC
GoVersion: go1.19
2024-03-05T16:36:05.000+0000 I starting PITR routine
2024-03-05T16:36:05.000+0000 I node: rs0/mongodb-percona-cluster-rs0-0.mongodb-percona-cluster-rs0.default.svc.cluster.local:27017
2024-03-05T16:36:05.000+0000 I listening for the commands

Versions

Kubernetes version v1.27.9 in a 8 nodes cluster with 4GB of RAM each, in Azure Cloud
Operator image percona/percona-server-mongodb-operator:1.15.0
Database image percona/percona-server-mongodb:5.0.20-17

Anything else?

Same bug applies also on cronjobs (so it's not an issue triggered by the on demand backup/restore requests): they are kept in Waiting status.
The bug does NOT happen when using a ReplicaSet with at least 3 replicas (the default topology).

[FEAT]: Arm64 support

This is written in Go, it should be 1h tops to add arm64 support to the container image. Please do this.

Error for setting up backup

Report

I am trying to setup backup for my Percona MongoDB. I notice percona-mongodb-operator has error log once I have setup the backup through CRD.

Steps to reproduce

kubectl apply -f https://raw.githubusercontent.com/percona/percona-server-mongodb-operator/v1.16.1/deploy/cw-bundle.yaml
Apply the following CRD:

---
apiVersion: v1
kind: Namespace
metadata:
  name: psmdb
  labels:
    pod-security.kubernetes.io/enforce: baseline
---
apiVersion: v1
kind: Secret
metadata:
  name: azure-blob-secret
  namespace: psmdb
type: Opaque
stringData:
  AZURE_STORAGE_ACCOUNT_NAME: "$AZURE_STORAGE_ACCOUNT_NAME"
  AZURE_STORAGE_ACCOUNT_KEY: "$AZURE_STORAGE_ACCOUNT_KEY"
---
apiVersion: v1
kind: Secret
metadata:
  name: mongodb-user-secret
  namespace: psmdb
type: Opaque
stringData:
  MONGODB_BACKUP_USER: "$MONGODB_BACKUP_USER"
  MONGODB_BACKUP_PASSWORD: "$MONGODB_BACKUP_PASSWORD"
  MONGODB_DATABASE_ADMIN_USER: "$MONGODB_DATABASE_ADMIN_USER"
  MONGODB_DATABASE_ADMIN_PASSWORD: "$MONGODB_DATABASE_ADMIN_PASSWORD"
  MONGODB_CLUSTER_ADMIN_USER: "$MONGODB_CLUSTER_ADMIN_USER"
  MONGODB_CLUSTER_ADMIN_PASSWORD: "$MONGODB_CLUSTER_ADMIN_PASSWORD"
  MONGODB_CLUSTER_MONITOR_USER: "$MONGODB_CLUSTER_MONITOR_USER"
  MONGODB_CLUSTER_MONITOR_PASSWORD: "$MONGODB_CLUSTER_MONITOR_PASSWORD"
  MONGODB_USER_ADMIN_USER: "$MONGODB_USER_ADMIN_USER"
  MONGODB_USER_ADMIN_PASSWORD: "$MONGODB_USER_ADMIN_PASSWORD"
---
apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDB
metadata:
  name: mongodb-cluster
  namespace: psmdb
  finalizers:
    - delete-psmdb-pods-in-order
spec:
  platform: kubernetes
  crVersion: 1.16.1
  image: percona/percona-server-mongodb:7.0.11-multi
  imagePullPolicy: IfNotPresent
  initImage: percona/percona-server-mongodb-operator:1.16.1
  clusterServiceDNSMode: Internal
  multiCluster:
    enabled: false
  secrets:
    users: mongodb-user-secret
    encryptionKey: mongodb-encryption-secret
  unsafeFlags:
    tls: false
    # set this to false, and replsets.size to the number of node, if we have a bigger node pool
    replsetSize: true
  tls:
    mode: preferTLS
    allowInvalidCertificates: false
  upgradeOptions:
    apply: Disabled

  replsets:
    - name: rs0
      size: 1
      serviceAccountName: default
      livenessProbe:
        failureThreshold: 4
        initialDelaySeconds: 60
        periodSeconds: 30
        timeoutSeconds: 10
        startupDelaySeconds: 7200
      readinessProbe:
        failureThreshold: 8
        initialDelaySeconds: 10
        periodSeconds: 3
        successThreshold: 1
        timeoutSeconds: 2
      volumeSpec:
        persistentVolumeClaim:
          accessModes: [ "ReadWriteOnce" ]
          storageClassName: managed-csi
          resources:
            requests:
              # NOTE it has to exactly match the size of the PV
              storage: 5Gi
      expose:
        enabled: true
        exposeType: ClusterIP
      nonvoting:
        enabled: false
        size: 1
      arbiter:
        enabled: false
        size: 1
  backup:
    enabled: true
    image: percona/percona-backup-mongodb:2.5.0-multi
    resources:
      limits:
        cpu: 300m
        memory: 1Gi
      requests:
        cpu: 100m
        memory: 0.5Gi
    containerSecurityContext:
      privileged: false
    storages:
      azure-blob:
        type: azure
        azure:
          container: supportassistant
          prefix: psmdb
          credentialsSecret: azure-blob-secret
    pitr:
      enabled: false
    tasks:
      - name: weekly
        enabled: true
        keep: 3
        schedule: 0 0 * * 0
        compressionType: gzip
        type: physical

check log in kubectl logs deploy/percona-server-mongodb-operator -n psmdb-operator

Versions

Kubernetes
Client Version: v1.29.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.9
Operator
1.16.1
Database
percona/percona-server-mongodb:7.0.11-multi

Anything else?

No response

Constant host unreachable errors within pod

Report

Hi all,
I’ve been trying to debug this for a good while now, I had a pretty large mongodb deployment and I thought I had a lot of logs simply due to its size. After some digging tho it seemed that they were mostly errors. The strange thing is that the cluster still appears to work. I’ve removed most of the components including sharding, backups, and pmm but I’m still seeing the errors even with a single replicaset. I’ve also disabled istio and effectively turned off the firewall. I’m deploying this cluster using the ansible helm module, and below I’ll paste the full config and a big sample of the logs.

More about the problem

and here is a sample of the logs:

{"t":{"$date":"2024-01-16T22:05:09.468+00:00"},"s":"D1", "c":"ASSERT",   "id":23074,   "ctx":"conn601","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":299}}
{"t":{"$date":"2024-01-16T22:05:09.468+00:00"},"s":"D1", "c":"ASSERT",   "id":23074,   "ctx":"conn601","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":444}}
{"t":{"$date":"2024-01-16T22:05:12.483+00:00"},"s":"D1", "c":"ASSERT",   "id":23074,   "ctx":"conn602","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":299}}
{"t":{"$date":"2024-01-16T22:05:12.484+00:00"},"s":"D1", "c":"ASSERT",   "id":23074,   "ctx":"conn602","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":444}}
{"t":{"$date":"2024-01-16T22:05:12.531+00:00"},"s":"D1", "c":"REPL",     "id":6208204, "ctx":"conn469","msg":"Error while waiting for hello response","attr":{"status":{"code":262,"codeName":"ExceededTimeLimit","errmsg":"operation exceeded time limit"}}}
{"t":{"$date":"2024-01-16T22:05:13.179+00:00"},"s":"D1", "c":"REPL",     "id":6208204, "ctx":"conn444","msg":"Error while waiting for hello response","attr":{"status":{"code":262,"codeName":"ExceededTimeLimit","errmsg":"operation exceeded time limit"}}}
{"t":{"$date":"2024-01-16T22:05:13.906+00:00"},"s":"D1", "c":"ASSERT",   "id":23074,   "ctx":"conn607","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":299}}
{"t":{"$date":"2024-01-16T22:05:13.906+00:00"},"s":"D1", "c":"ASSERT",   "id":23074,   "ctx":"conn607","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":444}}
{"t":{"$date":"2024-01-16T22:05:13.907+00:00"},"s":"D1", "c":"ASSERT",   "id":23074,   "ctx":"conn608","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":299}}
{"t":{"$date":"2024-01-16T22:05:13.907+00:00"},"s":"D1", "c":"REPL",     "id":6208204, "ctx":"conn606","msg":"Error while waiting for hello response","attr":{"status":{"code":279,"codeName":"ClientDisconnect","errmsg":"operation was interrupted"}}}
{"t":{"$date":"2024-01-16T22:05:13.907+00:00"},"s":"D1", "c":"ASSERT",   "id":23074,   "ctx":"conn606","msg":"User assertion","attr":{"error":"ClientDisconnect: operation was interrupted","file":"src/mongo/db/repl/replication_coordinator_impl.cpp","line":2453}}
{"t":{"$date":"2024-01-16T22:05:13.907+00:00"},"s":"D1", "c":"ASSERT",   "id":23074,   "ctx":"conn608","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":444}}
{"t":{"$date":"2024-01-16T22:05:13.908+00:00"},"s":"D1", "c":"ASSERT",   "id":23074,   "ctx":"conn606","msg":"User assertion","attr":{"error":"ClientDisconnect: operation was interrupted","file":"src/mongo/util/future_impl.h","line":1104}}
{"t":{"$date":"2024-01-16T22:05:13.908+00:00"},"s":"D1", "c":"ASSERT",   "id":23074,   "ctx":"conn606","msg":"User assertion","attr":{"error":"ClientDisconnect: operation was interrupted","file":"src/mongo/util/future_impl.h","line":1087}}
{"t":{"$date":"2024-01-16T22:05:13.908+00:00"},"s":"D1", "c":"ASSERT",   "id":23074,   "ctx":"conn606","msg":"User assertion","attr":{"error":"ClientDisconnect: operation was interrupted","file":"src/mongo/util/future_impl.h","line":1104}}
{"t":{"$date":"2024-01-16T22:05:13.908+00:00"},"s":"D1", "c":"ASSERT",   "id":23074,   "ctx":"conn606","msg":"User assertion","attr":{"error":"ClientDisconnect: operation was interrupted","file":"src/mongo/util/future_impl.h","line":1087}}
{"t":{"$date":"2024-01-16T22:05:13.908+00:00"},"s":"D1", "c":"COMMAND",  "id":21962,   "ctx":"conn606","msg":"Assertion while executing command","attr":{"command":"hello","db":"admin","commandArgs":{"hello":1,"helloOk":true,"topologyVersion":{"processId":{"$oid":"65a6fbe68c4bda018b2c0028"},"counter":13},"maxAwaitTimeMS":10000,"$db":"admin","$readPreference":{"mode":"primaryPreferred"}},"error":"ClientDisconnect: operation was interrupted"}}
{"t":{"$date":"2024-01-16T22:05:13.908+00:00"},"s":"D1", "c":"ASSERT",   "id":23074,   "ctx":"conn606","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":299}}
{"t":{"$date":"2024-01-16T22:05:13.908+00:00"},"s":"D1", "c":"ASSERT",   "id":23074,   "ctx":"conn606","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":444}}
{"t":{"$date":"2024-01-16T22:05:13.913+00:00"},"s":"D1", "c":"ASSERT",   "id":23074,   "ctx":"conn605","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":299}}
{"t":{"$date":"2024-01-16T22:05:13.913+00:00"},"s":"D1", "c":"ASSERT",   "id":23074,   "ctx":"conn604","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":299}}
{"t":{"$date":"2024-01-16T22:05:13.913+00:00"},"s":"D1", "c":"ASSERT",   "id":23074,   "ctx":"conn605","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":444}}
{"t":{"$date":"2024-01-16T22:05:13.913+00:00"},"s":"D1", "c":"ASSERT",   "id":23074,   "ctx":"conn604","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":444}}
{"t":{"$date":"2024-01-16T22:05:13.913+00:00"},"s":"D1", "c":"REPL",     "id":6208204, "ctx":"conn603","msg":"Error while waiting for hello response","attr":{"status":{"code":279,"codeName":"ClientDisconnect","errmsg":"operation was interrupted"}}}
{"t":{"$date":"2024-01-16T22:05:13.913+00:00"},"s":"D1", "c":"ASSERT",   "id":23074,   "ctx":"conn603","msg":"User assertion","attr":{"error":"ClientDisconnect: operation was interrupted","file":"src/mongo/db/repl/replication_coordinator_impl.cpp","line":2453}}
{"t":{"$date":"2024-01-16T22:05:13.913+00:00"},"s":"D1", "c":"ASSERT",   "id":23074,   "ctx":"conn603","msg":"User assertion","attr":{"error":"ClientDisconnect: operation was interrupted","file":"src/mongo/util/future_impl.h","line":1104}}
{"t":{"$date":"2024-01-16T22:05:13.913+00:00"},"s":"D1", "c":"ASSERT",   "id":23074,   "ctx":"conn603","msg":"User assertion","attr":{"error":"ClientDisconnect: operation was interrupted","file":"src/mongo/util/future_impl.h","line":1087}}
{"t":{"$date":"2024-01-16T22:05:13.913+00:00"},"s":"D1", "c":"ASSERT",   "id":23074,   "ctx":"conn603","msg":"User assertion","attr":{"error":"ClientDisconnect: operation was interrupted","file":"src/mongo/util/future_impl.h","line":1104}}
{"t":{"$date":"2024-01-16T22:05:13.913+00:00"},"s":"D1", "c":"ASSERT",   "id":23074,   "ctx":"conn603","msg":"User assertion","attr":{"error":"ClientDisconnect: operation was interrupted","file":"src/mongo/util/future_impl.h","line":1087}}
{"t":{"$date":"2024-01-16T22:05:13.913+00:00"},"s":"D1", "c":"COMMAND",  "id":21962,   "ctx":"conn603","msg":"Assertion while executing command","attr":{"command":"hello","db":"admin","commandArgs":{"hello":1,"helloOk":true,"topologyVersion":{"processId":{"$oid":"65a6fbe68c4bda018b2c0028"},"counter":13},"maxAwaitTimeMS":10000,"$db":"admin","$readPreference":{"mode":"primaryPreferred"}},"error":"ClientDisconnect: operation was interrupted"}}
{"t":{"$date":"2024-01-16T22:05:13.913+00:00"},"s":"D1", "c":"ASSERT",   "id":23074,   "ctx":"conn603","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":299}}
{"t":{"$date":"2024-01-16T22:05:13.913+00:00"},"s":"D1", "c":"ASSERT",   "id":23074,   "ctx":"conn603","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":444}}
{"t":{"$date":"2024-01-16T22:05:14.137+00:00"},"s":"D1", "c":"ASSERT",   "id":23074,   "ctx":"conn609","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":299}}
{"t":{"$date":"2024-01-16T22:05:14.137+00:00"},"s":"D1", "c":"ASSERT",   "id":23074,   "ctx":"conn609","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":444}}
{"t":{"$date":"2024-01-16T22:05:14.137+00:00"},"s":"D1", "c":"REPL",     "id":6208204, "ctx":"conn610","msg":"Error while waiting for hello response","attr":{"status":{"code":279,"codeName":"ClientDisconnect","errmsg":"operation was interrupted"}}}
{"t":{"$date":"2024-01-16T22:05:14.137+00:00"},"s":"D1", "c":"ASSERT",   "id":23074,   "ctx":"conn610","msg":"User assertion","attr":{"error":"ClientDisconnect: operation was interrupted","file":"src/mongo/db/repl/replication_coordinator_impl.cpp","line":2453}}
{"t":{"$date":"2024-01-16T22:05:14.137+00:00"},"s":"D1", "c":"ASSERT",   "id":23074,   "ctx":"conn610","msg":"User assertion","attr":{"error":"ClientDisconnect: operation was interrupted","file":"src/mongo/util/future_impl.h","line":1104}}
{"t":{"$date":"2024-01-16T22:05:14.138+00:00"},"s":"D1", "c":"ASSERT",   "id":23074,   "ctx":"conn610","msg":"User assertion","attr":{"error":"ClientDisconnect: operation was interrupted","file":"src/mongo/util/future_impl.h","line":1087}}
{"t":{"$date":"2024-01-16T22:05:14.138+00:00"},"s":"D1", "c":"ASSERT",   "id":23074,   "ctx":"conn610","msg":"User assertion","attr":{"error":"ClientDisconnect: operation was interrupted","file":"src/mongo/util/future_impl.h","line":1104}}
{"t":{"$date":"2024-01-16T22:05:14.138+00:00"},"s":"D1", "c":"ASSERT",   "id":23074,   "ctx":"conn610","msg":"User assertion","attr":{"error":"ClientDisconnect: operation was interrupted","file":"src/mongo/util/future_impl.h","line":1087}}
{"t":{"$date":"2024-01-16T22:05:14.138+00:00"},"s":"D1", "c":"COMMAND",  "id":21962,   "ctx":"conn610","msg":"Assertion while executing command","attr":{"command":"hello","db":"admin","commandArgs":{"hello":1,"helloOk":true,"topologyVersion":{"processId":{"$oid":"65a6fbe68c4bda018b2c0028"},"counter":13},"maxAwaitTimeMS":10000,"$db":"admin","$readPreference":{"mode":"primaryPreferred"}},"error":"ClientDisconnect: operation was interrupted"}}
{"t":{"$date":"2024-01-16T22:05:14.138+00:00"},"s":"D1", "c":"ASSERT",   "id":23074,   "ctx":"conn611","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":299}}
{"t":{"$date":"2024-01-16T22:05:14.138+00:00"},"s":"D1", "c":"ASSERT",   "id":23074,   "ctx":"conn611","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":444}}
{"t":{"$date":"2024-01-16T22:05:14.139+00:00"},"s":"D1", "c":"ASSERT",   "id":23074,   "ctx":"conn610","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":299}}
{"t":{"$date":"2024-01-16T22:05:14.139+00:00"},"s":"D1", "c":"ASSERT",   "id":23074,   "ctx":"conn610","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":444}}
{"t":{"$date":"2024-01-16T22:05:15.464+00:00"},"s":"D1", "c":"ASSERT",   "id":23074,   "ctx":"conn612","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":299}}
{"t":{"$date":"2024-01-16T22:05:15.464+00:00"},"s":"D1", "c":"ASSERT",   "id":23074,   "ctx":"conn612","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":444}}
"t":{"$date":"2024-01-16T22:10:25.733+00:00"},"s":"D1", "c":"ASSERT",   "id":23074,   "ctx":"conn1060","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":444}}

Custom resource is reporting a ready status, here are its logs:

2024-01-22T17:42:38.722Z    INFO    Created a new mongo key    {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "26ab8173-6077-437d-9d50-6123d1c182a2", "KeyName": "kev-test-psmdb-db-mongodb-keyfile"}
2024-01-22T17:42:38.731Z    INFO    Created a new mongo key    {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "26ab8173-6077-437d-9d50-6123d1c182a2", "KeyName": "kev-test-psmdb-db-mongodb-encryption-key"}
2024-01-22T17:42:38.779Z    INFO    Waiting for the pods    {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "26ab8173-6077-437d-9d50-6123d1c182a2", "replset": "shard", "size": 3, "pods": 0}
2024-01-22T17:42:38.799Z    INFO    Cluster state changed    {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "26ab8173-6077-437d-9d50-6123d1c182a2", "previous": "", "current": "initializing"}
2024-01-22T17:42:38.850Z    INFO    Waiting for the pods    {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "af333758-e95d-4100-9f52-cfe3cbf57cdb", "replset": "shard", "size": 3, "pods": 0}
2024-01-22T17:42:43.867Z    INFO    Waiting for the pods    {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "f074b1e2-166a-456d-b8c4-c27b61b0d77a", "replset": "shard", "size": 3, "pods": 1}
2024-01-22T17:42:43.947Z    INFO    Waiting for the pods    {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "009f498e-7342-427e-8e6e-17b82542081d", "replset": "shard", "size": 3, "pods": 1}
2024-01-22T17:42:48.943Z    INFO    Waiting for the pods    {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "987a4804-a466-4fe3-8ab1-3ae5140fd989", "replset": "shard", "size": 3, "pods": 1}
2024-01-22T17:42:53.989Z    INFO    Waiting for the pods    {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "cfab78a1-aa52-4adb-a08c-a25b6b5bf0a5", "replset": "shard", "size": 3, "pods": 1}
2024-01-22T17:42:59.029Z    INFO    Waiting for the pods    {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "e81e933d-7d9e-4ef5-b173-cf0bd43303ac", "replset": "shard", "size": 3, "pods": 1}
2024-01-22T17:43:04.078Z    INFO    Waiting for the pods    {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "50774dcf-0a7b-4a39-b645-b9c79e266449", "replset": "shard", "size": 3, "pods": 1}
2024-01-22T17:43:09.162Z    INFO    Waiting for the pods    {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "3fa6b94b-3b6b-4235-ad28-c66e97550762", "replset": "shard", "size": 3, "pods": 2}
2024-01-22T17:43:09.260Z    INFO    Waiting for the pods    {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "49550521-00e6-4d8c-be77-756ca1e99cd0", "replset": "shard", "size": 3, "pods": 2}
2024-01-22T17:43:14.248Z    INFO    Waiting for the pods    {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "df050785-9e5c-4a0e-b4ed-89eb420d1696", "replset": "shard", "size": 3, "pods": 2}
2024-01-22T17:43:19.312Z    INFO    Waiting for the pods    {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "fe180278-1aaf-480a-a804-398c9df72e9a", "replset": "shard", "size": 3, "pods": 2}
2024-01-22T17:43:24.358Z    INFO    Waiting for the pods    {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "c48ac446-c9f1-409d-91b5-f6047cc05b3c", "replset": "shard", "size": 3, "pods": 2}
2024-01-22T17:43:29.398Z    INFO    Waiting for the pods    {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "ad5fc822-752e-4b4e-8f4a-183bff39c200", "replset": "shard", "size": 3, "pods": 2}
2024-01-22T17:43:34.453Z    INFO    Waiting for the pods    {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "3ebe6019-550b-4ac0-b2cc-9c5e55b7a3f2", "replset": "shard", "size": 3, "pods": 2}
2024-01-22T17:43:49.532Z    INFO    initiating replset    {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "0d2ff7eb-2cdc-4d08-93e1-a4dc02dc10a4", "replset": "shard", "pod": "kev-test-psmdb-db-shard-0"}
2024-01-22T17:43:58.669Z    INFO    replset initialized    {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "0d2ff7eb-2cdc-4d08-93e1-a4dc02dc10a4", "replset": "shard", "pod": "kev-test-psmdb-db-shard-0"}
2024-01-22T17:43:59.205Z    INFO    Fixing member tags    {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "459a4f0b-f928-4fc3-ad20-e22c5d92c4c3", "replset": "shard"}
2024-01-22T17:43:59.205Z    DEBUG    Running replSetReconfig config    {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "459a4f0b-f928-4fc3-ad20-e22c5d92c4c3", "cfg": {"_id":"shard","version":2,"members":[{"_id":0,"host":"kev-test-psmdb-db-shard-0.kev-test-psmdb-db-shard.app.svc.dev.ahq:27017","arbiterOnly":false,"buildIndexes":true,"hidden":false,"priority":1,"tags":{"podName":"kev-test-psmdb-db-shard-0","serviceName":"kev-test-psmdb-db"},"secondaryDelaySecs":0,"votes":1}],"protocolVersion":1,"settings":{"chainingAllowed":true,"heartbeatIntervalMillis":2000,"heartbeatTimeoutSecs":10,"electionTimeoutMillis":10000,"catchUpTimeoutMillis":-1,"getLastErrorDefaults":{"w":1,"wtimeout":0},"replicaSetId":"65aea9578f40f2bb98bd0171"},"writeConcernMajorityJournalDefault":true}}
2024-01-22T17:43:59.209Z    INFO    Adding new nodes    {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "459a4f0b-f928-4fc3-ad20-e22c5d92c4c3", "replset": "shard"}
2024-01-22T17:43:59.209Z    DEBUG    Running replSetReconfig config    {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "459a4f0b-f928-4fc3-ad20-e22c5d92c4c3", "cfg": {"_id":"shard","version":3,"members":[{"_id":0,"host":"kev-test-psmdb-db-shard-0.kev-test-psmdb-db-shard.app.svc.dev.ahq:27017","arbiterOnly":false,"buildIndexes":true,"hidden":false,"priority":1,"tags":{"podName":"kev-test-psmdb-db-shard-0","serviceName":"kev-test-psmdb-db"},"secondaryDelaySecs":0,"votes":1},{"_id":1,"host":"kev-test-psmdb-db-shard-1.kev-test-psmdb-db-shard.app.svc.dev.ahq:27017","arbiterOnly":false,"buildIndexes":true,"hidden":false,"priority":2,"tags":{"podName":"kev-test-psmdb-db-shard-1","serviceName":"kev-test-psmdb-db"},"votes":1}],"protocolVersion":1,"settings":{"chainingAllowed":true,"heartbeatIntervalMillis":2000,"heartbeatTimeoutSecs":10,"electionTimeoutMillis":10000,"catchUpTimeoutMillis":-1,"getLastErrorDefaults":{"w":1,"wtimeout":0},"replicaSetId":"65aea9578f40f2bb98bd0171"},"writeConcernMajorityJournalDefault":true}}
2024-01-22T17:43:59.231Z    INFO    Configuring member votes and priorities    {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "459a4f0b-f928-4fc3-ad20-e22c5d92c4c3", "replset": "shard"}
2024-01-22T17:43:59.231Z    DEBUG    Running replSetReconfig config    {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "459a4f0b-f928-4fc3-ad20-e22c5d92c4c3", "cfg": {"_id":"shard","version":4,"members":[{"_id":0,"host":"kev-test-psmdb-db-shard-0.kev-test-psmdb-db-shard.app.svc.dev.ahq:27017","arbiterOnly":false,"buildIndexes":true,"hidden":false,"priority":2,"tags":{"podName":"kev-test-psmdb-db-shard-0","serviceName":"kev-test-psmdb-db"},"secondaryDelaySecs":0,"votes":1},{"_id":1,"host":"kev-test-psmdb-db-shard-1.kev-test-psmdb-db-shard.app.svc.dev.ahq:27017","arbiterOnly":false,"buildIndexes":true,"hidden":false,"priority":0,"tags":{"podName":"kev-test-psmdb-db-shard-1","serviceName":"kev-test-psmdb-db"},"votes":0}],"protocolVersion":1,"settings":{"chainingAllowed":true,"heartbeatIntervalMillis":2000,"heartbeatTimeoutSecs":10,"electionTimeoutMillis":10000,"catchUpTimeoutMillis":-1,"getLastErrorDefaults":{"w":1,"wtimeout":0},"replicaSetId":"65aea9578f40f2bb98bd0171"},"writeConcernMajorityJournalDefault":true}}
2024-01-22T17:44:04.188Z    INFO    Adding new nodes    {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "ca1d90c5-512e-4d47-b455-14193e6634af", "replset": "shard"}
2024-01-22T17:44:04.188Z    DEBUG    Running replSetReconfig config    {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "ca1d90c5-512e-4d47-b455-14193e6634af", "cfg": {"_id":"shard","version":6,"members":[{"_id":0,"host":"kev-test-psmdb-db-shard-0.kev-test-psmdb-db-shard.app.svc.dev.ahq:27017","arbiterOnly":false,"buildIndexes":true,"hidden":false,"priority":2,"tags":{"podName":"kev-test-psmdb-db-shard-0","serviceName":"kev-test-psmdb-db"},"secondaryDelaySecs":0,"votes":1},{"_id":1,"host":"kev-test-psmdb-db-shard-1.kev-test-psmdb-db-shard.app.svc.dev.ahq:27017","arbiterOnly":false,"buildIndexes":true,"hidden":false,"priority":0,"tags":{"podName":"kev-test-psmdb-db-shard-1","serviceName":"kev-test-psmdb-db"},"secondaryDelaySecs":0,"votes":0},{"_id":2,"host":"kev-test-psmdb-db-shard-2.kev-test-psmdb-db-shard.app.svc.dev.ahq:27017","arbiterOnly":false,"buildIndexes":true,"hidden":false,"priority":2,"tags":{"podName":"kev-test-psmdb-db-shard-2","serviceName":"kev-test-psmdb-db"},"votes":1}],"protocolVersion":1,"settings":{"chainingAllowed":true,"heartbeatIntervalMillis":2000,"heartbeatTimeoutSecs":10,"electionTimeoutMillis":10000,"catchUpTimeoutMillis":-1,"getLastErrorDefaults":{"w":1,"wtimeout":0},"replicaSetId":"65aea9578f40f2bb98bd0171"},"writeConcernMajorityJournalDefault":true}}
2024-01-22T17:44:04.216Z    INFO    Configuring member votes and priorities    {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "ca1d90c5-512e-4d47-b455-14193e6634af", "replset": "shard"}
2024-01-22T17:44:04.216Z    DEBUG    Running replSetReconfig config    {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "ca1d90c5-512e-4d47-b455-14193e6634af", "cfg": {"_id":"shard","version":7,"members":[{"_id":0,"host":"kev-test-psmdb-db-shard-0.kev-test-psmdb-db-shard.app.svc.dev.ahq:27017","arbiterOnly":false,"buildIndexes":true,"hidden":false,"priority":2,"tags":{"podName":"kev-test-psmdb-db-shard-0","serviceName":"kev-test-psmdb-db"},"secondaryDelaySecs":0,"votes":1},{"_id":1,"host":"kev-test-psmdb-db-shard-1.kev-test-psmdb-db-shard.app.svc.dev.ahq:27017","arbiterOnly":false,"buildIndexes":true,"hidden":false,"priority":2,"tags":{"podName":"kev-test-psmdb-db-shard-1","serviceName":"kev-test-psmdb-db"},"secondaryDelaySecs":0,"votes":1},{"_id":2,"host":"kev-test-psmdb-db-shard-2.kev-test-psmdb-db-shard.app.svc.dev.ahq:27017","arbiterOnly":false,"buildIndexes":true,"hidden":false,"priority":2,"tags":{"podName":"kev-test-psmdb-db-shard-2","serviceName":"kev-test-psmdb-db"},"votes":1}],"protocolVersion":1,"settings":{"chainingAllowed":true,"heartbeatIntervalMillis":2000,"heartbeatTimeoutSecs":10,"electionTimeoutMillis":10000,"catchUpTimeoutMillis":-1,"getLastErrorDefaults":{"w":1,"wtimeout":0},"replicaSetId":"65aea9578f40f2bb98bd0171"},"writeConcernMajorityJournalDefault":true}}
2024-01-22T17:44:16.958Z    INFO    Cluster state changed    {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "de7ea753-5e72-4be0-988a-08850b8ffd9d", "previous": "initializing", "current": "ready"}
2024-01-22T17:44:17.309Z    INFO    update Mongo version to 6.0.9-7 (fetched from db)    {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "d5aee049-1b1a-4b61-a34b-d46db7cb5736"}

Steps to reproduce

- name: Deploy Percona Server for MongoDB
  kubernetes.core.helm:
    name: kev-test-mongodb
    chart_ref: percona/psmdb-db
    chart_version: "1.15.1"
    release_namespace: app
    wait: true
    wait_timeout: "10m"
    values:
      clusterServiceDNSSuffix: 'svc.{{ cluster_domain }}'
      finalizers:
        - delete-psmdb-pvc
      nameOverride: ""
      fullnameOverride: ""
      crVersion: 1.15.0
      pause: false
      unmanaged: false
      allowUnsafeConfigurations: false
      multiCluster:
        enabled: false
      updateStrategy: SmartUpdate
      # updateStrategy: RollingUpdate
      upgradeOptions:
        versionServiceEndpoint: https://check.percona.com
        apply: disabled
        schedule: "0 2 * * *"
        setFCV: false
      image:
        repository: percona/percona-server-mongodb
        tag: 6.0.9-7
      imagePullPolicy: Always
      secrets: {}
      pmm:
        enabled: false
      replsets:
        - name: shard
          size: 3
          annotations:
            sidecar.istio.io/inject: "false"
          configuration: |
            security:
              enableEncryption: false
            systemLog:
              verbosity: 1
          serviceAccountName: app
          storage:
            engine: inMemory
            inMemory:
              engineConfig:
                inMemorySizeRatio: 0.9
          podDisruptionBudget:
            maxUnavailable: 1
          expose:
            enabled: true
            exposeType: ClusterIP
          nonvoting:
            enabled: false
            size: 1
          arbiter:
            enabled: false
            size: 1
          resources:
            limits:
              cpu: "2048m"
              memory: "5.0G"
            requests:
              cpu: "300m"
              memory: "0.5G"
          volumeSpec:
            pvc:
              storageClassName: "ceph-block"
              accessModes: [ "ReadWriteOnce" ]
              resources:
                requests:
                  storage: 1Gi
      sharding:
        enabled: false
      backup:
        enabled: false

Versions

Kubernetes RKE2 1.28.3
Operator percona/percona-server-mongodb-operator:1.15.0
Database percona/percona-server-mongodb:6.0.9-7

Anything else?

No response

No way to turn off authentication

Report

We cannot disable authentication even if we pass this as a conf

security:
  authorization: "disabled"

Steps to reproduce

Deploy with:

spec:
  ...
  replsets:
    - name: rs0
      size: 3
      configuration: |
        security:
          authorization: "disabled"

Versions

Kubernetes: 1.28
Operator: 1.16.1
Database: 7.0.8-5

Anything else?

This would be fine if there'a a way to create users using the crd.

Allow adding more initContainers and their securityContext

Proposal

Currently, only the mongo-init initContainer is run.
In order to tune mongodb a bit further I think It'd be good to have a way to configure extra initContainers and their securityContext.

Use-Case

I would like to be able to modify the tcp_keepalive of my service. In order to do so, I need to spawn a privileged initContainer changing this configuration. See https://www.mongodb.com/docs/kubernetes-operator/v1.25/tutorial/modify-resource-image/#tune-mongodb-kubernetes-resource-docker-images-with-an-initcontainer

Is this a feature you are interested in implementing yourself?

Maybe

Anything else?

No response

Backups failing on one cluster without error message

Report

We have two clusters managed by the same operator running on kubernetes. Daily backup is set up for them, which works for one cluster, but fails for the other. All the settings are the same for backups, both uses the same s3 bucket.

Steps to reproduce

Install cluster via mongodb-operator
Enable backups
Increase cluster resources (also requests/limits)
Backups will fail (?)

Versions

Kubernetes: 1.26.13-eks-508b6b3
Operator: percona/percona-server-mongodb-operator:1.15.0
Backup agent version: percona/percona-backup-mongodb:2.0.4
Mongo version: percona/percona-server-mongodb:5.0.15-13

Anything else?

I also tried to restart the whole cluster, but still the same.

We haven't changed the resources of the other cluster and the backups are working fine there.

Possibility to choose nodeport used port

Hi,
I want to deploy a database instance and expose it, my idea was to use a nodeport but I also wanted to choose the port, bur from what I saw there is no way to do that.
I am using the helm chart to deploy my instance but from what I saw it seems to be an issue coming from the operator.
I want to use a nodeport to make things easier and I want to choose the port to keep it consistant between my clusters

configurable externalTrafficPolicy for LoadBalancer

Proposal

I would like to be able to change externalTrafficPolicy to Local for the LoadBalancer service in order to preserve the realIP address
It would be nice if it's configurable with the helm chart values

Use-Case

I would like to see real IP address in the logs using LoadBalancer service

Is this a feature you are interested in implementing yourself?

Anything else?

No response

Allow creation of backups without delete-backup finalizers

Proposal

We should be able when enabling backups in PerconaServerMongoDb to choose the finalizers or at least to remove the delete-backup finalizer.

backup:
  enabled: true
  finalizers: false

backup:
  enabled: true
  finalizers: []

Use-Case

The credentials used to push a new backup (or even to retrieve them) should not be able to delete a backup as this would create a security bridge.
Most likely no-one is even able to delete a backup as enabling object-lock should be a standard practice.
This prevent the backup objects from being deleted without manually removing the useless finalizer in this case.

Is this a feature you are interested in implementing yourself?

Anything else?

No response

Unable to change the replica size, Operator seems to always pick the size as 3 for the StatefulSet

Report

I have been trying to get mongodb deployed in a kubernetes cluster. But the value set for size under replsets doesn't seem to have any effect. The operator is always creating the statefulset with 3 pods.

Steps to reproduce

Deploy the pmdb chart using the following values,
The contents of pmdb.yml.

replsets:
  - name: rs0
    size: 2
    podSecurityContext:
      fsGroup: 1001
      runAsGroup: 1001
      runAsUser: 1001
    resources:
      limits:
        cpu: "3900m"
        memory: 14Gi
      requests:
        cpu: 2
        memory: 10Gi

Deploy psmdb

helm install my-db percona/psmdb-db --namespace=percona -f ./deploy/mongodb/pmdb.yml

Verify if the expected values are set in the pmdb resource
The operator creates statefulset with 3 pods regardless whatever is passed in size

Versions

Kubernetes: 1.28
Operator: 1.15.0
Database: MongoDB 6

Anything else?

No response

Duplicate StatefulSet ControllerRevision, PerconaServerMongoDB status stucks in "Initializing"

Report

The PerconaServerMongoDB status stucks in "Initializing" because the field "status.updatedReplicas" is always smaller than "status.replicas" which is caused by multiple ControllerRevision

Steps to reproduce

Install MongoDB Operator in "mongodb-operator" namespace in ArgoCD
Install MongoDB Helm in "mongodb" namespace in ArgoCD

Versions

Kubernetes: AWS EKS 1.29
Operator: 1.16.0
Database: 1.16.0

Anything else?

The issue does not happen with Operator version 1.15.4 and Database version 1.15.3

Failed to delete history backup data in minio

Report

Failed to delete history backup data in minio.

pbm only deleted the history json file, but the directory file was not successfully deleted, and I doubt it was because --force was not added.

This is my values.yaml file

backup:
  enabled: true
  image:
    repository: harbor.devops.narwal.com/public_mirror/percona-backup-mongodb
    tag: 2.4.1
  storages:
    minio:
      type: s3
      s3:
        bucket: testing
        region: us-east-1
        credentialsSecret: backup-credentials-minio
        endpointUrl: https://10.10.180.14:9700/
        prefix: "mongodb"
        insecureSkipTLSVerify: false
  pitr:
    enabled: true
    oplogOnly: false
tasks:
  - name: daily-minio-s3
    enabled: true
    schedule: "0 */1 * * *"
    keep: 2
    storageName: minio
    compressionType: gzip
    type: physical

Steps to reproduce

Versions

Kubernetes
Operator
Database

Anything else?

No response

Cannot use custom issuer to generate tls certificate

Proposal

As far as I know, the CR definition of Percona MongoDB cluster does not provide any configuration attribute to use an existing custom Issuer to generate the tls certificate with cert-manager. A <cluster-name>-psmdb-issuer issuer resource is automatically generated by the operator, then <cluster-name>-ssl and <cluster-name>-ssl-internal certificate resources are issued from it.

The Percona XtraDB cluster provide the tls.issuerConf attributes to specify a custom Issuer for these certificates to be generated.

It would be a great feature to implement this as well for the future releases of Percona MongoDB operator if not planned yet !

	args := []string{
	"--bind_ip_all",
	"--auth",
	"--dbpath=" + MongodContainerDataDir,
	"--port=" + strconv.Itoa(int(api.DefaultMongodPort)),
	"--replSet=" + replset.Name,
	"--storageEngine=" + string(replset.Storage.Engine),
	"--relaxPermChecks",
	}