percona / percona-server-mongodb-operator Goto Github PK
View Code? Open in Web Editor NEWPercona Operator for MongoDB
Home Page: https://www.percona.com/doc/kubernetes-operator-for-psmongodb/
License: Apache License 2.0
Percona Operator for MongoDB
Home Page: https://www.percona.com/doc/kubernetes-operator-for-psmongodb/
License: Apache License 2.0
I'm using both the psmdb-operator and psmdb-db helm charts. I have deployed the operator (without the db deployment) and it was working fine, without errors/crashes.
However, now that I have deployed the db, the operator enters a crashloop. When the operator starts crashlooping, it causes a complete restart of all the pods from psmdb-db as well.
2024-06-12T21:42:34.549Z INFO setup Manager starting up {"gitCommit": "54e1b18dd9dac8e0ed5929bb2c91318cd6829a48", "gitBranch": "release-1-16-0", "goVersion": "go1.22.3", "os": "linux", "arch": "amd64"}
2024-06-12T21:42:34.565Z INFO server version {"platform": "kubernetes", "version": "v1.28.7+k3s1"}
2024-06-12T21:42:34.570Z INFO controller-runtime.metrics Starting metrics server
2024-06-12T21:42:34.570Z INFO starting server {"name": "health probe", "addr": "[::]:8081"}
2024-06-12T21:42:34.570Z INFO controller-runtime.metrics Serving metrics server {"bindAddress": ":8080", "secure": false}
I0612 21:42:34.570960 1 leaderelection.go:250] attempting to acquire leader lease mongodb/08db0feb.percona.com...
I0612 21:42:53.320941 1 leaderelection.go:260] successfully acquired lease mongodb/08db0feb.percona.com
2024-06-12T21:42:53.321Z INFO Starting EventSource {"controller": "psmdb-controller", "source": "kind source: *v1.PerconaServerMongoDB"}
2024-06-12T21:42:53.321Z INFO Starting Controller {"controller": "psmdb-controller"}
2024-06-12T21:42:53.321Z INFO Starting EventSource {"controller": "psmdbrestore-controller", "source": "kind source: *v1.PerconaServerMongoDBRestore"}
2024-06-12T21:42:53.321Z INFO Starting EventSource {"controller": "psmdbbackup-controller", "source": "kind source: *v1.PerconaServerMongoDBBackup"}
2024-06-12T21:42:53.321Z INFO Starting EventSource {"controller": "psmdbrestore-controller", "source": "kind source: *v1.Pod"}
2024-06-12T21:42:53.321Z INFO Starting Controller {"controller": "psmdbrestore-controller"}
2024-06-12T21:42:53.321Z INFO Starting EventSource {"controller": "psmdbbackup-controller", "source": "kind source: *v1.Pod"}
2024-06-12T21:42:53.321Z INFO Starting Controller {"controller": "psmdbbackup-controller"}
2024-06-12T21:42:53.444Z INFO Starting workers {"controller": "psmdbbackup-controller", "worker count": 1}
2024-06-12T21:42:53.445Z INFO Starting workers {"controller": "psmdb-controller", "worker count": 1}
2024-06-12T21:42:53.445Z INFO Starting workers {"controller": "psmdbrestore-controller", "worker count": 1}
E0612 21:42:53.685207 1 runtime.go:79] Observed a panic: "assignment to entry in nil map" (assignment to entry in nil map)
goroutine 313 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x1f11320, 0x298b1f0})
/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:75 +0x85
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000802fc0?})
/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:49 +0x6b
panic({0x1f11320?, 0x298b1f0?})
/usr/local/go/src/runtime/panic.go:770 +0x132
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).setUpdateMongosFirst.func1()
/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/smart.go:226 +0xd0
k8s.io/client-go/util/retry.OnError.func1()
/go/pkg/mod/k8s.io/[email protected]/util/retry/util.go:51 +0x30
k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection(0x411b9b?)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:145 +0x3e
k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff({0x989680, 0x4014000000000000, 0x3fb999999999999a, 0x4, 0x0}, 0xc000baaa18)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:461 +0x5a
k8s.io/client-go/util/retry.OnError({0x989680, 0x4014000000000000, 0x3fb999999999999a, 0x4, 0x0}, 0x4171ba?, 0x0?)
/go/pkg/mod/k8s.io/[email protected]/util/retry/util.go:50 +0xa5
k8s.io/client-go/util/retry.RetryOnConflict(...)
/go/pkg/mod/k8s.io/[email protected]/util/retry/util.go:104
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).setUpdateMongosFirst(0x1ef45e0?, {0x29affe0?, 0xc0011ad140?}, 0x6?)
/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/smart.go:220 +0xbc
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).createSSLByCertManager(0xc000b882d0, {0x29affe0, 0xc0011ad140}, 0xc000dcaf08)
/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/ssl.go:187 +0x794
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileSSL(0xc000b882d0, {0x29affe0, 0xc0011ad140}, 0xc000dcaf08)
/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/ssl.go:66 +0x30d
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile(0xc000b882d0, {0x29affe0, 0xc0011ad140}, {{{0xc0006dade8?, 0x5?}, {0xc0006dade0?, 0xc000d25d10?}}})
/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:368 +0x16d0
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x29b4dc8?, {0x29affe0?, 0xc0011ad140?}, {{{0xc0006dade8?, 0xb?}, {0xc0006dade0?, 0x0?}}})
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114 +0xb7
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000b820b0, {0x29b0018, 0xc0009c03c0}, {0x1fdf1a0, 0xc000dd27a0})
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311 +0x3bc
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000b820b0, {0x29b0018, 0xc0009c03c0})
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261 +0x1be
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222 +0x79
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 141
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:218 +0x486
2024-06-12T21:42:53.730Z INFO Observed a panic in reconciler: assignment to entry in nil map {"controller": "psmdb-controller", "object": {"name":"psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "psmdb-db", "reconcileID": "7676acba-b62f-4d00-a4dc-51c0e17bc27c"}
panic: assignment to entry in nil map [recovered]
panic: assignment to entry in nil map [recovered]
panic: assignment to entry in nil map
goroutine 313 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:111 +0x1e5
panic({0x1f11320?, 0x298b1f0?})
/usr/local/go/src/runtime/panic.go:770 +0x132
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000802fc0?})
/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:56 +0xcd
panic({0x1f11320?, 0x298b1f0?})
/usr/local/go/src/runtime/panic.go:770 +0x132
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).setUpdateMongosFirst.func1()
/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/smart.go:226 +0xd0
k8s.io/client-go/util/retry.OnError.func1()
/go/pkg/mod/k8s.io/[email protected]/util/retry/util.go:51 +0x30
k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection(0x411b9b?)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:145 +0x3e
k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff({0x989680, 0x4014000000000000, 0x3fb999999999999a, 0x4, 0x0}, 0xc000efea18)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:461 +0x5a
k8s.io/client-go/util/retry.OnError({0x989680, 0x4014000000000000, 0x3fb999999999999a, 0x4, 0x0}, 0x4171ba?, 0x0?)
/go/pkg/mod/k8s.io/[email protected]/util/retry/util.go:50 +0xa5
k8s.io/client-go/util/retry.RetryOnConflict(...)
/go/pkg/mod/k8s.io/[email protected]/util/retry/util.go:104
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).setUpdateMongosFirst(0x1ef45e0?, {0x29affe0?, 0xc0011ad140?}, 0x6?)
/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/smart.go:220 +0xbc
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).createSSLByCertManager(0xc000b882d0, {0x29affe0, 0xc0011ad140}, 0xc000dcaf08)
/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/ssl.go:187 +0x794
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileSSL(0xc000b882d0, {0x29affe0, 0xc0011ad140}, 0xc000dcaf08)
/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/ssl.go:66 +0x30d
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile(0xc000b882d0, {0x29affe0, 0xc0011ad140}, {{{0xc0006dade8?, 0x5?}, {0xc0006dade0?, 0xc000d25d10?}}})
/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:368 +0x16d0
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x29b4dc8?, {0x29affe0?, 0xc0011ad140?}, {{{0xc0006dade8?, 0xb?}, {0xc0006dade0?, 0x0?}}})
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114 +0xb7
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000b820b0, {0x29b0018, 0xc0009c03c0}, {0x1fdf1a0, 0xc000dd27a0})
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311 +0x3bc
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000b820b0, {0x29b0018, 0xc0009c03c0})
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261 +0x1be
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222 +0x79
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 141
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:218 +0x486
My values files are as follows:
perconaMongodb:
enabled: true
version: 1.16.1
values:
backup:
enabled: false
pitr:
enabled: true
storages:
gcp:
type: s3
s3:
credentialsSecret: gcp-backup-credentials
bucket: redacted
region: us
prefix: dev-onprem/mongodb
endpointUrl: https://storage.googleapis.com
tasks:
- name: daily-gcp-us
enabled: true
schedule: "0 0 * * *"
keep: 3
storageName: gcp
compressionType: gzip
pmm:
enabled: true
replsets:
rs0:
volumeSpec:
pvc:
storageClassName: ceph-block
resources:
requests:
storage: 10Gi
rs1:
resources:
limits:
cpu: "300m"
memory: "0.5G"
requests:
cpu: "300m"
memory: "0.5G"
size: 3
volumeSpec:
pvc:
storageClassName: ceph-block
resources:
requests:
storage: 10Gi
secrets:
users: percona-mongodb-credentials
sharding:
configrs:
volumeSpec:
pvc:
storageClassName: ceph-block
resources:
requests:
storage: 10Gi
tls:
issuerConf:
name: redacted
kind: ClusterIssuer
perconaMongodbOperator:
enabled: true
version: 1.16.1
values:
watchNamespace: "mongodb"
Everything else is using the default values.
Deployment is handled via ArgoCD.
test
test
Yes
test
Backup configuration cannot be updated via YAML
pbm config
pbm config
No response
I opened this issue when I attempted with previous version, but even latest versions do not work, so updated title.
Creation of cluster with unmanaged: true
fails, mongodb nodes bootloop.
Operator continues to log the following repeatedly as the mongo node bootloop.
INFO Replset is not exposed. Make sure each pod in the replset can reach each other. {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "4071c15b-9595-443f-bb20-5705204cbd3d", "replset": "rs0"}
I am attempting to follow this guide
to migrate one of our legacy mongodb without downtime.
SSL off as the internal db we are using does not have that on, I will turn it on later with short downtime after migration is successful.
Yaml
---
apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDB
metadata:
name: example-mongodb
namespace: mongodb
spec:
allowUnsafeConfigurations: true
unsafeFlags:
tls: true
unmanaged: true
crVersion: 1.15.0
image: percona/percona-server-mongodb:4.4.24
tls:
mode: disabled
replsets:
- affinity:
antiAffinityTopologyKey: kubernetes.io/hostname
name: rs0
size: 3
volumeSpec:
persistentVolumeClaim:
resources:
requests:
storage: 100Gi
expose:
enabled: false
exposeType: LoadBalancer
secrets:
users: users
sharding:
enabled: false
backup:
enabled: false
pitr:
enabled: false
upgradeOptions:
apply: disabled
schedule: 0 2 * * *
The DB I am attempting to migrate requires a target of 4.4, hence the cr version selection.
The deployment works without unmanaged: true
Two pod logs from start up to bootloop
example-mongodb-rs0-0_mongod.log
example-mongodb-rs0-1_mongod.log
Operator logs:
2024-07-24T18:21:17.244954170-04:00 2024-07-24T22:21:17.244Z INFO Replset is not exposed. Make sure each pod in the replset can reach each other. {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "de793d73-e501-4bac-93cd-351957acaa4a", "replset": "rs0"}
2024-07-24T18:21:17.519518649-04:00 2024-07-24T22:21:17.518Z INFO Created a new mongo key {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "de793d73-e501-4bac-93cd-351957acaa4a", "KeyName": "example-mongodb-mongodb-keyfile"}
2024-07-24T22:21:17.535Z INFO Created a new mongo key {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "de793d73-e501-4bac-93cd-351957acaa4a", "KeyName": "example-mongodb-mongodb-encryption-key"}
2024-07-24T18:21:17.699208811-04:00 2024-07-24T22:21:17.698Z INFO Waiting for the pods {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "de793d73-e501-4bac-93cd-351957acaa4a", "replset": "rs0", "size": 3, "pods": 1}
2024-07-24T18:21:17.751281423-04:00 2024-07-24T22:21:17.750Z INFO add new job {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "de793d73-e501-4bac-93cd-351957acaa4a", "name": "ensure-version/mongodb/example-mongodb", "schedule": "0 2 * * *"}
2024-07-24T18:21:17.779567104-04:00 2024-07-24T22:21:17.779Z INFO Cluster state changed {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "de793d73-e501-4bac-93cd-351957acaa4a", "previous": "", "current": "initializing"}
2024-07-24T18:21:17.816437308-04:00 2024-07-24T22:21:17.815Z INFO Replset is not exposed. Make sure each pod in the replset can reach each other. {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "a48c61c2-ac90-4f17-b581-cf8d1a3c5dbe", "replset": "rs0"}
2024-07-24T18:21:18.096219954-04:00 2024-07-24T22:21:18.095Z INFO Waiting for the pods {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "a48c61c2-ac90-4f17-b581-cf8d1a3c5dbe", "replset": "rs0", "size": 3, "pods":
1}
2024-07-24T18:21:22.815267111-04:00 2024-07-24T22:21:22.814Z INFO Replset is not exposed. Make sure each pod in the replset can reach each other. {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "97ee1bca-d292-44c0-ae9b-4adf1dc2570d", "replset": "rs0"}
2024-07-24T22:21:23.071Z INFO Waiting for the pods {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "97ee1bca-d292-44c0-ae9b-4adf1dc2570d", "replset": "rs0", "size": 3, "pods": 1}
2024-07-24T18:21:28.187110713-04:00 2024-07-24T22:21:28.186Z INFO Replset is not exposed. Make sure each pod in the replset can reach each other. {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "9be35d66-6137-4d9d-aa13-2d98d1658fc2", "replset": "rs0"}
2024-07-24T18:21:28.489985716-04:00 2024-07-24T22:21:28.489Z INFO Waiting for the pods {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "9be35d66-6137-4d9d-aa13-2d98d1658fc2", "replset": "rs0", "size": 3, "pods": 1}
2024-07-24T18:21:33.639456067-04:00 2024-07-24T22:21:33.639Z INFO Replset is not exposed. Make sure each pod in the replset can reach each other. {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "10c49def-f0d8-4766-96e6-adf1c185f949", "replset": "rs0"}
2024-07-24T22:21:33.943Z INFO Waiting for the pods {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "10c49def-f0d8-4766-96e6-adf1c185f949", "replset": "rs0", "size": 3, "pods": 1}
2024-07-24T22:21:39.363Z INFO Replset is not exposed. Make sure each pod in the replset can reach each other. {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "21bb72e2-b804-4246-9ce7-45b87bce823d", "replset": "rs0"}
2024-07-24T22:21:39.630Z INFO Waiting for the pods {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "21bb72e2-b804-4246-9ce7-45b87bce823d", "replset": "rs0", "size": 3, "pods": 1}
2024-07-24T22:21:45.632Z INFO Replset is not exposed. Make sure each pod in the replset can reach each other. {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "e8703952-b61d-47a0-9193-27b57146df16", "replset": "rs0"}
2024-07-24T22:21:45.919Z INFO Waiting for the pods {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "e8703952-b61d-47a0-9193-27b57146df16", "replset": "rs0", "size": 3, "pods": 2}
2024-07-24T22:21:46.084Z INFO Replset is not exposed. Make sure each pod in the replset can reach each other. {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "14129a56-3d0e-47fc-981a-7c86b951d285", "replset": "rs0"}
2024-07-24T22:21:46.405Z INFO Waiting for the pods {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "14129a56-3d0e-47fc-981a-7c86b951d285", "replset": "rs0", "size": 3, "pods": 2}
2024-07-24T22:21:51.312Z INFO Replset is not exposed. Make sure each pod in the replset can reach each other. {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "c667ba1e-1d82-4850-b17f-759d61e0e4dd", "replset": "rs0"}
2024-07-24T22:21:51.619Z INFO Waiting for the pods {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "c667ba1e-1d82-4850-b17f-759d61e0e4dd", "replset": "rs0", "size": 3, "pods": 2}
2024-07-24T22:21:57.004Z INFO Replset is not exposed. Make sure each pod in the replset can reach each other. {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "fce820a1-c739-44e9-b578-47c9e3defcee", "replset": "rs0"}
2024-07-24T22:21:57.282Z INFO Waiting for the pods {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "fce820a1-c739-44e9-b578-47c9e3defcee", "replset": "rs0", "size": 3, "pods": 2}
2024-07-24T22:22:02.817Z INFO Replset is not exposed. Make sure each pod in the replset can reach each other. {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "79d72aa6-f450-41f9-8997-e8869282f337", "replset": "rs0"}
2024-07-24T22:22:03.090Z INFO Waiting for the pods {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "79d72aa6-f450-41f9-8997-e8869282f337", "replset": "rs0", "size": 3, "pods": 2}
2024-07-24T22:22:08.509Z INFO Replset is not exposed. Make sure each pod in the replset can reach each other. {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "fbcd6837-1e99-49c7-a801-37a6ab0d143d", "replset": "rs0"}
2024-07-24T22:22:08.795Z INFO Waiting for the pods {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "fbcd6837-1e99-49c7-a801-37a6ab0d143d", "replset": "rs0", "size": 3, "pods": 2}
2024-07-24T22:22:14.213Z INFO Replset is not exposed. Make sure each pod in the replset can reach each other. {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "10fdccca-4165-452b-8b56-043fdf63f8db", "replset": "rs0"}
2024-07-24T22:22:14.511Z INFO Waiting for the pods {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "10fdccca-4165-452b-8b56-043fdf63f8db", "replset": "rs0", "size": 3, "pods": 2}
2024-07-24T22:22:19.927Z INFO Replset is not exposed. Make sure each pod in the replset can reach each other. {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "8a52d2f9-57a4-4a20-94a9-507980a8a66b", "replset": "rs0"}
2024-07-24T22:22:31.851Z INFO Replset is not exposed. Make sure each pod in the replset can reach each other. {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "6e9ad316-6787-4cb8-8146-ed34accbea83", "replset": "rs0"}
2024-07-24T22:22:43.739Z INFO Replset is not exposed. Make sure each pod in the replset can reach each other. {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "41260c8c-bd30-4b66-a827-5fbb1a2b7081", "replset": "rs0"}
2024-07-24T22:22:55.687Z INFO Replset is not exposed. Make sure each pod in the replset can reach each other. {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "7c3eb48f-59d5-4892-a2d6-f75118109e5c", "replset": "rs0"}
2024-07-24T22:23:07.532Z INFO Replset is not exposed. Make sure each pod in the replset can reach each other. {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "727a406f-4f22-4e05-b1f3-7d4856077402", "replset": "rs0"}
2024-07-24T22:23:19.398Z INFO Replset is not exposed. Make sure each pod in the replset can reach each other. {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "18cf442f-00ab-4605-82e3-4eec2c330abd", "replset": "rs0"}
2024-07-24T22:23:31.221Z INFO Replset is not exposed. Make sure each pod in the replset can reach each other. {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "d018f4e4-efa9-4749-95b4-fdd6e9929538", "replset": "rs0"}
2024-07-24T22:23:43.030Z INFO Replset is not exposed. Make sure each pod in the replset can reach each other. {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "5ebc4aed-b855-4483-b404-58580864283d", "replset": "rs0"}
2024-07-24T22:23:54.899Z INFO Replset is not exposed. Make sure each pod in the replset can reach each other. {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "8b2c2559-2804-4388-a798-732bd5986536", "replset": "rs0"}
2024-07-24T22:24:06.718Z INFO Replset is not exposed. Make sure each pod in the replset can reach each other. {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "1680e0de-9a6f-4669-9b8c-8a88e98c5397", "replset": "rs0"}
2024-07-24T22:24:18.539Z INFO Replset is not exposed. Make sure each pod in the replset can reach each other. {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "81a4d31a-1abc-4eaa-8b22-823200d9c57f", "replset": "rs0"}
2024-07-24T22:24:30.317Z INFO Replset is not exposed. Make sure each pod in the replset can reach each other. {"controller": "psmdb-controller", "object": {"name":"example-mongodb","namespace":"mongodb"}, "namespace": "mongodb", "name": "example-mongodb", "reconcileID": "58e30cba-39a6-4d2a-88af-2cb736dce724", "replset": "rs0"}
2024-07-24T22:24:43.680Z INFO Replset is not exposed. Make sure each pod in the replset can reach each other. {"controller": "psmdb-controller",
Hello,
I am experiencing an issue where the Persistent Volume Claim (PVC) remains in the Pending
state when deploying MongoDB using the Percona Server for MongoDB Operator. The PVC does not automatically get bound to a Persistent Volume (PV), despite having a default storage class configured in the Kubernetes cluster.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: p-pm-mongodbtest
spec:
project: default
destination:
namespace: p-pm-mongodbtest
server: https://kubernetes.default.svc
syncPolicy:
automated: {}
syncOptions:
- CreateNamespace=true
source:
repoURL: https://charts.rock8s.com
chart: mongodb
targetRevision: "1.15.0"
helm:
values: |
config:
debug: false
istio: false
mongodb:
replicas: 1
resources:
requests:
cpu: 300m
memory: 0.5G
limits:
cpu: 300m
memory: 0.5G
pmm:
enabled: false
service:
mongodb:
type: ClusterIP
tls:
enabled: false
persistence:
accessMode: ReadWriteOnce
size: 1Gi
storageClassName: ""
kanister:
enabled: false
schedule: '0 0 * * *'
kubectl get pvc -n p-pm-mongodbtest
The PVC should be bound to a PV and not remain in the Pending
state.
The PVC remains in the Pending
state and is not bound to any PV. Here are the details of the PVC:
kubectl describe pvc mongod-data-mongodb-rs0-0 -n p-pm-mongodbtest
Name: mongod-data-mongodb-rs0-0
Namespace: p-pm-mongodbtest
StorageClass:
Status: Pending
Volume:
Labels: app.kubernetes.io/component=mongod
app.kubernetes.io/instance=mongodb
app.kubernetes.io/managed-by=percona-server-mongodb-operator
app.kubernetes.io/name=percona-server-mongodb
app.kubernetes.io/part-of=percona-server-mongodb
app.kubernetes.io/replset=rs0
Annotations: <none>
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By: mongodbtest-rs0-0
Events: <none>
kubectl get storageclass
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
default kubernetes.io/aws-ebs Delete Immediate false 190d
efs-sc efs.csi.aws.com Delete Immediate false 142d
gp2 kubernetes.io/aws-ebs Delete Immediate false 190d
kops-csi-1-21 (default) ebs.csi.aws.com Delete WaitForFirstConsumer true 190d
kops-ssd-1-17 kubernetes.io/aws-ebs Delete WaitForFirstConsumer true 190d
As a temporary workaround, I manually specified the storage class in the PVC definition, which allowed it to get bound:
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 4Gi
storageClassName: "kops-csi-1-21"
volumeMode: Filesystem
I would appreciate any guidance on how to resolve this issue and ensure the PVC gets bound automatically without needing to specify the storage class explicitly.
Thank you!
Kubernetes: v1.22.0
Operator: Percona Server for MongoDB Operator 1.15.0
Database: Percona Server for MongoDB 4.4.6-8
Client Version: v1.30.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.26.11
Hi! I want to make sure my pvcs use a storageclass with allowVolumeExpansion, so I've added these helm values to make sure the storageclass is getting set:
replsets:
- name: rs0
volumeSpec:
pvc:
storageClassName: custom
sharding:
configrs:
volumeSpec:
pvc:
storageClassName: custom
And I'm seeing the pvcTemplate ending up in the corresponding statefulsets:
volumeClaimTemplates:
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
creationTimestamp: null
name: mongod-data
namespace: percona
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 3Gi
storageClassName: custom
volumeMode: Filesystem
status:
phase: Pending
But unfortunately, the cfg pvcs seem to be created w the EKS default storage class:
% k get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
mongod-data-mongodb-cfg-0 Bound pvc-5c40c40a-d1ce-4636-b3e2-1e9c36755047 3Gi RWO gp2 64m
mongod-data-mongodb-cfg-1 Bound pvc-d3b39739-780a-4358-8487-1db81dd9eb5e 3Gi RWO gp2 63m
mongod-data-mongodb-cfg-2 Bound pvc-115a1fd6-bd83-4235-beea-f366aa18bb3d 3Gi RWO gp2 63m
mongod-data-mongodb-rs0-0 Bound pvc-e966d021-f864-4122-9c3a-deb85118a252 3Gi RWO custom 64m
mongod-data-mongodb-rs0-1 Bound pvc-dc2846c4-4bf1-437d-b061-2e2feb372820 3Gi RWO custom 63m
mongod-data-mongodb-rs0-2 Bound pvc-6b3868e3-d880-47b6-923f-b635278f0cea 3Gi RWO custom 63m
Any idea why that would be? I've deleted and redeployed a few times. Any ideas?
setting
spec:
allowUnsafeConfigurations: true
replsets:
configuration: |
net:
tls:
mode: requireTLS
means that requireTLS is (silently) ignored. From the code this appears to be because "unsafe" means both "less than 3 replicas" (I would like to use a PSA config) and also "don't use TLS certificates for mongo replica authentication"
I would suggest either splitting this flag into two to allow for a PSA config that requires tls, or leaving it and considering a PSA configuration safe.
See repro steps
No response
When configuring AWS IAM Roles for Service Accounts (IRSA) for backups in the Percona MongoDB Operator, the backup process works correctly. However, restoring from the backup fails with a "Forbidden" error when trying to access S3 objects.
"error": "set resync backup list from the store: init storage: get S3 object header: Forbidden: Forbidden\n\tstatus code: 403"
backup:
enabled: true
pitr:
enabled: false
storages:
s3-us-west:
type: s3
s3:
bucket: S3-BACKUP-BUCKET-NAME-HERE
credentialsSecret: "" # empty and it works, the backup use the aws IRSA
prefix: "someprefix"
region: us-west-3
apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDBRestore
metadata:
name: restore1
spec:
clusterName: my-cluster-name
backupName: backup1
storageName: s3-us-west
backupSource:
type: logical
storageName: s3-us-west
destination: s3://S3-BACKUP-BUCKET-NAME-HERE/BACKUP-DESTINATION
s3:
credentialsSecret: "" # empty, i want to use the aws IRSA
region: us-west-2
bucket: S3-BACKUP-BUCKET-NAME-HERE
prefix: "some prefix"
No response
Hello,
We are deploying a sharded cluster in EKS containing 3 node. Two of the nodes are working as expected, however, the third is not. It stopped working after a restart.
It's building the Indexes when it suddenly restart and goes into crash loop.
I believe that it might have something to do with liveness probe.
I've altered the liveness probe directly on the operator, but it has not propagated.
No response
I am trying to configure MongoDB backup/restore using Percona Operator. Backups are being successfully uploaded to GCS:
Backups:
========
S3 us s3://https://storage.googleapis.com/mongodb-macpaw-staging-backup/test
Snapshots:
2024-08-01T10:23:21Z 26.75KB <logical> [restore_to_time: 2024-08-01T10:23:32Z]
When I try to create PerconaServerMongoDBRestore resource to restore DB from previously created backup I get an error:
Backups:
========
S3 us s3://https://storage.googleapis.com/mongodb-macpaw-staging-backup/test
Snapshots:
2024-08-01T10:23:21Z 26.75KB <logical> [ERROR: file "2024-08-01T10:23:21Z/test-rs1/oplog": no such file] [2024-08-01T10:23:36Z]
Expected Result:
It looks like restore process expects Oplog file to be named oplog
and to be placed in the root directory of the bucket.
Actual Result:
Oplog file is stored under oplog
directory: oplog/20240802113626-4.20240802113631-3.gz
Additional Information:
When I reupload oplog/20240802113626-4.20240802113631-3.gz
file back into the bucket but with a different name - oplog
, restore is succesfull.
backup:
enabled: true
image: perconalab/percona-server-mongodb-operator:main-backup
serviceAccountName: <serviceAccountName>
storages:
gcs:
type: s3
s3:
region: us
credentialsSecret: <credentialsSecret>
bucket: <bucket>
endpointUrl: https://storage.googleapis.com
prefix: <prefix>
pitr:
enabled: true
tasks:
- name: daily
enabled: true
schedule: "0 11 * * *"
keep: 3
storageName: gcs
apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDBRestore
metadata:
name: restore1
namespace: mongodb
spec:
clusterName: test
backupName: cron-test-20240802124900-fjk52
backupSource:
type: physical
destination: s3://<bucket>/<prefix>
s3:
region: us
credentialsSecret: <credentialsSecret>
bucket: <bucket>
endpointUrl: https://storage.googleapis.com
prefix: <prefix>
No response
If we try to increase the size of the replset mongodb instances storage defined in the spec.replsets[*].volumeSpec.persistentVolumeClaim.resources.requests.storage or spec.sharding.configsvrReplSet.volumeSpec.persistentVolumeClaim.resources.requests.storage CR attributes, the operator is not able to update the related replset statefulsets and pvc as we can see from its logs :
"Forbidden: updates to statefulset spec for fields other than โฆ"
As far as I know, the only way to scale up the storage size of these instances is to directly update the related pvc definition, assuming that these pvc are provided by storageclasses allowing the volume expansion of course. But this is not very convenient as the CR definition and related statefulsets are not updated behind.
It would be a great improvement to allow volume expansion from the CR definition by creating external pvc attached to the pod using Claims As Volumes method as implemented with the PostgreSQL operator instead of the dynamic pvc creation from statefulset definition with the Volume Claim Templates method.
Steps to reproduce this issue :
1 - Create a psmdb cluster with one replset configure with its storage size
2 - Wait for the cluster to be ready and see the newly created pvc with the desired storage size
3 - Try to scale up the instances by increasing spec.replsets[0].volumeSpec.persistentVolumeClaim.resources.requests.storage attribute in the CR definition
4 - See that the pvc storage size is not updated because the operator is unable to update the related statefulset definition as mentioned in its logs
No
No response
Add the ability to configure syslog and add option to mount additional volumes, so i can use a sidecar like fluentbit to collect audit logs
Currently i cannot access audit logs in syslog
or file
format as i cannot access these directories from a sidecar. Having the option to modify rsyslog.conf or mounting volumes on the mongod
pod will allow me to collect the logs using fluentbit
In the existing operator, we can add volumes and mounts to sidecars, but there is no extraVolumes
or additionalVolumes
property available for the mongod replicaset itself, so i cannot expose directories for fluentbit to collect from
For our SIEM monitoring, we have a requirement to collect audit events from MongoDB
Maybe
No response
secrets are deleted when delete-psmdb-pvc has been set.
the problem is sometimes the percona secrets (cr.Spec.Secrets.Users,
"internal-" + cr.Name + "-users") are respectively being recreated right after the deletion in reconcileUsersSecret, reconcileUsers func calls. Sometimes not.
This logic of deleting secrets causes issues/side effects in tha case that the psmdb CR might be re-created after to recover a deleted cluster for example.
To work around this logic we try to overwrite/create the user secret with the previous data (leaving to the operator to sync the secrets, etc), but sometimes recreating psmdb lateron ends up creating the mongodb pods successfully with some auth errors on the mongo pods though, sometimes not. the psmdb does not go in initializing status even. (there are logs on the operator: not found the internal users secret.)
Hi,
We use the operator to do automatic backups to S3 buckets on EKS, which works fine, as the backups are done by the agent container running in the mongodb pods.
We use IRSA, so the EKS pods are using the AWS IAM roles to access resources like S3 buckets, which is working fine on the agent container, but it seems that it's not working on the operator.
When the operator tries to list the backups it fails (for restore and reconcile):
2024-02-08T09:59:46.621Z ERROR failed to run finalizer {"controller": "perconaservermongodbbackup-controller", "object": {"name":"BACKUP-S3-BUCKET",
โ error": "delete files from storage: get file list: \"2024-01-09T00:47:41Z/\": get backup list: AccessDenied: Access Denied\n\tstatus code: 403
Now the operator uses the EKS node's role, if I grant access to the node role and allow the role in the s3 policy, everything works fine.
No response
MongoDB Backup is stuck on Status:Waiting and backup-agent container is not doing anything after Kubernetes scheduler restarted the backup-agent container during the execution of a restore:
I expect to see an ongoing backup after asking for a backup through the PerconaServerMongoDBBackup yml definition, when other actions (backups / restores) are not in progress.
Start a MongoDB cluster in unsafe mode with only 1 replica (this is usefull for development environments) and fill it with some data (let's say about 600MB of gzipped data);
Do a MongoDB backup and wait for the completion (Status = Ready) with the following yml (this will upload the backup to our AWS S3 bucket):
apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDBBackup
metadata:
finalizers:
- delete-backup
name: backup1
spec:
clusterName: mongodb-percona-cluster
storageName: eu-central-1
type: logical
Drop collections on MongoDB replicaset (just to avoid the _id clashes at next point);
Now ask for a restore of the above backup with the following yml (this works as intended since I saw the logs and the data inside MongoDB ReplicaSet):
apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDBRestore
metadata:
name: restore1
spec:
clusterName: mongodb-percona-cluster
backupName: backup1
Ask for another backup with the following yml (keep in mind that at this point the previous restore process is still in progress)
apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDBBackup
metadata:
finalizers:
- delete-backup
name: backup2
spec:
clusterName: mongodb-percona-cluster
storageName: eu-central-1
type: logical
The backup2 will be put on Status=Waiting;
At this point Kubernetes scheduler should kill the backup-agent container from the MongoDB replica pod because of memory issues and restart it;
Now if you do a kubectl get psmdb-backup
, you'll see that backup2 is in Error status and if you do a kubectl get psmdb-restore
, you'll see that restore1 is also in Error status (OK, I can take that);
From this point onwards, no backup/restore will be possible through any yml, because they'll be appended as Status=Waiting.
The new backup-agent container logs state that it is waiting for incoming requests:
2024/03/05 16:36:01 [entrypoint] starting `pbm-agent`
2024-03-05T16:36:05.000+0000 I pbm-agent:
Version: 2.3.0
Platform: linux/amd64
GitCommit: 3b1c2e263901cf041c6b83547f6f28ac2879911f
GitBranch: release-2.3.0
BuildTime: 2023-09-20_14:42_UTC
GoVersion: go1.19
2024-03-05T16:36:05.000+0000 I starting PITR routine
2024-03-05T16:36:05.000+0000 I node: rs0/mongodb-percona-cluster-rs0-0.mongodb-percona-cluster-rs0.default.svc.cluster.local:27017
2024-03-05T16:36:05.000+0000 I listening for the commands
Same bug applies also on cronjobs (so it's not an issue triggered by the on demand backup/restore requests): they are kept in Waiting status.
The bug does NOT happen when using a ReplicaSet with at least 3 replicas (the default topology).
This is written in Go, it should be 1h tops to add arm64 support to the container image. Please do this.
I am trying to setup backup for my Percona MongoDB. I notice percona-mongodb-operator
has error log once I have setup the backup through CRD.
I am seeing the following log in percona-mongodb-operator
:
2024-07-15 10:32:10.581 {"level":"info","ts":1721035930.5811331,"msg":"Configured PBM storage","controller":"psmdb-controller","object":{"name":"mongodb-cluster","namespace":"psmdb"},"namespace":"psmdb","name":"mongodb-cluster","reconcileID":"4a52e94c-4b22-4fa9-8f19-f518b4deea3e"}
2024-07-15 10:31:59.527 {"level":"info","ts":1721035919.5271955,"msg":"Setting PBM config","controller":"psmdb-controller","object":{"name":"mongodb-cluster","namespace":"psmdb"},"namespace":"psmdb","name":"mongodb-cluster","reconcileID":"4a52e94c-4b22-4fa9-8f19-f518b4deea3e","backup":"mongodb-cluster"}
2024-07-15 10:31:59.518 {"level":"info","ts":1721035919.5187798,"msg":"Configuring PBM with storage","controller":"psmdb-controller","object":{"name":"mongodb-cluster","namespace":"psmdb"},"namespace":"psmdb","name":"mongodb-cluster","reconcileID":"4a52e94c-4b22-4fa9-8f19-f518b4deea3e","storage":"azure-blob"}
2024-07-15 11:07:54.216 {"level":"error","ts":1721038074.215949,"msg":"Reconciler error","controller":"psmdb-controller","object":{"name":"mongodb-cluster","namespace":"psmdb"},"namespace":"psmdb","name":"mongodb-cluster","reconcileID":"6185461f-7820-4c21-a998-aa44130c905d","error":"reconcile backup tasks: there is no storage in cluster mongodb-cluster for weekly task","errorVerbose":"there is no storage in cluster mongodb-cluster for weekly task\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileBackupTasks\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/backup.go:46\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:395\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\nreconcile backup tasks\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:397\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222"}
2024-07-15 10:56:58.633 {"level":"error","ts":1721037418.6333563,"msg":"Reconciler error","controller":"psmdb-controller","object":{"name":"mongodb-cluster","namespace":"psmdb"},"namespace":"psmdb","name":"mongodb-cluster","reconcileID":"ae4e3f3c-8387-48e2-9afb-38c4b338131f","error":"reconcile backup tasks: there is no storage in cluster mongodb-cluster for weekly task","errorVerbose":"there is no storage in cluster mongodb-cluster for weekly task\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileBackupTasks\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/backup.go:46\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:395\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\nreconcile backup tasks\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:397\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222"}
2024-07-15 10:51:30.730 {"level":"error","ts":1721037090.730234,"msg":"Reconciler error","controller":"psmdb-controller","object":{"name":"mongodb-cluster","namespace":"psmdb"},"namespace":"psmdb","name":"mongodb-cluster","reconcileID":"24dabe0a-6e9d-4352-99f1-3549ecc265dd","error":"reconcile backup tasks: there is no storage in cluster mongodb-cluster for weekly task","errorVerbose":"there is no storage in cluster mongodb-cluster for weekly task\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileBackupTasks\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/backup.go:46\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:395\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\nreconcile backup tasks\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:397\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222"}
kubectl apply -f https://raw.githubusercontent.com/percona/percona-server-mongodb-operator/v1.16.1/deploy/cw-bundle.yaml
---
apiVersion: v1
kind: Namespace
metadata:
name: psmdb
labels:
pod-security.kubernetes.io/enforce: baseline
---
apiVersion: v1
kind: Secret
metadata:
name: azure-blob-secret
namespace: psmdb
type: Opaque
stringData:
AZURE_STORAGE_ACCOUNT_NAME: "$AZURE_STORAGE_ACCOUNT_NAME"
AZURE_STORAGE_ACCOUNT_KEY: "$AZURE_STORAGE_ACCOUNT_KEY"
---
apiVersion: v1
kind: Secret
metadata:
name: mongodb-user-secret
namespace: psmdb
type: Opaque
stringData:
MONGODB_BACKUP_USER: "$MONGODB_BACKUP_USER"
MONGODB_BACKUP_PASSWORD: "$MONGODB_BACKUP_PASSWORD"
MONGODB_DATABASE_ADMIN_USER: "$MONGODB_DATABASE_ADMIN_USER"
MONGODB_DATABASE_ADMIN_PASSWORD: "$MONGODB_DATABASE_ADMIN_PASSWORD"
MONGODB_CLUSTER_ADMIN_USER: "$MONGODB_CLUSTER_ADMIN_USER"
MONGODB_CLUSTER_ADMIN_PASSWORD: "$MONGODB_CLUSTER_ADMIN_PASSWORD"
MONGODB_CLUSTER_MONITOR_USER: "$MONGODB_CLUSTER_MONITOR_USER"
MONGODB_CLUSTER_MONITOR_PASSWORD: "$MONGODB_CLUSTER_MONITOR_PASSWORD"
MONGODB_USER_ADMIN_USER: "$MONGODB_USER_ADMIN_USER"
MONGODB_USER_ADMIN_PASSWORD: "$MONGODB_USER_ADMIN_PASSWORD"
---
apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDB
metadata:
name: mongodb-cluster
namespace: psmdb
finalizers:
- delete-psmdb-pods-in-order
spec:
platform: kubernetes
crVersion: 1.16.1
image: percona/percona-server-mongodb:7.0.11-multi
imagePullPolicy: IfNotPresent
initImage: percona/percona-server-mongodb-operator:1.16.1
clusterServiceDNSMode: Internal
multiCluster:
enabled: false
secrets:
users: mongodb-user-secret
encryptionKey: mongodb-encryption-secret
unsafeFlags:
tls: false
# set this to false, and replsets.size to the number of node, if we have a bigger node pool
replsetSize: true
tls:
mode: preferTLS
allowInvalidCertificates: false
upgradeOptions:
apply: Disabled
replsets:
- name: rs0
size: 1
serviceAccountName: default
livenessProbe:
failureThreshold: 4
initialDelaySeconds: 60
periodSeconds: 30
timeoutSeconds: 10
startupDelaySeconds: 7200
readinessProbe:
failureThreshold: 8
initialDelaySeconds: 10
periodSeconds: 3
successThreshold: 1
timeoutSeconds: 2
volumeSpec:
persistentVolumeClaim:
accessModes: [ "ReadWriteOnce" ]
storageClassName: managed-csi
resources:
requests:
# NOTE it has to exactly match the size of the PV
storage: 5Gi
expose:
enabled: true
exposeType: ClusterIP
nonvoting:
enabled: false
size: 1
arbiter:
enabled: false
size: 1
backup:
enabled: true
image: percona/percona-backup-mongodb:2.5.0-multi
resources:
limits:
cpu: 300m
memory: 1Gi
requests:
cpu: 100m
memory: 0.5Gi
containerSecurityContext:
privileged: false
storages:
azure-blob:
type: azure
azure:
container: supportassistant
prefix: psmdb
credentialsSecret: azure-blob-secret
pitr:
enabled: false
tasks:
- name: weekly
enabled: true
keep: 3
schedule: 0 0 * * 0
compressionType: gzip
type: physical
kubectl logs deploy/percona-server-mongodb-operator -n psmdb-operator
No response
Hi all,
Iโve been trying to debug this for a good while now, I had a pretty large mongodb deployment and I thought I had a lot of logs simply due to its size. After some digging tho it seemed that they were mostly errors. The strange thing is that the cluster still appears to work. Iโve removed most of the components including sharding, backups, and pmm but Iโm still seeing the errors even with a single replicaset. Iโve also disabled istio and effectively turned off the firewall. Iโm deploying this cluster using the ansible helm module, and below Iโll paste the full config and a big sample of the logs.
and here is a sample of the logs:
{"t":{"$date":"2024-01-16T22:05:09.468+00:00"},"s":"D1", "c":"ASSERT", "id":23074, "ctx":"conn601","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":299}}
{"t":{"$date":"2024-01-16T22:05:09.468+00:00"},"s":"D1", "c":"ASSERT", "id":23074, "ctx":"conn601","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":444}}
{"t":{"$date":"2024-01-16T22:05:12.483+00:00"},"s":"D1", "c":"ASSERT", "id":23074, "ctx":"conn602","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":299}}
{"t":{"$date":"2024-01-16T22:05:12.484+00:00"},"s":"D1", "c":"ASSERT", "id":23074, "ctx":"conn602","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":444}}
{"t":{"$date":"2024-01-16T22:05:12.531+00:00"},"s":"D1", "c":"REPL", "id":6208204, "ctx":"conn469","msg":"Error while waiting for hello response","attr":{"status":{"code":262,"codeName":"ExceededTimeLimit","errmsg":"operation exceeded time limit"}}}
{"t":{"$date":"2024-01-16T22:05:13.179+00:00"},"s":"D1", "c":"REPL", "id":6208204, "ctx":"conn444","msg":"Error while waiting for hello response","attr":{"status":{"code":262,"codeName":"ExceededTimeLimit","errmsg":"operation exceeded time limit"}}}
{"t":{"$date":"2024-01-16T22:05:13.906+00:00"},"s":"D1", "c":"ASSERT", "id":23074, "ctx":"conn607","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":299}}
{"t":{"$date":"2024-01-16T22:05:13.906+00:00"},"s":"D1", "c":"ASSERT", "id":23074, "ctx":"conn607","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":444}}
{"t":{"$date":"2024-01-16T22:05:13.907+00:00"},"s":"D1", "c":"ASSERT", "id":23074, "ctx":"conn608","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":299}}
{"t":{"$date":"2024-01-16T22:05:13.907+00:00"},"s":"D1", "c":"REPL", "id":6208204, "ctx":"conn606","msg":"Error while waiting for hello response","attr":{"status":{"code":279,"codeName":"ClientDisconnect","errmsg":"operation was interrupted"}}}
{"t":{"$date":"2024-01-16T22:05:13.907+00:00"},"s":"D1", "c":"ASSERT", "id":23074, "ctx":"conn606","msg":"User assertion","attr":{"error":"ClientDisconnect: operation was interrupted","file":"src/mongo/db/repl/replication_coordinator_impl.cpp","line":2453}}
{"t":{"$date":"2024-01-16T22:05:13.907+00:00"},"s":"D1", "c":"ASSERT", "id":23074, "ctx":"conn608","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":444}}
{"t":{"$date":"2024-01-16T22:05:13.908+00:00"},"s":"D1", "c":"ASSERT", "id":23074, "ctx":"conn606","msg":"User assertion","attr":{"error":"ClientDisconnect: operation was interrupted","file":"src/mongo/util/future_impl.h","line":1104}}
{"t":{"$date":"2024-01-16T22:05:13.908+00:00"},"s":"D1", "c":"ASSERT", "id":23074, "ctx":"conn606","msg":"User assertion","attr":{"error":"ClientDisconnect: operation was interrupted","file":"src/mongo/util/future_impl.h","line":1087}}
{"t":{"$date":"2024-01-16T22:05:13.908+00:00"},"s":"D1", "c":"ASSERT", "id":23074, "ctx":"conn606","msg":"User assertion","attr":{"error":"ClientDisconnect: operation was interrupted","file":"src/mongo/util/future_impl.h","line":1104}}
{"t":{"$date":"2024-01-16T22:05:13.908+00:00"},"s":"D1", "c":"ASSERT", "id":23074, "ctx":"conn606","msg":"User assertion","attr":{"error":"ClientDisconnect: operation was interrupted","file":"src/mongo/util/future_impl.h","line":1087}}
{"t":{"$date":"2024-01-16T22:05:13.908+00:00"},"s":"D1", "c":"COMMAND", "id":21962, "ctx":"conn606","msg":"Assertion while executing command","attr":{"command":"hello","db":"admin","commandArgs":{"hello":1,"helloOk":true,"topologyVersion":{"processId":{"$oid":"65a6fbe68c4bda018b2c0028"},"counter":13},"maxAwaitTimeMS":10000,"$db":"admin","$readPreference":{"mode":"primaryPreferred"}},"error":"ClientDisconnect: operation was interrupted"}}
{"t":{"$date":"2024-01-16T22:05:13.908+00:00"},"s":"D1", "c":"ASSERT", "id":23074, "ctx":"conn606","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":299}}
{"t":{"$date":"2024-01-16T22:05:13.908+00:00"},"s":"D1", "c":"ASSERT", "id":23074, "ctx":"conn606","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":444}}
{"t":{"$date":"2024-01-16T22:05:13.913+00:00"},"s":"D1", "c":"ASSERT", "id":23074, "ctx":"conn605","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":299}}
{"t":{"$date":"2024-01-16T22:05:13.913+00:00"},"s":"D1", "c":"ASSERT", "id":23074, "ctx":"conn604","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":299}}
{"t":{"$date":"2024-01-16T22:05:13.913+00:00"},"s":"D1", "c":"ASSERT", "id":23074, "ctx":"conn605","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":444}}
{"t":{"$date":"2024-01-16T22:05:13.913+00:00"},"s":"D1", "c":"ASSERT", "id":23074, "ctx":"conn604","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":444}}
{"t":{"$date":"2024-01-16T22:05:13.913+00:00"},"s":"D1", "c":"REPL", "id":6208204, "ctx":"conn603","msg":"Error while waiting for hello response","attr":{"status":{"code":279,"codeName":"ClientDisconnect","errmsg":"operation was interrupted"}}}
{"t":{"$date":"2024-01-16T22:05:13.913+00:00"},"s":"D1", "c":"ASSERT", "id":23074, "ctx":"conn603","msg":"User assertion","attr":{"error":"ClientDisconnect: operation was interrupted","file":"src/mongo/db/repl/replication_coordinator_impl.cpp","line":2453}}
{"t":{"$date":"2024-01-16T22:05:13.913+00:00"},"s":"D1", "c":"ASSERT", "id":23074, "ctx":"conn603","msg":"User assertion","attr":{"error":"ClientDisconnect: operation was interrupted","file":"src/mongo/util/future_impl.h","line":1104}}
{"t":{"$date":"2024-01-16T22:05:13.913+00:00"},"s":"D1", "c":"ASSERT", "id":23074, "ctx":"conn603","msg":"User assertion","attr":{"error":"ClientDisconnect: operation was interrupted","file":"src/mongo/util/future_impl.h","line":1087}}
{"t":{"$date":"2024-01-16T22:05:13.913+00:00"},"s":"D1", "c":"ASSERT", "id":23074, "ctx":"conn603","msg":"User assertion","attr":{"error":"ClientDisconnect: operation was interrupted","file":"src/mongo/util/future_impl.h","line":1104}}
{"t":{"$date":"2024-01-16T22:05:13.913+00:00"},"s":"D1", "c":"ASSERT", "id":23074, "ctx":"conn603","msg":"User assertion","attr":{"error":"ClientDisconnect: operation was interrupted","file":"src/mongo/util/future_impl.h","line":1087}}
{"t":{"$date":"2024-01-16T22:05:13.913+00:00"},"s":"D1", "c":"COMMAND", "id":21962, "ctx":"conn603","msg":"Assertion while executing command","attr":{"command":"hello","db":"admin","commandArgs":{"hello":1,"helloOk":true,"topologyVersion":{"processId":{"$oid":"65a6fbe68c4bda018b2c0028"},"counter":13},"maxAwaitTimeMS":10000,"$db":"admin","$readPreference":{"mode":"primaryPreferred"}},"error":"ClientDisconnect: operation was interrupted"}}
{"t":{"$date":"2024-01-16T22:05:13.913+00:00"},"s":"D1", "c":"ASSERT", "id":23074, "ctx":"conn603","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":299}}
{"t":{"$date":"2024-01-16T22:05:13.913+00:00"},"s":"D1", "c":"ASSERT", "id":23074, "ctx":"conn603","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":444}}
{"t":{"$date":"2024-01-16T22:05:14.137+00:00"},"s":"D1", "c":"ASSERT", "id":23074, "ctx":"conn609","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":299}}
{"t":{"$date":"2024-01-16T22:05:14.137+00:00"},"s":"D1", "c":"ASSERT", "id":23074, "ctx":"conn609","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":444}}
{"t":{"$date":"2024-01-16T22:05:14.137+00:00"},"s":"D1", "c":"REPL", "id":6208204, "ctx":"conn610","msg":"Error while waiting for hello response","attr":{"status":{"code":279,"codeName":"ClientDisconnect","errmsg":"operation was interrupted"}}}
{"t":{"$date":"2024-01-16T22:05:14.137+00:00"},"s":"D1", "c":"ASSERT", "id":23074, "ctx":"conn610","msg":"User assertion","attr":{"error":"ClientDisconnect: operation was interrupted","file":"src/mongo/db/repl/replication_coordinator_impl.cpp","line":2453}}
{"t":{"$date":"2024-01-16T22:05:14.137+00:00"},"s":"D1", "c":"ASSERT", "id":23074, "ctx":"conn610","msg":"User assertion","attr":{"error":"ClientDisconnect: operation was interrupted","file":"src/mongo/util/future_impl.h","line":1104}}
{"t":{"$date":"2024-01-16T22:05:14.138+00:00"},"s":"D1", "c":"ASSERT", "id":23074, "ctx":"conn610","msg":"User assertion","attr":{"error":"ClientDisconnect: operation was interrupted","file":"src/mongo/util/future_impl.h","line":1087}}
{"t":{"$date":"2024-01-16T22:05:14.138+00:00"},"s":"D1", "c":"ASSERT", "id":23074, "ctx":"conn610","msg":"User assertion","attr":{"error":"ClientDisconnect: operation was interrupted","file":"src/mongo/util/future_impl.h","line":1104}}
{"t":{"$date":"2024-01-16T22:05:14.138+00:00"},"s":"D1", "c":"ASSERT", "id":23074, "ctx":"conn610","msg":"User assertion","attr":{"error":"ClientDisconnect: operation was interrupted","file":"src/mongo/util/future_impl.h","line":1087}}
{"t":{"$date":"2024-01-16T22:05:14.138+00:00"},"s":"D1", "c":"COMMAND", "id":21962, "ctx":"conn610","msg":"Assertion while executing command","attr":{"command":"hello","db":"admin","commandArgs":{"hello":1,"helloOk":true,"topologyVersion":{"processId":{"$oid":"65a6fbe68c4bda018b2c0028"},"counter":13},"maxAwaitTimeMS":10000,"$db":"admin","$readPreference":{"mode":"primaryPreferred"}},"error":"ClientDisconnect: operation was interrupted"}}
{"t":{"$date":"2024-01-16T22:05:14.138+00:00"},"s":"D1", "c":"ASSERT", "id":23074, "ctx":"conn611","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":299}}
{"t":{"$date":"2024-01-16T22:05:14.138+00:00"},"s":"D1", "c":"ASSERT", "id":23074, "ctx":"conn611","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":444}}
{"t":{"$date":"2024-01-16T22:05:14.139+00:00"},"s":"D1", "c":"ASSERT", "id":23074, "ctx":"conn610","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":299}}
{"t":{"$date":"2024-01-16T22:05:14.139+00:00"},"s":"D1", "c":"ASSERT", "id":23074, "ctx":"conn610","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":444}}
{"t":{"$date":"2024-01-16T22:05:15.464+00:00"},"s":"D1", "c":"ASSERT", "id":23074, "ctx":"conn612","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":299}}
{"t":{"$date":"2024-01-16T22:05:15.464+00:00"},"s":"D1", "c":"ASSERT", "id":23074, "ctx":"conn612","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":444}}
"t":{"$date":"2024-01-16T22:10:25.733+00:00"},"s":"D1", "c":"ASSERT", "id":23074, "ctx":"conn1060","msg":"User assertion","attr":{"error":"HostUnreachable: Connection closed by peer","file":"src/mongo/transport/service_state_machine.cpp","line":444}}
Custom resource is reporting a ready status, here are its logs:
2024-01-22T17:42:38.722Z INFO Created a new mongo key {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "26ab8173-6077-437d-9d50-6123d1c182a2", "KeyName": "kev-test-psmdb-db-mongodb-keyfile"}
2024-01-22T17:42:38.731Z INFO Created a new mongo key {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "26ab8173-6077-437d-9d50-6123d1c182a2", "KeyName": "kev-test-psmdb-db-mongodb-encryption-key"}
2024-01-22T17:42:38.779Z INFO Waiting for the pods {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "26ab8173-6077-437d-9d50-6123d1c182a2", "replset": "shard", "size": 3, "pods": 0}
2024-01-22T17:42:38.799Z INFO Cluster state changed {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "26ab8173-6077-437d-9d50-6123d1c182a2", "previous": "", "current": "initializing"}
2024-01-22T17:42:38.850Z INFO Waiting for the pods {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "af333758-e95d-4100-9f52-cfe3cbf57cdb", "replset": "shard", "size": 3, "pods": 0}
2024-01-22T17:42:43.867Z INFO Waiting for the pods {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "f074b1e2-166a-456d-b8c4-c27b61b0d77a", "replset": "shard", "size": 3, "pods": 1}
2024-01-22T17:42:43.947Z INFO Waiting for the pods {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "009f498e-7342-427e-8e6e-17b82542081d", "replset": "shard", "size": 3, "pods": 1}
2024-01-22T17:42:48.943Z INFO Waiting for the pods {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "987a4804-a466-4fe3-8ab1-3ae5140fd989", "replset": "shard", "size": 3, "pods": 1}
2024-01-22T17:42:53.989Z INFO Waiting for the pods {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "cfab78a1-aa52-4adb-a08c-a25b6b5bf0a5", "replset": "shard", "size": 3, "pods": 1}
2024-01-22T17:42:59.029Z INFO Waiting for the pods {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "e81e933d-7d9e-4ef5-b173-cf0bd43303ac", "replset": "shard", "size": 3, "pods": 1}
2024-01-22T17:43:04.078Z INFO Waiting for the pods {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "50774dcf-0a7b-4a39-b645-b9c79e266449", "replset": "shard", "size": 3, "pods": 1}
2024-01-22T17:43:09.162Z INFO Waiting for the pods {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "3fa6b94b-3b6b-4235-ad28-c66e97550762", "replset": "shard", "size": 3, "pods": 2}
2024-01-22T17:43:09.260Z INFO Waiting for the pods {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "49550521-00e6-4d8c-be77-756ca1e99cd0", "replset": "shard", "size": 3, "pods": 2}
2024-01-22T17:43:14.248Z INFO Waiting for the pods {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "df050785-9e5c-4a0e-b4ed-89eb420d1696", "replset": "shard", "size": 3, "pods": 2}
2024-01-22T17:43:19.312Z INFO Waiting for the pods {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "fe180278-1aaf-480a-a804-398c9df72e9a", "replset": "shard", "size": 3, "pods": 2}
2024-01-22T17:43:24.358Z INFO Waiting for the pods {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "c48ac446-c9f1-409d-91b5-f6047cc05b3c", "replset": "shard", "size": 3, "pods": 2}
2024-01-22T17:43:29.398Z INFO Waiting for the pods {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "ad5fc822-752e-4b4e-8f4a-183bff39c200", "replset": "shard", "size": 3, "pods": 2}
2024-01-22T17:43:34.453Z INFO Waiting for the pods {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "3ebe6019-550b-4ac0-b2cc-9c5e55b7a3f2", "replset": "shard", "size": 3, "pods": 2}
2024-01-22T17:43:49.532Z INFO initiating replset {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "0d2ff7eb-2cdc-4d08-93e1-a4dc02dc10a4", "replset": "shard", "pod": "kev-test-psmdb-db-shard-0"}
2024-01-22T17:43:58.669Z INFO replset initialized {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "0d2ff7eb-2cdc-4d08-93e1-a4dc02dc10a4", "replset": "shard", "pod": "kev-test-psmdb-db-shard-0"}
2024-01-22T17:43:59.205Z INFO Fixing member tags {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "459a4f0b-f928-4fc3-ad20-e22c5d92c4c3", "replset": "shard"}
2024-01-22T17:43:59.205Z DEBUG Running replSetReconfig config {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "459a4f0b-f928-4fc3-ad20-e22c5d92c4c3", "cfg": {"_id":"shard","version":2,"members":[{"_id":0,"host":"kev-test-psmdb-db-shard-0.kev-test-psmdb-db-shard.app.svc.dev.ahq:27017","arbiterOnly":false,"buildIndexes":true,"hidden":false,"priority":1,"tags":{"podName":"kev-test-psmdb-db-shard-0","serviceName":"kev-test-psmdb-db"},"secondaryDelaySecs":0,"votes":1}],"protocolVersion":1,"settings":{"chainingAllowed":true,"heartbeatIntervalMillis":2000,"heartbeatTimeoutSecs":10,"electionTimeoutMillis":10000,"catchUpTimeoutMillis":-1,"getLastErrorDefaults":{"w":1,"wtimeout":0},"replicaSetId":"65aea9578f40f2bb98bd0171"},"writeConcernMajorityJournalDefault":true}}
2024-01-22T17:43:59.209Z INFO Adding new nodes {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "459a4f0b-f928-4fc3-ad20-e22c5d92c4c3", "replset": "shard"}
2024-01-22T17:43:59.209Z DEBUG Running replSetReconfig config {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "459a4f0b-f928-4fc3-ad20-e22c5d92c4c3", "cfg": {"_id":"shard","version":3,"members":[{"_id":0,"host":"kev-test-psmdb-db-shard-0.kev-test-psmdb-db-shard.app.svc.dev.ahq:27017","arbiterOnly":false,"buildIndexes":true,"hidden":false,"priority":1,"tags":{"podName":"kev-test-psmdb-db-shard-0","serviceName":"kev-test-psmdb-db"},"secondaryDelaySecs":0,"votes":1},{"_id":1,"host":"kev-test-psmdb-db-shard-1.kev-test-psmdb-db-shard.app.svc.dev.ahq:27017","arbiterOnly":false,"buildIndexes":true,"hidden":false,"priority":2,"tags":{"podName":"kev-test-psmdb-db-shard-1","serviceName":"kev-test-psmdb-db"},"votes":1}],"protocolVersion":1,"settings":{"chainingAllowed":true,"heartbeatIntervalMillis":2000,"heartbeatTimeoutSecs":10,"electionTimeoutMillis":10000,"catchUpTimeoutMillis":-1,"getLastErrorDefaults":{"w":1,"wtimeout":0},"replicaSetId":"65aea9578f40f2bb98bd0171"},"writeConcernMajorityJournalDefault":true}}
2024-01-22T17:43:59.231Z INFO Configuring member votes and priorities {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "459a4f0b-f928-4fc3-ad20-e22c5d92c4c3", "replset": "shard"}
2024-01-22T17:43:59.231Z DEBUG Running replSetReconfig config {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "459a4f0b-f928-4fc3-ad20-e22c5d92c4c3", "cfg": {"_id":"shard","version":4,"members":[{"_id":0,"host":"kev-test-psmdb-db-shard-0.kev-test-psmdb-db-shard.app.svc.dev.ahq:27017","arbiterOnly":false,"buildIndexes":true,"hidden":false,"priority":2,"tags":{"podName":"kev-test-psmdb-db-shard-0","serviceName":"kev-test-psmdb-db"},"secondaryDelaySecs":0,"votes":1},{"_id":1,"host":"kev-test-psmdb-db-shard-1.kev-test-psmdb-db-shard.app.svc.dev.ahq:27017","arbiterOnly":false,"buildIndexes":true,"hidden":false,"priority":0,"tags":{"podName":"kev-test-psmdb-db-shard-1","serviceName":"kev-test-psmdb-db"},"votes":0}],"protocolVersion":1,"settings":{"chainingAllowed":true,"heartbeatIntervalMillis":2000,"heartbeatTimeoutSecs":10,"electionTimeoutMillis":10000,"catchUpTimeoutMillis":-1,"getLastErrorDefaults":{"w":1,"wtimeout":0},"replicaSetId":"65aea9578f40f2bb98bd0171"},"writeConcernMajorityJournalDefault":true}}
2024-01-22T17:44:04.188Z INFO Adding new nodes {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "ca1d90c5-512e-4d47-b455-14193e6634af", "replset": "shard"}
2024-01-22T17:44:04.188Z DEBUG Running replSetReconfig config {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "ca1d90c5-512e-4d47-b455-14193e6634af", "cfg": {"_id":"shard","version":6,"members":[{"_id":0,"host":"kev-test-psmdb-db-shard-0.kev-test-psmdb-db-shard.app.svc.dev.ahq:27017","arbiterOnly":false,"buildIndexes":true,"hidden":false,"priority":2,"tags":{"podName":"kev-test-psmdb-db-shard-0","serviceName":"kev-test-psmdb-db"},"secondaryDelaySecs":0,"votes":1},{"_id":1,"host":"kev-test-psmdb-db-shard-1.kev-test-psmdb-db-shard.app.svc.dev.ahq:27017","arbiterOnly":false,"buildIndexes":true,"hidden":false,"priority":0,"tags":{"podName":"kev-test-psmdb-db-shard-1","serviceName":"kev-test-psmdb-db"},"secondaryDelaySecs":0,"votes":0},{"_id":2,"host":"kev-test-psmdb-db-shard-2.kev-test-psmdb-db-shard.app.svc.dev.ahq:27017","arbiterOnly":false,"buildIndexes":true,"hidden":false,"priority":2,"tags":{"podName":"kev-test-psmdb-db-shard-2","serviceName":"kev-test-psmdb-db"},"votes":1}],"protocolVersion":1,"settings":{"chainingAllowed":true,"heartbeatIntervalMillis":2000,"heartbeatTimeoutSecs":10,"electionTimeoutMillis":10000,"catchUpTimeoutMillis":-1,"getLastErrorDefaults":{"w":1,"wtimeout":0},"replicaSetId":"65aea9578f40f2bb98bd0171"},"writeConcernMajorityJournalDefault":true}}
2024-01-22T17:44:04.216Z INFO Configuring member votes and priorities {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "ca1d90c5-512e-4d47-b455-14193e6634af", "replset": "shard"}
2024-01-22T17:44:04.216Z DEBUG Running replSetReconfig config {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "ca1d90c5-512e-4d47-b455-14193e6634af", "cfg": {"_id":"shard","version":7,"members":[{"_id":0,"host":"kev-test-psmdb-db-shard-0.kev-test-psmdb-db-shard.app.svc.dev.ahq:27017","arbiterOnly":false,"buildIndexes":true,"hidden":false,"priority":2,"tags":{"podName":"kev-test-psmdb-db-shard-0","serviceName":"kev-test-psmdb-db"},"secondaryDelaySecs":0,"votes":1},{"_id":1,"host":"kev-test-psmdb-db-shard-1.kev-test-psmdb-db-shard.app.svc.dev.ahq:27017","arbiterOnly":false,"buildIndexes":true,"hidden":false,"priority":2,"tags":{"podName":"kev-test-psmdb-db-shard-1","serviceName":"kev-test-psmdb-db"},"secondaryDelaySecs":0,"votes":1},{"_id":2,"host":"kev-test-psmdb-db-shard-2.kev-test-psmdb-db-shard.app.svc.dev.ahq:27017","arbiterOnly":false,"buildIndexes":true,"hidden":false,"priority":2,"tags":{"podName":"kev-test-psmdb-db-shard-2","serviceName":"kev-test-psmdb-db"},"votes":1}],"protocolVersion":1,"settings":{"chainingAllowed":true,"heartbeatIntervalMillis":2000,"heartbeatTimeoutSecs":10,"electionTimeoutMillis":10000,"catchUpTimeoutMillis":-1,"getLastErrorDefaults":{"w":1,"wtimeout":0},"replicaSetId":"65aea9578f40f2bb98bd0171"},"writeConcernMajorityJournalDefault":true}}
2024-01-22T17:44:16.958Z INFO Cluster state changed {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "de7ea753-5e72-4be0-988a-08850b8ffd9d", "previous": "initializing", "current": "ready"}
2024-01-22T17:44:17.309Z INFO update Mongo version to 6.0.9-7 (fetched from db) {"controller": "psmdb-controller", "object": {"name":"kev-test-psmdb-db","namespace":"app"}, "namespace": "app", "name": "kev-test-psmdb-db", "reconcileID": "d5aee049-1b1a-4b61-a34b-d46db7cb5736"}
- name: Deploy Percona Server for MongoDB
kubernetes.core.helm:
name: kev-test-mongodb
chart_ref: percona/psmdb-db
chart_version: "1.15.1"
release_namespace: app
wait: true
wait_timeout: "10m"
values:
clusterServiceDNSSuffix: 'svc.{{ cluster_domain }}'
finalizers:
- delete-psmdb-pvc
nameOverride: ""
fullnameOverride: ""
crVersion: 1.15.0
pause: false
unmanaged: false
allowUnsafeConfigurations: false
multiCluster:
enabled: false
updateStrategy: SmartUpdate
# updateStrategy: RollingUpdate
upgradeOptions:
versionServiceEndpoint: https://check.percona.com
apply: disabled
schedule: "0 2 * * *"
setFCV: false
image:
repository: percona/percona-server-mongodb
tag: 6.0.9-7
imagePullPolicy: Always
secrets: {}
pmm:
enabled: false
replsets:
- name: shard
size: 3
annotations:
sidecar.istio.io/inject: "false"
configuration: |
security:
enableEncryption: false
systemLog:
verbosity: 1
serviceAccountName: app
storage:
engine: inMemory
inMemory:
engineConfig:
inMemorySizeRatio: 0.9
podDisruptionBudget:
maxUnavailable: 1
expose:
enabled: true
exposeType: ClusterIP
nonvoting:
enabled: false
size: 1
arbiter:
enabled: false
size: 1
resources:
limits:
cpu: "2048m"
memory: "5.0G"
requests:
cpu: "300m"
memory: "0.5G"
volumeSpec:
pvc:
storageClassName: "ceph-block"
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 1Gi
sharding:
enabled: false
backup:
enabled: false
No response
We cannot disable authentication even if we pass this as a conf
security:
authorization: "disabled"
--auth
is being passed, which overrides the mongod.conf
percona-server-mongodb-operator/pkg/psmdb/container.go
Lines 176 to 184 in f10e1aa
Deploy with:
spec:
...
replsets:
- name: rs0
size: 3
configuration: |
security:
authorization: "disabled"
This would be fine if there'a a way to create users using the crd.
Currently, only the mongo-init
initContainer is run.
In order to tune mongodb a bit further I think It'd be good to have a way to configure extra initContainers
and their securityContext
.
I would like to be able to modify the tcp_keepalive
of my service. In order to do so, I need to spawn a privileged initContainer
changing this configuration. See https://www.mongodb.com/docs/kubernetes-operator/v1.25/tutorial/modify-resource-image/#tune-mongodb-kubernetes-resource-docker-images-with-an-initcontainer
Maybe
No response
We have two clusters managed by the same operator running on kubernetes. Daily backup is set up for them, which works for one cluster, but fails for the other. All the settings are the same for backups, both uses the same s3 bucket.
Error message on the CRD:
some of pbm-agents were lost during the backup
State of the backup is error
Checking the logs of the backup-agent container in one of the pods I see, that it's writing the collections, and then it stopps with the following error message:
2024-04-10T11:47:19.097+0000 Mux close namespace XXXXX
2024-04-10T11:47:19.097+0000 done dumping XXXX (0 documents)
2024-04-10T11:47:19.098+0000 writing XXXXX to archive on stdout
2024/04/10 11:47:21 [entrypoint] `pbm-agent` exited with code -1
2024/04/10 11:47:21 [entrypoint] restart in 5 sec โ
2024/04/10 11:47:26 [entrypoint] starting `pbm-agent`
We had a change on this cluster, when it stopped working, but it was just to increase the resources from c5a.large to c5a.4xlarge. First I thought that maybe the backup agent gets OOMKilled, as it now sees, that there are plenty more resources available, so I decreased the resources (as we don't need increased anymore) to c5a.xlarge, but the issue is still the same.
I was not able to enable debug loggin on the backup-agent, maybe it's not even possible. How could I get more details on the error?
I also tried to restart the whole cluster, but still the same.
We haven't changed the resources of the other cluster and the backups are working fine there.
Hi,
I want to deploy a database instance and expose it, my idea was to use a nodeport but I also wanted to choose the port, bur from what I saw there is no way to do that.
I am using the helm chart to deploy my instance but from what I saw it seems to be an issue coming from the operator.
I want to use a nodeport to make things easier and I want to choose the port to keep it consistant between my clusters
I would like to be able to change externalTrafficPolicy to Local for the LoadBalancer service in order to preserve the realIP address
It would be nice if it's configurable with the helm chart values
I would like to see real IP address in the logs using LoadBalancer service
No
No response
We should be able when enabling backups in PerconaServerMongoDb to choose the finalizers or at least to remove the delete-backup finalizer.
backup:
enabled: true
finalizers: false
or
backup:
enabled: true
finalizers: []
The credentials used to push a new backup (or even to retrieve them) should not be able to delete a backup as this would create a security bridge.
Most likely no-one is even able to delete a backup as enabling object-lock should be a standard practice.
This prevent the backup objects from being deleted without manually removing the useless finalizer in this case.
No
No response
I have been trying to get mongodb deployed in a kubernetes cluster. But the value set for size
under replsets doesn't seem to have any effect. The operator is always creating the statefulset with 3 pods.
The operator should respect the passed size and create pods based on it.
pmdb.yml
.replsets:
- name: rs0
size: 2
podSecurityContext:
fsGroup: 1001
runAsGroup: 1001
runAsUser: 1001
resources:
limits:
cpu: "3900m"
memory: 14Gi
requests:
cpu: 2
memory: 10Gi
Deploy psmdb
helm install my-db percona/psmdb-db --namespace=percona -f ./deploy/mongodb/pmdb.yml
No response
The PerconaServerMongoDB status stucks in "Initializing" because the field "status.updatedReplicas" is always smaller than "status.replicas" which is caused by multiple ControllerRevision
When creating new MongoDB cluster from the Helm chart percona/psmdb-db in ArgoCD, there are two controller revisions of the Config Server and ReplicaSet StatefulSet,
kubectl get controllerrevision command result
The issue happens in the logic of "smart.go" function, the controller compares the StatefulSet field "status.updatedReplicas" with "status.replicas". If it's not equal, then the StatefulSet is not up to date
Log details
{"level":"info","ts":1716685730.240358,"msg":"StatefulSet is not up to date","controller":"psmdb-controller","object":{"name":"mongodb-cluster","namespace":"mongodb"},"namespace":"mongodb","name":"mongodb-cluster","reconcileID":"75a29a96-608c-45f4-ac87-e132bf180b29","sts":"mongodb-cluster-cfg"}
The issue does not happen with Operator version 1.15.4 and Database version 1.15.3
Failed to delete history backup data in minio.
pbm only deleted the history json file, but the directory file was not successfully deleted, and I doubt it was because --force was not added.
This is my values.yaml file
backup:
enabled: true
image:
repository: harbor.devops.narwal.com/public_mirror/percona-backup-mongodb
tag: 2.4.1
storages:
minio:
type: s3
s3:
bucket: testing
region: us-east-1
credentialsSecret: backup-credentials-minio
endpointUrl: https://10.10.180.14:9700/
prefix: "mongodb"
insecureSkipTLSVerify: false
pitr:
enabled: true
oplogOnly: false
tasks:
- name: daily-minio-s3
enabled: true
schedule: "0 */1 * * *"
keep: 2
storageName: minio
compressionType: gzip
type: physical
Permissions are all allowed
No response
As far as I know, the CR definition of Percona MongoDB cluster does not provide any configuration attribute to use an existing custom Issuer to generate the tls certificate with cert-manager. A <cluster-name>-psmdb-issuer issuer resource is automatically generated by the operator, then <cluster-name>-ssl and <cluster-name>-ssl-internal certificate resources are issued from it.
The Percona XtraDB cluster provide the tls.issuerConf attributes to specify a custom Issuer for these certificates to be generated.
It would be a great feature to implement this as well for the future releases of Percona MongoDB operator if not planned yet !
No response
No
No response
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.