Path is not writable(/opt/app-root/src), DAS backup script is not able to create noobaa_db.backup at this location: opt/app-root/src in the pod

[nigamshaurya14]

Path is not writable(/opt/app-root/src), DAS backup script is not able to create noobaa_db.backup at this location: opt/app-root/src in the pod

[romayalon]

As discussed on slack, there was no related NooBaa change that could cause this, we suspect it was due to db image change on downstream build.
our suggestion is to change the path, as @baum suggested to /var/lib/pgsql, which is writable by the DB by design.

[nimrod-becker]

@liranmauda @dannyzaken
Was this solved as part of ODF builds?

[liranmauda]

@nimrod-becker I think this should be solved in DAS operator, and the path should be different.
talking to @romayalon it seems that it is a new postgress image that downstream is using.

As discussed on slack, there was no related NooBaa change that could cause this, we suspect it was due to db image change on downstream build. our suggestion is to change the path, as @baum suggested to /var/lib/pgsql, which is writable by the DB by design.

[rkomandu]

@nimrod-becker , as per our last interlock discussion, you would check the Postgres image in ODF 4.12 and the reason for this change.

If you could let us know in a day or two, we would need to work on this change accordingly in our DAS code base.

[nimrod-becker]

It seems 4.12 doesn't have these issues anymore (with no code changes).
Can you please verify with a new deployment? If this still occurs, we need to go with Liran's suggestion

[rkomandu]

we will check once the system is deployed with the ODF downstream build. Asked one person in the team to check for the same but didn't get any response

[nigamshaurya14]

The issue exist on ODF 4.12 as well.
We have verified the same issue, and could able to re-produced on ODF 4.12.0-rc.6 build.
Followed below steps:

[[email protected] backup-folder]# mkdir -p das/scripts

[[email protected] backup-folder]# oc cp ibm-spectrum-scale-das/$(oc -n ibm-spectrum-scale-das get pods -l app=das-endpoint -o=jsonpath='{.items[0].metadata.name}'):scripts/ /tmp/das/scripts

[[email protected] backup-folder]# chmod +x /tmp/das/scripts/*

[[email protected] backup-folder]# ls -ltr /tmp/das/scripts
total 12
-rwxr-xr-x 1 root root 8138 Jan 17 02:03 dasS3Restore.sh
-rwxr-xr-x 1 root root 3953 Jan 17 02:03 dasS3Backup.sh

[[email protected] backup-folder]# mkdir /tmp/das/backup

ERROR :

[[email protected] backup-folder]# /tmp/das/scripts/dasS3Backup.sh /tmp/das/backup
2023-01-17T02:04:38 ERROR: Failed to run pg_dump in the noobaa-db-pg-0 pod

ODF Version:

[[email protected] backup-folder]# oc get csv -n openshift-storage
NAME                                         DISPLAY                       VERSION               REPLACES   PHASE
mcg-operator.v4.12.0-152.stable              NooBaa Operator               4.12.0-152.stable                Succeeded
metallb-operator.4.12.0-202301042354         MetalLB Operator              4.12.0-202301042354              Succeeded
ocs-operator.v4.12.0-152.stable              OpenShift Container Storage   4.12.0-152.stable                Succeeded
odf-csi-addons-operator.v4.12.0-152.stable   CSI Addons                    4.12.0-152.stable                Succeeded
odf-operator.v4.12.0-152.stable              OpenShift Data Foundation     4.12.0-152.stable                Succeeded

[[email protected] backup-folder]# oc get subscription -n openshift-storage
NAME                                                                          PACKAGE                   SOURCE              CHANNEL
mcg-operator-stable-4.12-ocs-catalogsource-openshift-marketplace              mcg-operator              ocs-catalogsource   stable-4.12
ocs-operator-stable-4.12-ocs-catalogsource-openshift-marketplace              ocs-operator              ocs-catalogsource   stable-4.12
odf-csi-addons-operator-stable-4.12-ocs-catalogsource-openshift-marketplace   odf-csi-addons-operator   ocs-catalogsource   stable-4.12
odf-operator                                                                  odf-operator              ocs-catalogsource   stable-4.12

OCP Version:

[[email protected] backup-folder]# oc version
Client Version: 4.11.9
Kustomize Version: v4.5.4
Server Version: 4.12.0-rc.6
Kubernetes Version: v1.25.4+77bec7a

Images :

- name: ROOK_CEPH_IMAGE
          value: quay.io/rhceph-dev/odf4-rook-ceph-rhel8-operator@sha256:60f1ae2a2a28802fceca9a75252cec045755ca2fc0679f9693b185188561d86e
        - name: CEPH_IMAGE
          value: quay.io/rhceph-dev/rhceph@sha256:c6fe7e71ad1b13281d1d2399ceb98d3d6927df40e5d442a15fa0dee2976ccbcf
        - name: NOOBAA_CORE_IMAGE
          value: quay.io/rhceph-dev/odf4-mcg-core-rhel8@sha256:b495b59219d78ab468d1e1faedacfda59cb4b9fe13b253157897ff6899811de5
        - name: NOOBAA_DB_IMAGE
          value: quay.io/rhceph-dev/rhel8-postgresql-12@sha256:f4d8f5f165da493568802b4115f5e68af7cc11a3f14769e495de4a3f61a58238
        - name: PROVIDER_API_SERVER_IMAGE
          value: quay.io/rhceph-dev/odf4-ocs-rhel8-operator@sha256:c4e3463ccb0cf38f7feb71b1cfcd55de006e598d4b8fa3c9eb9175c8083fe0ce
        - name: OPERATOR_CONDITION_NAME
          value: ocs-operator.v4.12.0-152.stable
        image: quay.io/rhceph-dev/odf4-ocs-rhel8-operator@sha256:c4e3463ccb0cf38f7feb71b1cfcd55de006e598d4b8fa3c9eb9175c8083fe0ce
        imagePullPolicy: Always

[nimrod-becker]

Something differs in the way this deployment is done since we don't see it at all during 4.12 runs

[rkomandu]

@nimrod-becker , as the downstream is been used to test this from Quay.io (the procedure is similar to what was followed earlier). Any other suggestion for this ? Are the Postgres image same across 4.12 that was tried in your env and in our env ?

[nimrod-becker]

If it's the same build it's the same images.
In any case, a suggestion was already made 2 weeks ago, and it should solve the issue, its also not a big change
Liran's Comment

[rkomandu]

If it's the same build it's the same images. In any case, a suggestion was already made 2 weeks ago, and it should solve the issue, its also not a big change Liran's Comment

@nimrod-becker , we will change this in DAS code, however what you meant by Deployment is different ?
I am not sure what/how it was tried in your ODF env

[rkomandu]

If it's the same build it's the same images. In any case, a suggestion was already made 2 weeks ago, and it should solve the issue, its also not a big change Liran's Comment

@nimrod-becker , we will change this in DAS code, however what you meant by Deployment is different ?
I am not sure what/how it was tried in your ODF env

[rkomandu]

@nimrod-becker , update:
This was tried on the ODF 4.12 GA code level with the below postgres image

oc get csv -n openshift-storage -o yaml |grep -i full
full_version: 4.12.0-173
full_version: 4.12.0-173
full_version: 4.12.0-173

Image: registry.redhat.io/rhel8/postgresql-12@sha256:3d805540d777b09b4da6df99e7cddf9598d5ece4af9f6851721a9961df40f5a1

We need to change our scripts to ensure that back-up is created. So with Liran proposed change we will deal with it in our upcoming release.