Account update with account name is not updating bucket_name.json file automatically

[anandhu-karattu]

Account update with account name is not updating bucket_name.json file automatically

[shirady]

@romayalon I agree with you.
just for it to be clear the long-term solution would be to change the config structure and map the account and bucket IDs to the config file, and use only these IDs as a reference.
I didn't close the issue because the fix is partial.

[romayalon]

@anandhu-karattu
Can you share the I/O failure you see?

[anandhu-karattu]

I am trying to upload an object in to the bucket "bucket-old", it is failing

[root@anan-rhel921 ~]# AWS_ACCESS_KEY_ID=BjLLk92yXAEFxOKaOFyA AWS_SECRET_ACCESS_KEY=xYPDVxbTaH6g8b42T2ecOk/j3I53spnA/JkDtEi2 aws --endpoint https://10.11.71.87:6443 --no-verify-ssl s3 cp awscliv2.zip s3://bucket-old
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.11.71.87'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
upload failed: ./awscliv2.zip to s3://bucket-old/awscliv2.zip An error occurred (AccessDenied) when calling the CreateMultipartUpload operation: Access Denied

Issue is only with the existing buckets , since the owner name is not updated to new_name.

[shirady]

Hi @anandhu-karattu,
I added a fix for this issue, please pay attention that it is a short-term fix:

1. After this fix, the S3 request should not return an error.
2. But the buckets that the account owns still have the properties system_owner and bucket_owner with the previous name (in the bucket config) <- this would be handled in a long-term fix

I'm adding the label "Request Validation" so you can test it (and not close it for now).

cc: @romayalon

[anandhu-karattu]

1. After this fix, the S3 request should not return an error.

What does this mean? @shirady

[shirady]

@anandhu-karattu I meant for the error Access Denied.

[romayalon]

@anandhu-karattu @shirady Shira fixed the IO failure but not the actual update of the bucket config.json, a fix for the update will be taken care in #7734, Shira please keep me honest here.

[anandhu-karattu]

Verified on build "noobaa-core-5.15.4-20240605.el9.x86_64"
Looks like the IO FAILURE is still there.

How to reproduce:

1. Create an account > Bucket > Upload object in to bucket ==> PASS
2. Change the account name to new name ==> PASS

[root@anan-21 ~]# noobaa-cli account update --name account-65675 --new_name account_test_new
{
  "response": {
    "code": "AccountUpdated",
    "reply": {
      "_id": "6662ad6dc999b40f7e95e9bf",
      "name": "account_test_new",
      "email": "account_test_new",
      "creation_date": "2024-06-07T06:49:17.510Z",
      "access_keys": [
        {
          "access_key": "5ZQ5YaCYDOGfZhDC07sM",
          "secret_key": "JpEojbabc57kMfNk/rGBPUW0bW/jTHWXwqhLfIJs"
        }
      ],
      "nsfs_account_config": {
        "uid": 3844,
        "gid": 4890,
        "new_buckets_path": "/mnt/gpfs0/account_65675/",
        "fs_backend": "GPFS"
      },
      "allow_bucket_creation": true,
      "master_key_id": "6662ad6dadbc37d3ecb52d68"
    }
  }
}

3. Upload the object again to the same bucket ==> FAILED with access denied error

[root@akarattuparambil-scale-host ~]# AWS_ACCESS_KEY_ID=5ZQ5YaCYDOGfZhDC07sM AWS_SECRET_ACCESS_KEY=JpEojbabc57kMfNk/rGBPUW0bW/jTHWXwqhLfIJs aws --endpoint https://10.0.100.21:6443 --no-verify-ssl s3 cp awscliv2.zip s3://bucket-65675
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.0.100.21'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.0.100.21'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.0.100.21'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.0.100.21'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.0.100.21'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.0.100.21'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.0.100.21'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.0.100.21'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.0.100.21'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.0.100.21'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.0.100.21'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.0.100.21'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.0.100.21'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.0.100.21'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.0.100.21'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.0.100.21'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.0.100.21'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.0.100.21'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.0.100.21'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.0.100.21'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
upload failed: ./awscliv2.zip to s3://bucket-65675/awscliv2.zip An error occurred (AccessDenied) when calling the UploadPart operation: Access Denied

[anandhu-karattu]

@shirady looks like the fix is failing for me. Can you please check the issue again?

[anandhu-karattu]

This is the corresponding error found in noobaa.log

Jun  7 14:45:36 anan-21 [230684]: [nsfs/230684]    [L0] core.sdk.bucketspace_fs:: BucketSpaceFS.read_bucket_sdk_info: bucket_config_path /mnt/cesSharedRoot/ces/s3-config/buckets/bucket-65675.json
Jun  7 14:45:36 anan-21 [230684]: [nsfs/230684]    [L0] core.sdk.bucketspace_fs:: BucketSpaceFS.read_bucket_sdk_info: bucket_config_path /mnt/cesSharedRoot/ces/s3-config/buckets/bucket-65675.json
Jun  7 14:45:36 anan-21 [230684]: [nsfs/230684] [ERROR] core.endpoint.s3.s3_rest:: S3 ERROR <?xml version="1.0" encoding="UTF-8"?><Error><Code>AccessDenied</Code><Message>Access Denied</Message><Resource>/bucket-65675/awscliv2.zip?uploads</Resource><RequestId>lx4oj92t-f3f5kt-oxw</RequestId></Error> POST /bucket-65675/awscliv2.zip?uploads {"host":"10.0.100.21:6443","accept-encoding":"identity","content-type":"application/zip","user-agent":"aws-cli/2.15.30 Python/3.11.8 Linux/4.18.0-477.10.1.el8_8.x86_64 exe/x86_64.rhel.8 prompt/off command/s3.cp","x-amz-date":"20240607T124536Z","x-amz-content-sha256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","authorization":"AWS4-HMAC-SHA256 Credential=5ZQ5YaCYDOGfZhDC07sM/20240607/us-east-1/s3/aws4_request, SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date, Signature=5476b626e3c290390c773896cf9c2276ba88c9eead4bccfcc293d4f8fc8b4b9b","content-length":"0"} Error: Access Denied    at authorize_request_policy (/usr/local/noobaa-core/src/endpoint/s3/s3_rest.js:243:15)    at async Promise.all (index 1)    at async authorize_request (/usr/local/noobaa-core/src/endpoint/s3/s3_rest.js:205:5)    at async handle_request (/usr/local/noobaa-core/src/endpoint/s3/s3_rest.js:115:5)    at async Object.s3_rest [as handler] (/usr/local/noobaa-core/src/endpoint/s3/s3_rest.js:65:9)

[romayalon]

@anandhu-karattu we didn't backport it to stage_5.15.4...
@madhuthorat please evaluate this priority and let us know if this should be backported..
cc: @nimrod-becker

[anandhu-karattu]

@romayalon we are not supporting account name update in scale release 5.2.1.
So this fix is not mandatory for us in 5.15.4 @madhuthorat
But we may support this in future releases, so we will expect a future fix (complete fix as per your plan)

[madhuthorat]

@romayalon we are not supporting account name update in scale release 5.2.1. So this fix is not mandatory for us in 5.15.4 @madhuthorat But we may support this in future releases, so we will expect a future fix (complete fix as per your plan)

Right, we won't support accountName update in CES S3 MVP GA, so please take a call if you want to add to 5.15.4 or not.

[romayalon]

@madhuthorat @anandhu-karattu We decided not to backport it to 5.15.4,
@anandhu-karattu could you validate it on master?

[anandhu-karattu]

Verified on build "noobaa-core-5.17.0-20240617.el9.x86_64"
There is no access denied error this time. (verification steps are mentioned above)
Marking as verified. @romayalon @shirady

[shirady]

Hi,
The short-term solution was verified (see comment above):

1. Removing the "request validation" label as it was validated.
2. I will remove myself from the assignee since another developer might be assigned for the fix of the long-term solution

cc: @romayalon