After account update with new access keys, old access keys are valid for next 5min ~

[anandhu-karattu]

After account update with new access keys, old access keys are valid for next 5min ~

[romayalon]

Adding some internal info -
When calling noobaa-cli account update the account's config file is being updated.
NooBaa service maintains an account cache with a default expiry_ms = 10 minutes, after the expiry the old account won't be valid anymore.
NooBaa CLI and NooBaa service are different processes, we need to come up with a way to communicate to the service that the account was updated so we will get an updated view of the account's data.

[rkomandu]

@romayalon , are we planning to fix this in 5.15.3/.4

[anandhu-karattu]

I have a use case:

1. Update newbuckets path in the account. old_bucketspath to new_bucketspath
2. Create a new bucket via aws mb or s3api bucket-create, this is still creating new buckets under old_bucketspath
3. After 5-10 mins and account cache expiry, new buckets are getting created under new_bucketspath as expected.

So the issue I am facing here is , I need to wait 5-10 mins to test my scenarios or even in test automation.
Do we have a way to trigger account cache expiry manually? So that as a work around I can trigger it and proceed with my next steps?
@romayalon @guymguym

[romayalon]

@anandhu-karattu We do have process.env.ACCOUNTS_CACHE_EXPIRY which you can set on tests to be 1 ms, Can you try it? (add it to /etc/sysconfig/noobaa env file)
we can consider exporting it as a config as a W/A for customers who will complain the same issue, like we have config.OBJECT_SDK_BUCKET_CACHE_EXPIRY_MS...
Adding a point regarding IAM, IAM server will be on the same process as the service so we can easily invalidate accounts cache when updating accounts, this case can be a motivation for exploiting the tags for updating accounts config...
@guymguym @nimrod-becker @shirady
WDYT?

[romayalon]

Updating that we had an internal discussion, we wouldn't recommend exporting it to customers as it could create very wasteful config files reads.
@anandhu-karattu as we discsussed on slack, you can use it on tests if you wish as we do.

[anandhu-karattu]

@romayalon I verified the method you suggested . Please find my observations:

1. Create a new file with config : echo ACCOUNTS_CACHE_EXPIRY=1 > /etc/sysconfig/noobaa
2. Restart the S3 service .

Observations:

1. On a 2 node cluster, I updated this on node1. This parameter is effective after restart of services and right after account update (for --accesskeys as well as --newbucketspath) I could find the new value is effective.
2. After the test , I deleted the file “etc/sysconfig/noobaa” and restarted S3 service. Now the account expiry is back to normal 10 mins.

However this requires a restart of S3 service and we don't want to keep this config file option for all the tests.
I suspect, I might miss some issues with this config.

Instead do we have any way to call the server and refresh the cache individually ? If yes, We can even include that call in our account update cli (mms3) and refresh the cache after each update operation? Will that be possible?

[romayalon]

Hey @anandhu-karattu
If there was an easy option for invalidating the cache from outside of NooBaa endpoint process we would do it from our CLI...
As I wrote before, currently noobaa endpoint process supports only S3 operations (invalidate cache is a NooBaa internal function and not S3 request, so it's not supported) but in the future (currently WIP) the endpoint
process will receive IAM requests for updating access keys (and invalidating the cache), and we need to consider supporting different configurations like new_buckets_path..
CC: @guymguym @shirady @nimrod-becker

@anandhu-karattu We do have process.env.ACCOUNTS_CACHE_EXPIRY which you can set on tests to be 1 ms, Can you try it? (add it to /etc/sysconfig/noobaa env file) we can consider exporting it as a config as a W/A for customers who will complain the same issue, like we have config.OBJECT_SDK_BUCKET_CACHE_EXPIRY_MS... Adding a point regarding IAM, IAM server will be on the same process as the service so we can easily invalidate accounts cache when updating accounts, this case can be a motivation for exploiting the tags for updating accounts config... @guymguym @nimrod-becker @shirady WDYT?

[romayalon]

@anandhu-karattu Could you set echo ACCOUNTS_CACHE_EXPIRY=1 > /etc/sysconfig/noobaa at the beginning of the tests (before enabling noobaa service) like we do instead of setting it for a specific test and restart the service?

[anandhu-karattu]

@romayalon I can set the parameter as we discussed and manage my tests (if there is no other way).
@madhuthorat how we can handle this use case ? Can we advise the customers to wait for 10 mins to create a new bucket (via s3 mb or s3api) after account update with --newbucketspath ?