NSFS Presto uploads large file as multiparts, calls complete with short timeout, causing retries and race on the mpu dir

[guymguym]

NSFS Presto uploads large file as multiparts, calls complete with short timeout, causing retries and race on the mpu dir

[guymguym]

Hi @rkomandu
Right, cosbench is using put object, not multi part upload (intel-cloud/cosbench#138).

awscli does use multipart above an object size threshold that can be configured.
The size of each part in a multipart upload is 8M by default but can also be configured - see multipart_chunksize.

I know that some clients will be easily configured to overcome this issue in the meanwhile, such as awscli that exposes many options to the user, but other applications that use s3 as a client could be less accessible to configure these settings and therefore will have to use multipart, which could hit this issue during the final complete-multipart-upload request. In that case, we can try to configure the timeout to allow the server to complete the parts copy on time.

Does this clarifies a bit more?

[cxhong]

What's the timeline for this fixes?

This problem is critical for the client B4RESTORE .
We need a timeline when the fix can be expected that I can communicate to the client.

[guymguym]

@cxhong I don't know, it requires initial effort to implement the suggestion, but then also the question is if there is a path to backport this code, because this will not be a small fix. FYI @nimrod-becker.

For an immediate workaround, we better explore disabling multiparts in the client configuration (for example see aws cli multipart_threshold config option that can be used to increase the size threshold), or alternatively try to keep the multipart but increase the timeout (for example in aws cli there is the --cli-read-timeout flag).

[cxhong]

Thanks @guymguym , let me update to customer

[rkomandu]

@guymguym the current chunks is 8M correct or this is configured differently ? also it is less if we have 8G file to have too much time and deleting this in the .nsfs directory of the directory?

I am trying to run 20GB file via cosbench hence asking (Fyre cluster though), what is the definition of large file here is 8GB ?

[rkomandu]

Every client application is performing this thing of upload and S3 cli did it with 8M chunks, whereas cosbench or any other application will do on its own the chunk size as it looks like.

-- s3 client (showed to @romayalon in the interlock call)
when a 2M file is uploaded then there is no multi-part upload directory in the .noobaa-nsfs
when a 20G file is uploaded then there is multi-part upload directory in the .noobaa-nfs

-- Running via cosbench the directory created is upload in the .noobaa-nsfs and not multi-part upload for obj sizes of 2GB or 4GB or 8GB. This indicates that the upload depends on the client.

Is my understanding correct w/r/t application level chunks (not shown) and upload directory is created and not a multi-part ?

[cxhong]

@guymguym feedback from customer:

We have tested uploading without multipart several times and that seems to fix the issue.. but as we rely on applications to handle the data streams towards our S3 API we can not always disable it as it is an inbuilt feature of the products.
AWS cli we only use for manual labor which is less than 1% of the work...
if you need me to run these test for internal value i will happily do it but it does not solve anything for us to use these parameters as a workaround seen as we do not use AWS cli.
let me know if its meant as a workaround or for further information for development.

they are asking if there is any workaround that can be implemented or when will a fix be available? @nimrod-becker

[cxhong]

@guymguym and @nimrod-becker , any timeline? customer is waiting and like to know when this fixes will be available.