NSFS | unable to do aws s3 cp with obj larger than 1T

[javieramirez1]

NSFS | unable to do aws s3 cp with obj larger than 1T

[javieramirez1]

It's in progress, it's not finished yet.

Upload Speed: 0.0 MB/sec (average 330.0)
Upload Speed: 476.7 MB/sec (average 330.4)
Upload Speed: 0.0 MB/sec (average 330.0)
Upload Speed: 487.4 MB/sec (average 330.4)
Upload Speed: 0.0 MB/sec (average 330.0)
Upload Speed: 489.5 MB/sec (average 330.4)
Upload Speed: 0.0 MB/sec (average 330.0)
Upload Speed: 543.5 MB/sec (average 330.4)
Upload Speed: 0.0 MB/sec (average 330.1)
Upload Speed: 517.4 MB/sec (average 330.5)
Upload Speed: 0.0 MB/sec (average 330.1)
Upload Speed: 487.9 MB/sec (average 330.5)
Upload Speed: 0.0 MB/sec (average 330.1)
Upload Speed: 483.2 MB/sec (average 330.5)
Upload Speed: 0.0 MB/sec (average 330.1)
Upload Speed: 494.7 MB/sec (average 330.5)
Upload Speed: 0.0 MB/sec (average 330.1)
Upload Speed: 488.1 MB/sec (average 330.5)
Upload Speed: 0.0 MB/sec (average 330.1)

[guymguym]

(BTW, unrelated to this large size issue, when I use concurrency --upload --size $((100*1024)) --part_size 32 --concur 16 I managed to reproduce the "cancelled due to ctime change" issue - #7833, so for now please keep testing with large part size and concur of 1, and I will reopen the other issue, fyi @romayalon)

[javieramirez1]

[root@c83f2-dan9 bucket1]# ls -lh /gpfs/remote_rw_fs0/buckets_1/bucket1/.noobaa-nsfs_/multipart-uploads//parts-size-*
-rw-rw----. 1 bin bin 830G Jun 17 15:33 /gpfs/remote_rw_fs0/buckets_1/bucket1/.noobaa-nsfs_666730c4482e5fed3d54547d/multipart-uploads/87e8575e-b6c0-4611-bcec-d88aff50d1f2/parts-size-1073741824

[guymguym]

@javieramirez1 this seems like a client issue - I decoded the raw packet that the server received and it's not a valid url -

> node -p 'Buffer.from("47 45 54 20 2e 2e 2f 2e 2e 2f 2e 2e 2f 2e 2e 2f 2e 2e 2f 2e 2e 2f 2e 2e 2f 2e 2e 2f 2e 2e 2f 2e 2e 2f 2e 2e 2f 2e 2e 2f 77 69 6e 64 6f 77 73 2f 77 69".split(" ").map(i => Number("0x"+i))).toString()'

GET ../../../../../../../../../../../../windows/wi

[javieramirez1]

@guymguym But what is the difference with small objects, because there are no issues?

[guymguym]

@javieramirez1 the difference is the size :)

my point is that I think it might be a problem of the client and not the server, but lets check that to make sure.

run the following client tool we have - it uploads random generated data without saving it to disk first so there is no source file needed, notice that the sizes are in MB so this example will upload 1TB in 1024 parts of 1024MB each (the total number of parts cannot exceed 10,000).

> cd /usr/local/noobaa-core
> bin/node src/tools/s3cat --endpoint http://localhost:6001 --access_key XXX --secret_key YYY --bucket BBB --upload --size $((1024*1024)) --part_size 1024 --concur 1

this tool shows only the speed of writing, but not real progress, so you can monitor the progress by looking at the buckets internal dir like so:

> ls -lh /gpfs/path/to/bucket/.noobaa-nsfs_*/multipart-uploads/*/parts-size-*

[javieramirez1]

@guymguym in my clients i dont have the nooba-core dir Do I copy it from my dan nodes?

[guymguym]

no, lets start simple, for now just run it from inside the dan node

[javieramirez1]

[root@c83f2-dan9 noobaa-core]#  bin/node src/tools/s3cat --endpoint http://172.20.100.60:6001 --access_key Nk6lx6msTmKXUM04Chu9 --secret_key CzD+xVImh3WocHG1Ag670S+WG0+lqZ3GZMrQ1BMD --bucket bucket1 --upload --size $((1024*1024)) --part_size 1024 --concur 1
Uploading upload-lxjbbjpb from generated data of size 1.0 TB
UPLOAD ERROR: UnknownEndpoint: Inaccessible host: `172.20.100.60' at port `6001'. This service may not be available in the `us-east-1' region.
    at Request.ENOTFOUND_ERROR (/usr/local/noobaa-core/node_modules/aws-sdk/lib/event_listeners.js:611:46)
    at Request.callListeners (/usr/local/noobaa-core/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/usr/local/noobaa-core/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/usr/local/noobaa-core/node_modules/aws-sdk/lib/request.js:686:14)
    at error (/usr/local/noobaa-core/node_modules/aws-sdk/lib/event_listeners.js:443:22)
    at ClientRequest.<anonymous> (/usr/local/noobaa-core/node_modules/aws-sdk/lib/http/node.js:100:9)
    at ClientRequest.emit (node:events:518:28)
    at ClientRequest.emit (node:domain:488:12)
    at Socket.socketErrorListener (node:_http_client:495:9)
    at Socket.emit (node:events:518:28) {
  code: 'UnknownEndpoint',
  region: 'us-east-1',
  hostname: '172.20.100.60',
  retryable: true,
  originalError: Error: connect ECONNREFUSED 172.20.100.60:6001
      at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1595:16) {
    errno: -111,
    code: 'NetworkingError',
    syscall: 'connect',
    address: '172.20.100.60',
    port: 6001,
    region: 'us-east-1',
    hostname: '172.20.100.60',
    retryable: true,
    time: 2024-06-17T18:32:16.380Z
  },
  time: 2024-06-17T18:32:16.380Z

[guymguym]

try --endpoint http://localhost:6001

[guymguym]

and check that your config.json has "ALLOW_HTTP": true, (needs systemctl restart noobaa after changing)

[guymguym]

@javieramirez1 I managed to upload 1TB object with s3cat tool - does it upload for you now?

> bin/node src/tools/s3cat ...
...
...
upload done. 215 MB/sec

> ls -lh /gpfs/fs1/bucket1/
-rw-rw----. 1 root root 1.0T Jun 17 15:12 upload-lxj9v2bv

[guymguym]

this tool shows only the speed of writing, but not real progress, so you can monitor the progress by looking at the buckets internal dir like so:

> ls -lh /gpfs/path/to/bucket/.noobaa-nsfs_*/multipart-uploads/*/parts-size-*

[javieramirez1]

Upload Speed: 1023.0 MB/sec (average 330.7)
Upload Speed: 0.0 MB/sec (average 330.6)
upload done. 331 MB/sec

[javieramirez1]

Ok then I would use s3cat now from my app nodes, or what would I do?

[guymguym]

Hey @javieramirez1

TLDR -- yes, s3cat is a convenient test tool and we can extend it to do more as needed, it uses the standard aws sdk for js, and you can install it with the noobaa-core 5.15.4 rpm on the app nodes too.

Few points:

1. As you increase the object size you also have to increase the part_size because the maximum number of parts is not allowed to exceed 10,000, which means that part_size of 1 GB can only get you up to object size of 10000 GB = 9.76 TB (which is a bit under 10 TB).
2. We can open an issue to aws-cli about this problem. It's probably coming from its deps botocore or s3transfer. It might get rejected if they do not plan to support it but still worth reporting imho.
3. We should verify extra-large uploads also with other s3 clients that can do above 1TB because it's not healthy to verify a server only with your own test client.
4. We can try aws s3api commands in a script which should probably work and be easy to use for testing - see create-mpu, upload-part, complete-mpu.
5. Two other popular cli clients to try are rclone and mc.
6. I wonder which real world S3 apps use extra-large objects - for example is presto parquet using hive connector can use such object sizes, see this question.

[guymguym]

Closing this issue for now, since it seems to be a client side issue only. Thanks @javieramirez1 !