Fail to download full folder with 15K images and 7K text files

[awslabs]

Fail to download full folder with 15K images and 7K text files

[dbickson]

filelist.zip

Those are the full files as downloaded by aws s3 sync vs s3 c demo (sorted). As can be seen there are many missing files without any specific pattern.

[waahm7]

@dbickson Thank you for creating the issue with details. We will take a look and will let you know as soon as we have any updates.

[TingDaoK]

We have opened the bucket permissions so you can try on your own.

I still got an AccessDenied from s3://vl-sample-dataset-kitti/Kitti/

[TingDaoK]

I tried to create a bucket with the same file directory, but fake files with a single char in it.

I cannot reproduce the error you seen as well. I successfully downloaded all 22480 files with ./s3 cp -v ERROR -r us-west-2 s3://test-bucket-asd/raw/ ./test &>cp-out

I saw something in the output that's suspicious

�[1431B22.20user 17.14system 0:59.49elapsed 66%CPU (0avgtext+0avgdata 182988maxresident)k
16096inputs+12727272outputs (64major+161888minor)pagefaults 0swaps

And after that, you did another download of the same file? I am not sure why?

[dbickson]

Hi @TingDaoK can we do a short zoom session I can share my screen it takes less than a minute to reproduce the issue on my side.

according to our management console the bucket is public

Can the issue originate from the fact I am using t2.xlarge instance?

[TingDaoK]

Not sure. And from your nohup.out, we can actually see all 22480 files are downloaded. So, I guess something went wrong writing to the disk?

[dbickson]

HI @TingDaoK I have given additional bucket permissions, can you try accessing again.

[dbickson]

I agree in the nohup there are exactly 22,480 download unique filename printouts, which is correct. But not all of them are saved to disk.
grep download nohup.out | sort -u > 1
cut -f 2 1 -d ' '> 2
sort -u 2 | wc
22480 22480 1498642
ubuntu@ip-172-31-30-217:/mnt/data/wakar$ find ~/Kitti -type f | wc
11287 11287 571716

[dbickson]

HI @TingDaoK I am a little closer to breakthrough, I have added the following trace:
if (!transfer_ctx->output_sink) {
printf("Failed to download to %s\n", (char*)file_path);
return AWS_OP_ERR;
}

I see a lot of failures here which may explain the missing files. Please advise?

[TingDaoK]

Can you do -v ERROR, and attach the log to us?

[TingDaoK]

HI @TingDaoK I have given additional bucket permissions, can you try accessing again.

Cool. Now, I have access, I'll try to reproduce it with your bucket.

[dbickson]

Some more hints:

aws-error:45(AWS_ERROR_MAX_FDS_EXCEEDED)
[ERROR] [2023-07-07T19:56:51Z] [00007f0146ffd700] [common-io] - static: Failed to open file. path:'/home/ubuntu/Kitti/raw/testing/image_2/005993.png' mode:'wb' errno:24 aws-error:45(AWS_ERROR_MAX_FDS_EXCEEDED)
[ERROR] [2023-07-07T19:56:51Z] [00007f0146ffd700] [common-io] - static: Failed to open file. path:'/home/ubuntu/Kitti/raw/testing/image_2/005994.png' mode:'wb' errno:24 aws-error:45(AWS_ERROR_MAX_FDS_EXCEEDED)
[ERROR] [2023-07-07T19:56:51Z] [00007f0146ffd700] [common-io] - static: Failed to open file. path:'/home/ubuntu/Kitti/raw/testing/image_2/005995.png' mode:'wb' errno:24 aws-error:45(AWS_ERROR_MAX_FDS_EXCEEDED)
[ERROR] [2023-07-07T19:56:51Z] [00007f0146ffd700] [common-io] - static: Failed to open file. path:'/home/ubuntu/Kitti/raw/testing/image_2/005996.png' mode:'wb' errno:24 aws-error:45(AWS_ERROR_MAX_FDS_EXCEEDED)
[ERROR] [2023-07-07T19:56:51Z] [00007f0146ffd700] [common-io] - static: Failed to open file. path:'/home/ubuntu/Kitti/raw/testing/image_2/005997.png' mode:'wb' errno:24 aws-error:45(AWS_ERROR_MAX_FDS_EXCEEDED)
[ERROR] [2023-07-07T19:56:51Z] [00007f0146ffd700] [common-io] - static: Failed to open file. path:'/home/ubuntu/Kitti/raw/testing/image_2/005998.png' mode:'wb' errno:24 aws-error:45(AWS_ERROR_MAX_FDS_EXCEEDED)
[ERROR] [2023-07-07T19:56:51Z] [00007f0146ffd700] [common-io] - static: Failed to open file. path:'/home/ubuntu/Kitti/raw/testing/image_2/005999.png' mode:'wb' errno:24 aws-error:45(AWS_ERROR_MAX_FDS_EXCEEDED)

[dbickson]

hi @TingDaoK what is the correct way to increase the open file limit? I tried with ulimit but it did not work here.

[TingDaoK]

aha, so it's basically too many open files that hit the limits of file descriptor from system.

Easiest way maybe https://stackoverflow.com/questions/11342167/how-to-increase-ulimit-on-amazon-ec2-instance

[dbickson]

Issue is solved and now all files are received.
My only suggestion is to make -v ERROR the default since I was expecting some feedback regarding the problem and the run seems to finish fine.
Thanks again for your great support!

[TingDaoK]

I have an ongoing PR #330 to improve the error handling for samples. Should actually error out in your case with the change.

[TingDaoK]

We have updated the error handling here #332. It should properly error out now