same size and same file name with different content will be ignored

[aws-samples]

same size and same file name with different content will be ignored

[huangzbaws]

This is a trade-off for performance, known issue. It only check file name and size as duplicated file.
Might improve this with Verification Option in coming release.

[liangruibupt]

Can we add ETag as 3rd parameter to determine the duplicated file? It can be gotten from response of list_objects_v2 and head_object

[huangzbaws]

First. If we don't know how you calculated the Etag while upload, it cannot be verified while download. But the files can be uploaded with many different tool, with different Etag algorithm. Except you force to use the same tool to upload to the first source S3.

Second. And if we use the auto-generated Etag by S3, then:
When you upload a file, the file will be split into multipart and merge on S3. For example upload 14MB file with aws cli, it may be split into 5MB+5MB+4MB. The Etag on S3 is calculated as part1 MD5 + part2 MD5 + part3 MD5, and MD5 again, then add "-" with parts number. The S3 auto-generated Etag will be "1111111111111111-3".
The other tool may upload the file with 8MB+6MB. The Etag will be "222222222222222-2"
So if you upload with different split way, Etag will be different.

Funny thing is, I actually use this way to verify MD5 while uploading to destination S3 with S3 auto-generated Etag.

[huangzbaws]

Mark a real example Etag generated by S3 of a file 19.9MB.
Split part size: 5MB, Etag: e4289d4b51256e46b15e58de0b8961e3-4
Split part size: 6MB, Etag: 1c9f78d2cca8c1772e97535282d67423-4

[liangruibupt]

Yes, I see the code "ifVerifyMD5" part, understand the limitation of adding the ETag. Thanks for clarify.