s3: ls doesn't read all the files

[rclone]

s3: ls doesn't read all the files

[ncw]

Hmm, looking at the code I think you are right. I'm only listing the first 10,000 files by the look of it.

Do you have 17,000 files by any chance?

I'm currently rewriting the S3 code to use a different library, and I'll try to fix this as part of that re-write.

[ChrisSLT]

I don't think I had that many, but I'm not sure. Thanks!
On Jan 30, 2015 4:22 AM, "Nick Craig-Wood" [email protected] wrote:

Hmm, looking at the code I think you are right. I'm only listing the first
10,000 files by the look of it.

Do you have 17,000 files by any chance?

I'm currently rewriting the S3 code to use a different library, and I'll
try to fix this as part of that re-write.

—
Reply to this email directly or view it on GitHub
#22 (comment).

[ncw]

I fixed this with the old library. Please re-open if it doesn't work!

Thanks

Nick

[ChrisSLT]

Hi Nick - I'll give it a shot over the next few days

FYI - I noticed a bunch more LIST BUCKET requests in my s3 account than I
expected over the past few days. I'm trying to figure out if this is a
problem w/ the AWS CLI tool, or do you think it could have been caused by
the bug in the way rclone called "LIST BUCKET?"

On Thu, Feb 12, 2015 at 1:03 PM, Nick Craig-Wood [email protected]
wrote:

I fixed this with the old library. Please re-open if it doesn't work!

Thanks

Nick

—
Reply to this email directly or view it on GitHub
#22 (comment).

347.708.6549

[ncw]

Rclone now lists in chunks of 1024 whereas it used to use 10,000 so you'll definitely get more list bucket requests. Using a smaller number than 10,000 means rclone can get on with transferring stuff once the first 1024 have come back, while it is fetching the next chunk.

If this is a problem I could add a flag to control the chunk size.

[ChrisSLT]

For what it's worth, the Amazon people told me that the api only ever
supports listing 1000 keys at a time no matter what method is used.

I only bring it up because I have a weirdly high amount of list requests
now so it could point to something.
On Feb 13, 2015 9:33 AM, "Nick Craig-Wood" [email protected] wrote:

Rclone now lists in chunks of 1024 whereas it used to use 10,000 so you'll
definitely get more list bucket requests. Using a smaller number than
10,000 means rclone can get on with transferring stuff once the first 1024
have come back, while it is fetching the next chunk.

—
Reply to this email directly or view it on GitHub
#22 (comment).

[ncw]

I've just done a bit of debugging with rclone and it definitely only does 1 request per 1024 directory entries.

I can only see itemised $0.005 per 1,000 PUT, COPY, POST, or LIST requests in my console, so could these be PUT requests? rclone will do PUT requests both when uploading a file, and when updating its metadata (eg timestamp).

[ChrisSLT]

Thanks Nick - can you tell me when rclone updates metadata? Would it do so
on thousands of files?

On Fri, Feb 13, 2015 at 12:40 PM, Nick Craig-Wood [email protected]
wrote:

I've just done a bit of debugging with rclone and it definitely only does
1 request per 1024 directory entries.

I can only see itemised $0.005 per 1,000 PUT, COPY, POST, or LIST requests
in my console, so could these be PUT requests? rclone will do PUT
requests both when uploading a file, and when updating its metadata (eg
timestamp).

—
Reply to this email directly or view it on GitHub
#22 (comment).

347.708.6549

[ncw]

If the timestamp of the file changed - that is the only reason with s3 to update the metadata.