Behaviour of `ls`

[fsspec]

Behaviour of `ls`

[mrocklin]

I think so? I don't know that I have enough experience with S3 to be sure. What are your thoughts?

[martindurant]

I will think about this for the evening. I did a little trial and error and am not really happy with the various combinations.
There are additional complexities, such as 'bucket/dir' and 'bucket/dir/' both being valid keys with or without data, with or without content at 'bucket/dir/stuff'.

[martindurant]

Let me step back and describe a simpler ls that more closely reflects the workings of S3 (which has no real directories, only paths that happen to contain/). What if we go back to a flat structure, allowing arbitrary filenames?

ls('') -> list of buckets
ls('mybucket') -> all keys in bucket, or notfound if bucket inaccessible
ls('mybucket/myfile') -> one exact match or notfound
info('mybucket/myfile') == ls('mybucket/myfile', detail=True)

mkdir('bucket') -> creates bucket
mkdir('bucket/path') -> notimplemented, or no-op (choice)

walk("") -> all keys in all buckets accessed so far
walk(path) -> all keys that startwith path

glob(pattern) -> normal match, the only one here where '/' has significance within buckets

In short, distributed.s3 would use walk or glob, not ls in this scheme.

[mrocklin]

This seems like a straightforward approach that is easy to understand, implement, and convey to users.

I can see wanting to change it at some point in the future to impose a more traditional directory structure. But I can also imagine not wanting to change it, depending on how people use S3. Regardless there are probably higher priority things.