File functions - maybe use s3fs?

[sckott]

File functions - maybe use s3fs?

[seankross]

Oh wow this is cool! Interesting that for many of the functions the main argument is path which is a URI. I wonder if that's am abstraction we should be using?

[sckott]

I wonder if that's an abstraction we should be using?

maybe? It can be verbose if there's lots of nesting in folders within a single path. I imagine URIs as a principle may not be familiar to users of this pkg? Maybe I'm wrong though

I do like the s3_path idea where users could construct paths rather than having to make each one, e.g.,

s3_path("foo", "bar", letters[1:3], ext = "txt")
#> [1] "s3://foo/bar/a.txt" "s3://foo/bar/b.txt" "s3://foo/bar/c.txt"

[sckott]

I just noticed that the abstraction of using URIs may be a good idea to borrow here for at least the reason that it makes it easy then to differentiate between local files and those on s3, e.g.,

# copy local to s3 remote
s3_file_copy("LICENSE", new_path = "s3://s64-test-2/license.txt")
[1] "s3://s64-test-2/license.txt"

# copy s3 remote to local 
s3_file_copy("s3://s64-test-2/license.txt", "stuff.txt")
[1] "stuff.txt"

that is, if a file string doesn't have a s3:// in front of it, then it's i think assumed to be local.

[sckott]

Ran into an issue with s3fs::s3_dir_exists - it isn't working to detect a dir that exists, or that doesn't exist correctly. That definitely gives me pause about using the package. Will keep exploring though

[seankross]

This might be overkill but I wonder if we should have function called something like s3_path() that would act like fs::path() that would help folks construct paths correctly, by appending s3://, by indicating the file is on S3, etc. Stinks to hear that s3fs might have some idiosyncrasies :(

[sckott]

there is! see above

[seankross]

Ha wow need to work on my reading skills.

[sckott]

we could probably just re-export it in this pkg, kinda like alot of the tidyverse pkgs do

[seankross]

I think that would be cool to do, so that end users have one interface.

[sckott]

using it now, closing