Target archetype that stores to S3

[ropensci]

Target archetype that stores to S3

[mdneuzerling]

I've had some time to think about this and put some code down. Interacting with S3 requires (at least) three pieces of information:

• Object key, analogous to a file path
• Bucket name
• Configuration, including credentials

The first two are easy. The configuration (possibly inspired by boto's Session.client) is more involved. Refer to . It optionally specifies the following configuration fields, some of which are considered secrets:

• access_key_id
• secret_access_key
• session_token
• profile ()
• endpoint
• region

On my machine and with my straightforward AWS configuration, I can get away with using the default paws::s3() configuration. This is probably because I've run aws configure in a terminal. So we do have a sensible default value.

So I'm expecting two target archetypes:

tar_s3_get_object(key, bucket, config = paws::s3(), <other_targets_args>, ...)
tar_s3_push_object(key, bucket, config = paws::s3(), <other_targets_args>, ...)

And then using either the etag/hash or the last modified date to avoid re-downloading. These checks are actually performed server-side, so there's no need for targets to compare hashes.

The intent here is that ... is passed to PAWS so that users can deal with things like server-side encryption without it weighing down the targets syntax. I'll get some draft code together but I fully expect this to take a few rounds of iteration.

[wlandau]

Thanks for starting on this!

I agree that most of the config should happen up front and not encumber the target's interface.

After looking at your proposal, I see a couple alternative potential routes. Not sure which one I like more yet.

1. Just get and push existing files. This is how I read your proposal. Maybe I am missing an implicit argument for the target's command.
2. For the "push" archetype, act more like tar_target(format = "file"), which contains some customizable R code to run that returns a file path.

tar_s3_push_object(name, command, key, bucket, config = paws::s3(), ...)

[mdneuzerling]

Concerns with my approach:

1. How do CRAN feel about using the system command like this?
2. Does the AWS CLI approach work on Windows?
3. How do we handle the authentication side of things when uploading data to S3? (Things like server-side encryption). Downloading is generally simpler than uploading with S3.

These issues may be somewhat resolved if we wrap a fully-featured AWS API like the PAWS package. This has the added bonus of opening up the possibility of other integrations with AWS besides S3.

I'm a bit busy at the moment, despite being in strict lockdown, but I can try to have a look if you'd like?

[wlandau]

Yeah, I was hoping to use paws. If you have time, I would appreciate help and input. I plan to learn paws for ropensci/targets#152, but you have a huge head start.

[wlandau]

Probably best to build this on top of tar_change_raw() like in #9. Another thought is to use #9 somehow like Miles mentioned here.

[mdneuzerling]

I think I may still be in a drake state of mind, trying to replicate file_in and file_out. I'll try to get a better understanding of the targets equivalent concepts.

[wlandau]

Yeah, with targets, all files are dynamic (e.g. tar_target(format = "file")).

On reflection, I would actually prefer ropensci/targets#154 if it works out. I think S3 will be more seamless and efficient that way.

[wlandau]

Let's go with ropensci/targets#176 instead. I think it's as seamless as Metaflow.