daroczig/botor

Reticulate wrapper on 'boto3' with convenient helper functions -- aka "boto fo(u)r R"

botor's Issues

s3_ls function not working in CRAN version 0.3.0

First of all, great idea to create an R-wrapper for boto3 to directly communicate with s3 within scripts!

I have found the s3_ls function not working in version 0.3.0 with R version 4.0.3 on Windows 10 (tested also in linux environments with same effect). The error message is:

Error in data.frame(bucket_name = uri_parts$bucket_name, key = object$data$Key, :
arguments imply differing number of rows: 1, 0

s3_ls fails when object owner is not provided

We have encountered a situation where

botor/R/s3.R

Line 274 in 652996b

owner = object$data$Owner$DisplayName,

is failing with

Error in data.frame(bucket_name = uri_parts$bucket_name, key = object$data$Key,  : 
  arguments imply differing number of rows: 1, 0

I ran browser() in the above function and discovered that $Owner was not defined in objects[[1]]$meta$__dict__. Commenting out the above line (L274) solved the issue. I did not create this object in question so I'm not sure if it's possible to create an object without an owner or if perhaps the credentials I'm using do not have permissions to view who the Owner is. (All of the other metadata referenced in that code chunk is there. )

cc @orenscarmeli

`s3_read` doesn't play nice with `openxlsx`

It seem that when trying to read an xlsx file with the openxlsx package doesn't work. botor::s3_read("s3://mybucket/example_file.xlsx", fun=openxlsx::read.xlsx) results in Error: openxlsx can only read .xlsx files

I can confirm that reading the file locally (openxlsx::read.xlsx("example_file.xlsx")) works fine. And readxl's read_xlsx also works with botor. I would probably prefer readxl over openxlsx anyway, but it would be useful to understand why these aren't working nicely together.

botor does not play nice with `aws sso login`

When I am working with boto3 I can connect to AWS by typing aws sso login in the terminal.

However, when I am trying this with botor, its not working.

Example

# Go through the sso process via browser
system("aws sso login")

s3 <- botor::botor_client("s3")
pr<-s3$list_objects_v2(Bucket = "<BUCKET>",Prefix = "<PREFIX>",Delimiter = "/")
Error: botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjectsV2 operation: The specified bucket does not exist

`s3_ls` fails without a useful error if uri does not end in `/`

Actual Behavior: s3_ls("s3://my-bucket") returns NULL in the case where my_bucket is valid. (ie s3_ls("s3://my-bucket/") would work)

My expectation would be that this would either work (returning a data frame of objects) or raise an error (similar to if the regex matching fails).

Persisting boto3 session different s3_read / s3_write calls

Hi all,
The helper read and write functions are really helpful when interacting with R and AWS S3. However there is a problem when it comes to reading / write large amount of files/R objects to AWS S3. If a the user has AWS KMS enabled on their AWS S3 bucket then AWS KMS is called each time s3_read/s3_write is used. This means the KMS could be called hundreds of times if s3_read/s3_write is put in a loop. If another boto_session or s3_client was passed as an parameter then the same session can be used over multiple calls to the function s3_read/s3_write. This should reduce a cost to AWS KMS dramatically.

# example of s3_read
s3_read(uri, fun, ..., extract = c("none", "gzip", "bzip2", "xz"), s3_client = s3())

# Example code in practise:
library(botor)
library(data.table)

# create s3 client
s3 <- s3()

# list files to be read:
s3_files <- s3_ls("s3://mybucket/my_files/")

# persisting s3 client session
df = rbindlist(lapply(s3_files$uri, s3_read, fun = fread, s3_client = s3))

provide a way to override endpoint_url in the default botor session

This would be SUPER helpful for s3 'compatible' object stores

Write to S3 with KMS encryption

Hi,

Thanks a lot for this great library.
It works fine, but I am not sure how to realize the following case: Upload a file to S3 with KMS key encryption. E.g. in Python Boto3 I would do the following:

s3.Bucket('test-bucket').upload_file('test.txt', 'experiments/test.txt', ExtraArgs={"ServerSideEncryption": "aws:kms", "SSEKMSKeyId": "alias/test-key" })

Can this be done already? Or would it be possible to add?

Thank you a lot.

Best regards

's3_ls()' gives error when accessing a specific bucket

Hi,

I am trying to list all the objects in an S3 bucket so I can read them later but I get the following error:

My code is this one where I want to read the file with the most recent modification date and show too the structure of the bucket:

Thanks!

Add Logging

Option to have logging of s3 paths when running s3_read/s3_write etc