yanganto/s3handler

S3handler

A s3 handler library for s3rs and nu-shell s3 plugin Here is the document.

Blocking API is ready

use s3handler = { features = ["blocking"] }

let config = s3handler::CredentialConfig{
    host: "s3.us-east-1.amazonaws.com".to_string(),
    access_key: "akey".to_string(),
    secret_key: "skey".to_string(),
    user: None,
    region: None, // default will be treated as us-east-1
    s3_type: None, // default will try to config as AWS S3 handler
    secure: None, // dafault is false, because the integrity protect by HMAC
};
let mut handler = s3handler::Handler::from(&config);
let _ = handler.la();

Async API

Basic CRUD is implemented, other advance features are under developing. add this dependency to your cargo.toml s3handler = { features = ["tokio-async"] }

Download a file with async api use s3handler = { features = ["tokio-async"] }

// Public resource
let s3_pool = s3handler::none_blocking::primitives::S3Pool::new("somewhere.in.the.world".to_string());
let obj = s3_pool.bucket("bucket_name").object("objcet_name");
async {
    obj.download_file("/path/to/save/a/file").await;
};

S3 async handler to manipulate objects and buckets. This treat all data as pool and create a canal to bridge two pool. It is easy to management and sync data from folder to S3, S3 to S3, event folder to folder.

   +------+
   | Pool | (UpPool)  modify by `from_*` api
   +------+
     |  ^
Pull |  | Push
     v  |
   +------+
   | Pool | (DownPool) modify by `toward_*` api
   +------+

use s3handler::none_blocking::traits::DataPool;

// Resource with AWS version 2 auth
let s3_pool = s3handler::none_blocking::primitives::S3Pool::new("somewhere.in.the.world".to_string())
        .aws_v2("access-key".to_string(), "secrete-key".to_string());
let bucket = s3_pool.bucket("bucket_name");
// Actually the bucket is a unconnnected canal
assert!(!bucket.is_connect());
let canal = bucket.toward("/path/to/another/folder").unwrap();
// The canal bridges the two folder and ready to transfer data between bucket and folder
assert!(canal.is_connect());
canal.sync().await;

let s3_pool = S3Pool::new(env::var("S3_HOST").unwrap()).aws_v4(
    akey.to_string(),
    env::var("SECRET_KEY").unwrap(),
    env::var("REGION").unwrap(),
);
let mut object_list = s3_pool
    .bucket(&env::var("BUCKET_NAME").unwrap())
    .list()
    .await
    .unwrap();
let obj = object_list.next_object().await.unwrap();

s3handler's Issues

Implement fetch meta async api

upgrade reqwest to 0.11

Detect file mime type when uploading

check the mime type of local file before upload

List objects api

Follow the design of ReadDir in tokio
https://docs.rs/tokio/0.3.4/tokio/fs/fn.read_dir.html

Async api with virtural host style request

The virtual host style for async APIs is not correct, the following is the response details.

<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>NoSuchBucket</Code><Message>The specified bucket does not exist</Message><BucketName>object-name</BucketName><RequestId>E01339A11961B4AF</RequestId><HostId>/fIPlnl9eXM8hyd9joj1NdaNQf8lf+AErLOT1dHIcUHHOfyhvdRPTc2YuRiMjnzlm+L9KJZU8wQ=</HostId></Error>

Put single object async

Test compatibility with OSS

Implement Trait tokio::stream::Stream for S3Pool

Handle start_after when listing objects

Async multipart upload

Version 2 signature for async api

• compatible with AWS S3
• compatible with Ceph (move to #24 )

Get single object async

async api

add async API and treat resource as pool and stream

let s3_pool = DataPool {
    host: "s3.us-east-1.amazonaws.com".to_string(),
    access_identifier: "access key".to_string(),
    secret: "secret key".to_string(),
    region: None,  // default None will treat as Some("us-east-1")
    data_type: s3handler::DataType:AWS4,
    secure: None,  // dafault None is false, because the integrity protect by HMAC
};

let mut bucket = s3_pool.as_base_from("s3://bucket_name").unwarp();

for obj in bucket.list().await.unwrap().into_iter() {
    obj.pull().await;
};

// upload objects
for obj in bucket.upstream_list().await.unwrap().into_iter() {
    obj.push().await;
};

// `list_after` is a short cut for `upstream_list_after`, if view_point: s3handler::ViewPoint::Upstream,
for obj in bucket.list_after("index").unwarp().await.into_iter(){
    obj.pull().await;
};

// `from` will put the pool to the up_pool of a Canal
// `toward` will put the pool to the down_pool of a Canal
let mut bucket = s3_pool.as_base_from("s3://bucket_name/").unwarp();
let mut object = s3_pool.as_base_from("s3://bucket_name/file").unwarp();
let mut object = s3_pool.as_target_to("/a/absolute/path/to/file").unwarp();
let mut object = s3_pool.as_target_to("./a/relations/path/to/file").unwarp();
let mut folder = s3_pool.as_target_to("/a/absolute/path/to/folder/").unwarp();
let mut folder = s3_pool.as_target_to("./a/relations/path/to/folder/").unwarp();
let error = s3_pool.as_target_to("file");
let error = s3_pool.as_base_from("file/");

// `list_from` is a short cut for `downstream_list`, if view_point: s3handler::ViewPoint::Downstream,
for obj in folder.set_upstream(s3://bucket_name).list().await.unwarp().into_iter() {
    obj.up().await;
};

// `object` will base on the view point to call `from_object` or `toward_object`
// the `file`, `key` method do the same thing
let mut object = bucket.object("/a/absolute/path/to/file");
// `sync` method will check before the timestamp than call `up` or `down`
let result = object.sync().await;


// flowing code present to  upload a local file to s3 object /bucket_name/new_file
let mut canal = s3_pool.as_base_from("s3://bucket_name/").unwarp();
canal.toward_object("/a/path/to/file").key("new_file");
let result = canal.up().await;

// These methods of Canal will consume and return a new Canal
// `object()`, `bucket()`, `folder()`, `key()`
let object = s3_pool.as_base_from("s3://bucket_name/object_name").unwarp();
let new_object = object.bucket("another_bucket");
// now `new_object` equal to `s3_pool.as_base_from("s3://bucket_name/object_name").unwarp()`
let bucket = s3_pool.as_base_from("s3://bucket_name").unwarp();
let new_object = bucket.object("object_name");
// now `new_object` equal to `s3_pool.as_base_from("s3://bucket_name/object_name").unwarp()`

// These methods of Canal will modify Canal
// `toward_object()`, `toward_bucket()`, `toward_folder()`, `toward_key()`
// `from_object()`, `from_bucket()`, `from_folder()`, `from_key()`
// `toward`, `from`

let mut object = s3_pool.as_base_from("s3://bucket_name/object_name").unwarp();
object.from_bucket("another_bucket");
// now `object` equal to `s3_pool.from("s3://bucket_name/object_name").unwarp()`

Test compatibility with Ceph

• version2
• version4

Multipart download for blocking api

Support different S3 provider

As you can see the aws.rs is implement the AWS2Client and AWS4Client.
Some S3 provider, such as Aliyun, change the wording from amz to oss, and the sign process of the HTTP requests may a slightly different from others. But the overall upload and download logic is the same, you may implement a client with S3Client trait to make a valid request for other S3 providers.
https://github.com/yanganto/s3handler/blob/master/src/lib.rs#L100

There are methods need to be implemented for S3Client trait, the option methods can use unimplemented! to skip, and all function can work without redirect scenario:

• fn request - to sign a request follow the rule of S3 provider (required)
• fn redirect_parser, update, current_region, are used to handle the scenario that the service provider want you redirect.

Async multipart download

Version 4 signature for async api

• compatible with AWS

Document & polish the api