jkobejs/zio-google-cloud-storage

zio-google-cloud-storage's Issues

Implement retries

Per google docs clients should use truncated exponential backoff for all requests to Cloud Storage that return HTTP 5xx and 429 response codes.

Design and use truncated exponential backoff where it is necessary. See https://cloud.google.com/storage/docs/exponential-backoff and https://cloud.google.com/storage/docs/json_api/v1/how-tos/multipart-upload#handling_errors for more info.

Implement missing object and bucket API

Implement missing bucket and storage object API

Reply below to claim an item.
Current missing bucket API with documentation:

• delete - https://cloud.google.com/storage/docs/json_api/v1/buckets/delete
• get - https://cloud.google.com/storage/docs/json_api/v1/buckets/get
• insert - https://cloud.google.com/storage/docs/json_api/v1/buckets/insert
• list - https://cloud.google.com/storage/docs/json_api/v1/buckets/list
• patch - https://cloud.google.com/storage/docs/json_api/v1/buckets/patch
• update - https://cloud.google.com/storage/docs/json_api/v1/buckets/update

Current missing storage object API with documentation:

• copy - https://cloud.google.com/storage/docs/json_api/v1/objects/copy
• patch - https://cloud.google.com/storage/docs/json_api/v1/objects/patch
• update - https://cloud.google.com/storage/docs/json_api/v1/objects/update
• rewrite - https://cloud.google.com/storage/docs/json_api/v1/objects/rewrite

Add support for all fields in domain objects

Currently domain objects are very thin, we need to add all field that are used in google cloud storage api.

Reply below to claim an item.
Current objects with missing fields and links with documentation:

• Storage object - https://cloud.google.com/storage/docs/json_api/v1/objects#resource
• Compose object - https://cloud.google.com/storage/docs/json_api/v1/objects/compose
• Bucket object - https://cloud.google.com/storage/docs/json_api/v1/buckets#resource

Add support for API optional query parameters

Add support for API optional query parameters.
Reply below to claim an item.

• delete - https://cloud.google.com/storage/docs/json_api/v1/buckets/delete#parameters
• get - https://cloud.google.com/storage/docs/json_api/v1/buckets/get#parameters
• insert - https://cloud.google.com/storage/docs/json_api/v1/buckets/insert#parameters
• list - https://cloud.google.com/storage/docs/json_api/v1/buckets/list#parameters
• patch - https://cloud.google.com/storage/docs/json_api/v1/buckets/patch#parameters
• update - https://cloud.google.com/storage/docs/json_api/v1/buckets/update#parameters

Current missing storage object API with documentation:

• compose - https://cloud.google.com/storage/docs/json_api/v1/objects/compose#parameters
• copy - https://cloud.google.com/storage/docs/json_api/v1/objects/copy#parameters
• delete - https://cloud.google.com/storage/docs/json_api/v1/objects/delete#parameters
• get - https://cloud.google.com/storage/docs/json_api/v1/objects/get#parameters
• insert - https://cloud.google.com/storage/docs/json_api/v1/objects/insert#parameters
• list - https://cloud.google.com/storage/docs/json_api/v1/objects/list#parameters
• patch - https://cloud.google.com/storage/docs/json_api/v1/objects/patch#parameters
• update - https://cloud.google.com/storage/docs/json_api/v1/objects/update#parameters
• rewrite - https://cloud.google.com/storage/docs/json_api/v1/objects/rewrite#parameters

Add support for content types

Currently content type when uploading and downloading files is hardcoded to application/octet-stream.
Add support for file content types. See https://cloud.google.com/storage/docs/json_api/v1/parameters#contenttype for more info.

Provide access token as API method parameter wrapped in ZIO

Provide access token as API method parameter wrapped in ZIO

---

Motivation

zio-google-cloud-storage currently depends on zio-google-cloud-oauth2 library and uses it to get access token to perform HTTP calls when API methods are called.

Imagine that user wants to list bucket content to get stream of storage objects and then for every object download it. For bucket with N storage objects that means that we need to perform authentication at least N times. Since one access tokens can be used for 1h it should be cached for <1h to reduce HTTP calls to google servers.
To reduce number of HTTP calls to auth servers we can do following:

1. cache access tokens internally
2. leave caching to user and and new parameter to API methods accessToken: ZIO[R, AuthError, String]

Opting for caching access token internally means that user will be able to use only one service account to access his cloud storage when using storage service. If user has many service accounts used in different projects user will need to create many Storages and keep track of them to use library properly.

Caching should be left to user and described in documentation with an example how perform authentication and cache it to use in API calls.

---

Remove zio-google-cloud-oauth2 dependency and add new parameter to storage API methods that will be description of effectful authentication to google servers, accessToken: ZIO[R, AuthError, String]

Add support for standard query paramaters

Add support for standard query parameters where it makes sense.
See https://cloud.google.com/storage/docs/json_api/v1/parameters#query for more info.