beyondstorage/go-storage

A vendor-neutral storage library for Golang: Write once, run on every storage service.

go-storage's Issues

proposal: Support both directory and prefix based storager list

Issues #133, #102, #91 are about the same problem: Both directory and prefix based list are needed. We need to find a way out to support both directory and prefix based storager list.

And this issue will close all of them.

services/qingstor: DirFunc should be removed

services: Unify error handling

Proposal: https://github.com/Xuanwo/storage/blob/master/docs/design/11-error-handling.md
Spec: https://github.com/Xuanwo/storage/blob/master/docs/spec/1-error-handling.md

• azblob
• cos
• dropbox
• fs
• gcs
• kodo
• oss
• qingstor
• s3
• uss

Proposal: Support context in all public API

proposal: Add default set for operations

Allow user set storage class while init services.

proposal: Add config struct

proposal: Go version policy

Start from go 1.13, we will test target the latest two major version.

• At go1.13, we will test on go1.13
• After go1.14, we will test on go1.13 and go1.14
• After go1.15, we will test on go1.14 and go1.15

Any install/setup/build error report on not tested version will be marked as wontfix

serivces/*: Errors returned by New not handled

Proposal: Support service init via config string

Background

This project intents to be a unified storage layer for Golang, but different storage layers' configuration are so different we can't unify them well.

For posixfs: we only need to specify the workdir.
For object storage: we need to specify host, port, protocol, access key id and others.

We used to support them by type and options, like we did in qscamel:

Every service(endpoint in qscamel) should handle their own options:

type Client struct {
	BucketName          string `yaml:"bucket_name"`
	Endpoint            string `yaml:"endpoint"`
	Region              string `yaml:"region"`
	AccessKeyID         string `yaml:"access_key_id"`
	SecretAccessKey     string `yaml:"secret_access_key"`
	DisableSSL          bool   `yaml:"disable_ssl"`
	UseAccelerate       bool   `yaml:"use_accelerate"`
	PathStyle           bool   `yaml:"path_style"`
	EnableListObjectsV2 bool   `yaml:"enable_list_objects_v2"`
	EnableSignatureV2   bool   `yaml:"enable_signature_v2"`
	DisableURICleaning  bool   `yaml:"disable_uri_cleaning"`

	Path string

	client *s3.S3
}

func New(ctx context.Context, et uint8, hc *http.Client) (c *Client, err error) {
	...
	content, err := yaml.Marshal(e.Options)
	if err != nil {
		return
	}
	err = yaml.Unmarshal(content, c)
	if err != nil {
		return
	}
    ...
}

Developer who want to use this service should handle the type:

switch t.Src.Type {
...
case constants.EndpointS3:
    src, err = s3.New(ctx, constants.SourceEndpoint, contexts.Client)
    if err != nil {
        return
    }
...
default:
    logrus.Errorf("Type %s is not supported.", t.Src.Type)
    err = constants.ErrEndpointNotSupported
    return
}

User should set them in config directly:

source:
  type: s3
  path: "/path/to/source"
  options:
    bucket_name: example_bucket
    endpoint: example_endpoint
    region: example_region
    access_key_id: example_access_key_id
    secret_access_key: example_secret_access_key
    disable_ssl: false
    use_accelerate: false
    path_style: false
    enable_list_objects_v2: false
    enable_signature_v2: false
    disable_uri_cleaning: false

It works, but it doesn't meet our goal. To address this problem, we split endpoint and credential in PR services: Split endpoint and credential into different pair. In this PR, we can init an object service like:

srv := qingstor.New()
err = srv.Init(
    pairs.WithCredential(credential.NewStatic(accessKey, secretKey)),
    pairs.WithEndpoint(endpoint.NewStaticFromParsedURL(protocol, host, port)),
)
if err != nil {
    log.Printf("service init failed: %v", err)
}

It's better, but not enough. We need a general way to init all service like:

srv := storage.SomeCall("<type>", something)

Proposal

So I propose following changes:

Introduce the concept of "config string"

config string is widely used in db connections:

mysql: user:password@/dbname?charset=utf8&parseTime=True&loc=Local
postgres: host=myhost port=myport user=gorm dbname=gorm password=mypassword
sqlserver: sqlserver://username:password@localhost:1433?database=dbname

Like we did in URL, we can use different part in a formatted string to represent different meaning.

Config string in storage would be like:

<type>://<config>
             +
             |
             v
<credential>@<endpoint>/<name>?<options>
     +            +                 +
     |            +------------+    +----------------------+
     v                         v                           v
<protocol>:<data>   <protocol>:<host>:<port>   <key>:<value>[&<key>:<value>]

• credential: <protocol>:<data>, static credential could be static://<access_key>:<secret_key>.
• endpoint: <protocol>:<host>:<port>, qingstor's valid endpoint could be https://qingstor.com:443, 80 and 443 can be emitted with matched protocol.
• name: a valid storager name for this services
• options: multiple <key>=<value> connected with &

So a valid config string could be:

• qos://static:<access_key_id>:<secret_access_key>@https:qingstor.com:443/<bucket_name>?zone=pek3b&work_dir=/storage
• posixfs:///<path>

services/*: Stat/List should return Dir if it exists

proposal: Fix the process for proposal & spec

Split services into different go modules?

proposal: Remove invalid object type

Add basic cos support

Proposal: Normalize metadata name

Move object storage's getAbsPath and getRelPath to pkg

services/fs: Size and Offset in Read handled incorrectly

if opt.HasSize && opt.HasOffset {
	return iowrap.SectionReadCloser(f, opt.Offset, opt.Size), nil
}
if opt.HasSize {
	return iowrap.LimitReadCloser(f, opt.Size), nil
}
if opt.HasOffset {
	_, err = f.Seek(opt.Offset, 0)
	if err != nil {
		return nil, fmt.Errorf(errorMessage, s, path, handleOsError(err))
	}
}
if opt.HasReadCallbackFunc {
	return iowrap.CallbackReadCloser(f, opt.ReadCallbackFunc), nil
}

should be like

if opt.HasSize {
	r = iowrap.LimitReadCloser(f, opt.Size), nil
}

proposal: Remove config string

Expose Open(t string, opt []*types.Pair) directly, and allow user to design their config format.

This proposal will close proposal: #167, and deprecate proposal 3-support-service-init-via-config-string

services/oss: DirFunc should be removed

proposal: Add strict mode for services

Let user to decide what we should do when we meet missing features:

• with strict, we always throw errors.
• without strict, we will ignore them.

Proposal: Split bucket and object metadata to different struct

services: Add ObjectFunc to list objects without delimiter

Split service init into Endpoint and Credential

Proposal: Unify error handle

• https://blog.golang.org/error-handling-and-go
• http://joeduffyblog.com/2016/02/07/the-error-model
• https://blog.golang.org/go1.13-errors

Test services like k8s sanity tester

ref: https://github.com/kubernetes-csi/csi-test/blob/master/pkg/sanity/README.md

services/*: Different between Name and ID should be aligned

• Name will be used in current Storager which is relative to WorkDir
• ID is the ID in current services which is the AbsPath or business ID

services/*: Content type should be set if available

Add config struct to help parse from string to pairs

Setup bdd test env for all storage services

• azblob: #331
• cos: #342
• fs
• gcs: #341
• kodo: #347
• oss: #343
• qingstor: #325
• s3: #338
  - [ ] uss

services/qingstor: List buckets could be incomplete

Add support for progress bar

Dependabot can't parse your go.mod

Dependabot couldn't parse the go.mod found at /go.mod.

The error Dependabot encountered was:

go: cloud.google.com/go/[email protected] requires
	golang.org/x/[email protected] requires
	dmitri.shuralyov.com/gpu/[email protected]: unrecognized import path "dmitri.shuralyov.com/gpu/mtl" (https fetch: Get https://dmitri.shuralyov.com/gpu/mtl?go-get=1: EOF)

View the update logs.

pkg/endpoint: Port could be omitted

Port is not always required in pkg/endpoint Parse.

• For https: port could be default to 443
• For http: port could be default to 80

tests: Some errors in standard library doesn't implement xerrors.Wrapper

strconv.NumError as an exmaple: we need to write following code for unittest:

e := &strconv.NumError{}
ok := errors.As(err, &e)
if ok {
	assert.True(t, errors.Is(e.Err, tt.err))
} else {
	assert.True(t, errors.Is(err, tt.err))
}

After go 1.14 released, we can change them into:

assert.True(t, errors.Is(err, tt.err))

Proposal: Move id to object type instead of metadata

PR #53 adds a new metadata Id.

This inspires me that we need an Id across all services:

For local file system: Id should be whole path towords root.
For object storage: Id should be the whole key.
For dropbox alike SaaS: Id should be their Id in business.

So let's move Id to object type instead of in metadata.

Unify behavior across all services

services/*: Object type in List is not correct

proposal: Normalize content md5 and etag

services: Pair work_dir seems to be ignored when new storage

https://github.com/Xuanwo/storage/blob/master/services/qingstor/servicer.go#L152

services/dropbox: meta.Name used incorrectly

o := &types.Object{
	ID:         meta.Id,
	Type:       types.ObjectTypeFile,
	Name:       filepath.Join(path, meta.Name),
	Size:       int64(meta.Size),
	UpdatedAt:  meta.ServerModified,
	ObjectMeta: metadata.NewObjectMeta(),
}

filepath.Join(path, meta.Name) is not the valid name for this file, use path directly?

Add support for metrics

• Op/s?
• read/write speed?
• error rate?

ref:

• https://github.com/rcrowley/go-metrics

Proposal: Remove storager.Init

Roadmap to v1.0.0

Roadmap

• After 2020-03-13, all new proposals that could lead to breaking changes will be postponed to next major release (v2.0.0)
• After 2020-03-20, all proposals that not implemented will be postponed to next minor release (v1.1.0)

Proposals

Following proposals are sure to be included in v1.0.0 (a.k.a v0.9.0)

• #100
• #113
• #134
• #156

Services

There are no ensurement for any new service to be included in v1.0.0

Steps

There will be two minor release before v1.0.0: v0.8.0 and v0.9.0

v0.8.0 will be released shortly after 2020-03-06, and v0.9.0 expected to be released in 2020-03-20.

After v0.9.0, no new proposals will be implemented until v1.0.0 released, and I will focus on bug fix and code cleanup.

Add basic uss support

services/*: Metadata set should ignore empty value

services/cos: newStorage Should be removed

Dependabot can't parse your go.mod

Dependabot couldn't parse the go.mod found at /go.mod.

The error Dependabot encountered was:

go: cloud.google.com/go/[email protected] requires
	golang.org/x/[email protected] requires
	dmitri.shuralyov.com/gpu/[email protected]: unrecognized import path "dmitri.shuralyov.com/gpu/mtl" (https fetch: Get https://dmitri.shuralyov.com/gpu/mtl?go-get=1: EOF)

View the update logs.

pkg/segment: Move segment map into a exported struct

segments    map[string]*segment.Segment
segmentLock sync.RWMutex

can be a struct in package segment.

Dependabot can't parse your go.mod

Dependabot couldn't parse the go.mod found at /go.mod.

The error Dependabot encountered was:

go: cloud.google.com/go/[email protected] requires
	golang.org/x/[email protected] requires
	dmitri.shuralyov.com/gpu/[email protected]: unrecognized import path "dmitri.shuralyov.com/gpu/mtl" (https fetch: Get https://dmitri.shuralyov.com/gpu/mtl?go-get=1: EOF)

View the update logs.

Proposal: Normalize metadata storage class

As described in Proposal: Normalize Metadata, we need to normalize private meta's value.

The first one is storage-class. Every object storage service has their own storage class set, we need to do convert between their SDK and storage.