Giter Site home page Giter Site logo

cos-go-sdk-v5's Introduction

cos-go-sdk-v5

腾讯云对象存储服务 COS(Cloud Object Storage) Go SDK(API 版本:V5 版本的 XML API)。

Install

go get -u github.com/tencentyun/cos-go-sdk-v5

Usage

package main

import (
	"context"
	"fmt"
	"io/ioutil"
	"net/http"
	"net/url"
	"os"
	"time"
	
	"github.com/tencentyun/cos-go-sdk-v5"
)

func main() {
	//将<bucket>和<region>修改为真实的信息
	//bucket的命名规则为{name}-{appid} ,此处填写的存储桶名称必须为此格式
	u, _ := url.Parse("https://<bucket>.cos.<region>.myqcloud.com")
	b := &cos.BaseURL{BucketURL: u}
	c := cos.NewClient(b, &http.Client{
		//设置超时时间
		Timeout: 100 * time.Second,
		Transport: &cos.AuthorizationTransport{
			//如实填写账号和密钥,也可以设置为环境变量
			SecretID:  os.Getenv("COS_SECRETID"),
			SecretKey: os.Getenv("COS_SECRETKEY"),
		},
	})

	name := "test/hello.txt"
	resp, err := c.Object.Get(context.Background(), name, nil)
	if err != nil {
		panic(err)
	}
	bs, _ := ioutil.ReadAll(resp.Body)
	resp.Body.Close()
	fmt.Printf("%s\n", string(bs))
}

所有的 API 在 example 目录下都有对应的使用示例。

Service API:

Bucket API:

Object API:

数据处理 API:

内容审核 API:

cos-go-sdk-v5's People

Contributors

agin719 avatar chloroplastyu avatar dt3310321 avatar jimmyyan avatar konakonall avatar likexian avatar mossju avatar tencentyuncos avatar toranger avatar vintmd avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

cos-go-sdk-v5's Issues

`tencentcloud/tencentcloud-sdk-go` 的版本冲突问题

sdk 依赖的tencentcloud-sdk-go是早期版本的 sdk,后面tencentcloud-sdk-go升级过 SDK,在每个子模块下都增加了 module。

github.com/tencentcloud/tencentcloud-sdk-go v1.0.96 

很多项目都会依赖高版本的tencentcloud-sdk-go,如果同时引入 cos 的 sdk,就会出现 path 模糊的问题,烦请更新一下tencentcloud-sdk-go依赖的版本。

附上冲突的情况

 github.com/abc/smsx imports
        github.com/tencentcloud/tencentcloud-sdk-go/tencentcloud/sms/v20190711: ambiguous import: found package github.com/tencentcloud/tencentcloud-sdk-go/tencentcloud/sms/v20190711 in multiple modules:
        github.com/tencentcloud/tencentcloud-sdk-go v1.0.96 (/Users/abc/go/pkg/mod/github.com/tencentcloud/[email protected]/tencentcloud/sms/v20190711)
        github.com/tencentcloud/tencentcloud-sdk-go/tencentcloud/sms v1.0.509 (/Users/abc/go/pkg/mod/github.com/tencentcloud/tencentcloud-sdk-go/tencentcloud/[email protected]/v20190711)

上传 sdk 报错 411

试了一下官方文档示例里的上传直接报错:

https://cloud.tencent.com/document/product/436/31215

// https://cloud.tencent.com/document/product/436/31215
package main

import (
	"context"
	"net/http"
	"net/url"
	"strings"

	"github.com/tencentyun/cos-go-sdk-v5"
)

var SecretID = ""    // 测试的时候是真实的ID,这里隐藏
var SecretKey = ""
var OCSURL = ""

func main() {
	// 将<BucketName-APPID>和<region>修改为真实的信息
	// 例如:http://examplebucket-1250000000.cos.ap-guangzhou.myqcloud.com
	u, _ := url.Parse(OCSURL)
	b := &cos.BaseURL{BucketURL: u}
	c := cos.NewClient(b, &http.Client{
		Transport: &cos.AuthorizationTransport{
			SecretID:  SecretID,
			SecretKey: SecretKey,
		},
	})
	// 对象键(Key)是对象在存储桶中的唯一标识。
	// 例如,在对象的访问域名 `examplebucket-1250000000.cos.ap-guangzhou.myqcloud.com/test/objectPut.go` 中,对象键为 test/objectPut.go
	name := "test/objectPut.go"
	// 1.Normal put string content
	f := strings.NewReader("test")

	_, err := c.Object.Put(context.Background(), name, f, nil)
	if err != nil {
		panic(err)
	}
	// 2.Put object by local file path
	_, err = c.Object.PutFromFile(context.Background(), name, "./test", nil)
	if err != nil {
		panic(err)
	}
}

测试文件 test 也存在,报错如下,修改了敏感信息:

panic: PUT http://xxx-appid.cos.ap-beijing.myqcloud.com/test%2FobjectPut.go: 411 (Message: , RequestId: , TraceId: )

goroutine 1 [running]:
main.main()
        testupload.go:40 +0x29e
exit status 2

看起来是 PutFromFile 没有发送文件长度头导致返回的 411。

Copy 接口返回「expected element type <CopyObjectResult> but have <Error>」

调用的接口原型:

// ObjectCopyResult is the result of Copy
type ObjectCopyResult struct {
	XMLName      xml.Name `xml:"CopyObjectResult"`
	ETag         string   `xml:"ETag,omitempty"`
	LastModified string   `xml:"LastModified,omitempty"`
	CRC64        string   `xml:"CRC64,omitempty"`
	VersionId    string   `xml:"VersionId,omitempty"`
}

func (s *ObjectService) Copy(ctx context.Context, key, sourceURL string, opt *ObjectCopyOptions, id ...string) (*ObjectCopyResult, *Response, error)

接口返回了「expected element type <CopyObjectResult> but have <Error>」这个错误,请问是预期内的吗?

v0.7.26 checksum mismatch

Our CI failed for this error: beyondstorage/go-service-cos#33

verifying github.com/tencentyun/[email protected]/go.mod: checksum mismatch
	downloaded: h1:O5LmgDLDk6mAsFLhQ2L/BNiJJDQIXE+mYFs2UiUMLaI=
	go.sum:     h1:4E4+bQ2gBVJcgEC9Cufwylio4mXOct2iu05WjgEBx1o=

SECURITY ERROR
This download does NOT match an earlier download recorded in go.sum.
The bits may have been replaced on the origin server, or an attacker may
have intercepted the download attempt.

For more information, see 'go help module-auth'.
make: *** [Makefile:38: tidy] Error 1

删除对象功能不支持删除对象的某一个版本

我看到API和Python SDK等一些其他SDK支持删除对象时指定对象的VersionID,但是在Go语言的SDK下我没有发现待删除的Object指定VersionID的地方。删除多个对象的接口也有同样的问题。

GetBucket或者GetBucketACL应该返回ACL

Hi,

PutBucket/PutBucketACL均支持ACL参数(存储桶公共权限),但是GetBucket/GetBucketACL的响应体里面并没有返回ACL,作为存储桶的一个重要参数,可否添加到响应体里面去?谢谢。

Support `POST` method for object presign

I've used the GetPresignedURL with POST method. But when I call the presign url with POST, it still return MethodNotAllow ? Could you support for POST method ? Or is there any document could guide me to solve this problem ? thx

ObjectPutHeaderOptions设置header值,值存在空格,会出现报错。

ObjectPutHeaderOptions设置header值,值存在空格,会出现报错。

主要的测试代码如下,错误处见代码注释:

func TestHelloWorld(t *testing.T) {
	u, _ := url.Parse("https://dochub-private-1251298948.cos.ap-guangzhou.myqcloud.com")
	b := &cos.BaseURL{BucketURL: u}
	c := cos.NewClient(b, &http.Client{
		Transport: &cos.AuthorizationTransport{
			SecretID:  "your SecretID",
			SecretKey: "your SecretKey",
			Transport: &debug.DebugRequestTransport{
				RequestHeader: true,
				// Notice when put a large file and set need the request body, might happend out of memory error.
				RequestBody:    false,
				ResponseHeader: true,
				ResponseBody:   true,
			},
		},
	})

	// Case1: without header
	name := "without-header.go"
	f := strings.NewReader("cos.go's content")
	opt := &cos.ObjectPutOptions{}
	_, err := c.Object.Put(context.Background(), name, f, opt)
	if err != nil {
		panic(err)
	}

	// Case2: with header ContentType
	name = "with-header-content-type.go"
	f = strings.NewReader("cos.go's content")
	opt = &cos.ObjectPutOptions{}
	header := &cos.ObjectPutHeaderOptions{
		ContentType: "text/html;%20charset=utf-8",
	}
	opt.ObjectPutHeaderOptions = header
	_, err = c.Object.Put(context.Background(), name, f, opt)
	if err != nil {
		panic(err)
	}

	// Case3: with header ContentDisposition
	name = "with-header-content-disposition.go"
	f = strings.NewReader("cos.go's content")
	opt = &cos.ObjectPutOptions{}
	header = &cos.ObjectPutHeaderOptions{
		// 这里的值没有空格,可以设置成功
		// 在对象存储的控制台,自定义header里看到 Content-Disposition值为attachment;%20filename=hello-world.go
		// 但是下载的附件,文件名不是"hello-world.go"
		// 如果把Content-Disposition值编辑为attachment; filename=hello-world.go
		// 则下载的附件名为 hello-world.go
		ContentDisposition: "attachment;%20filename=" + url.PathEscape("hello-world.go"),
	}
	opt.ObjectPutHeaderOptions = header
	_, err = c.Object.Put(context.Background(), name, f, opt)
	if err != nil {
		panic(err)
	}

	// Case4: with header ContentDisposition
	name = "with-header-content-disposition-error.go"
	f = strings.NewReader("cos.go's content")
	opt = &cos.ObjectPutOptions{}
	header = &cos.ObjectPutHeaderOptions{
		// 这里有空格,会报错,提示签名不匹配,导致上传失败
		ContentDisposition: "attachment; filename=" + url.PathEscape("hello-world.go"),
	}
	opt.ObjectPutHeaderOptions = header
	_, err = c.Object.Put(context.Background(), name, f, opt)
	if err != nil {
		panic(err)
	}
}

Object.GetPresignedURL 存在问题

按照管网的示例操作,获取到的url应该是可以直接用于请求的。
但我的输出是:

/hello_world.txt?sign=q-sign-algorithm%3Dsha1%26q-ak%3D%22AKID55pLcVJPQ0gjX8Dly03sWz6SdjfF2293%22%26q-sign-time%3D1572699886%3B1572699946%26q-key-time%3D1572699886%3B1572699946%26q-header-list%3D%26q-url-param-list%3D%26q-signature%3D231182c4fbd8df3d8366e5123e2267fb47dc1855
不仅没有Scheme和Host,Url还是encode后的,这与管网文档表现不符,是否为bug?
我的golang sdk版本为1.13.3

为什么 v0.7.12 的 presignURL 会有 u0026 转义问题?

image

package main

import (
	"context"
	"encoding/json"
	"fmt"
	"net/http"
	"net/url"
	"time"

	"github.com/tencentyun/cos-go-sdk-v5"
)

const (
	ak = "删除敏感信息"
	sk = "删除敏感信息"
)

func main() {

	//将<bucket>和<region>修改为真实的信息
	//bucket的命名规则为{name}-{appid} ,此处填写的存储桶名称必须为此格式
	u, _ := url.Parse("https://{这里是桶名字}.cos.ap-guangzhou.myqcloud.com")
	b := &cos.BaseURL{BucketURL: u}
	c := cos.NewClient(b, &http.Client{
		//设置超时时间
		Timeout: 100 * time.Second,
		Transport: &cos.AuthorizationTransport{
			//如实填写账号和密钥,也可以设置为环境变量
			SecretID:  ak,
			SecretKey: sk,
		},
	})
	name := `PICTURE/JN1DSG4EFV/test01/a_test.jpg`

	ans, err := c.Object.GetPresignedURL(context.TODO(), http.MethodPut, name, ak, sk, time.Minute*3, nil)
	fmt.Printf("ans:%v, err:%v\n", ans.String(), err)

	jsonBytes, _ := json.Marshal(map[string]interface{}{
		"PresignedURL": ans.String(),
	})

	fmt.Printf("ans_json marhsal:%v\n", string(jsonBytes))

}

Not supporting golang modules

go mod is the currently supported package manager for Golang, and it is supported by the offical branch. QCloud COS SDK shall support this as well or maintain a go mod version, instead of leaving the vendor directory there.

out of memory when use `PutFromFile`

My program reports out of memory when calling PutFromFile

Comment says that Notice that when use this put large file need set non-body of debug req/resp, otherwise will out of memory.

My question is : How can i set non-body of debug req/resp?

// PutFromFile put object from local file
// Notice that when use this put large file need set non-body of debug req/resp, otherwise will out of memory
func (s *ObjectService) PutFromFile(ctx context.Context, name string, filePath string, opt *ObjectPutOptions) (*Response, error) {
	fd, err := os.Open(filePath)
	if err != nil {
		return nil, err
	}
	defer fd.Close()

	return s.Put(ctx, name, fd, opt)
}

这个SDK目前应该不能用到生产环境的

粗略的看了代码发现了一些问题
来review一下这个函数吧

func (s *ObjectService) MultiUpload(ctx context.Context, name string, r io.Reader, opt *MultiUploadOptions) (*CompleteMultipartUploadResult, *Response, error) {

	optini := opt.OptIni
	res, _, err := s.InitiateMultipartUpload(ctx, name, optini)
	if err != nil {
		return nil, nil, err
	}
	uploadID := res.UploadID
	bufSize := opt.PartSize * 1024 * 1024
	buffer := make([]byte, bufSize)
	optcom := &CompleteMultipartUploadOptions{}

	PartUpload := func(ch chan *Response, ctx context.Context, name string, uploadId string, partNumber int, data io.Reader, opt *ObjectUploadPartOptions) {

		defer func() {
			if err := recover(); err != nil {
				fmt.Println(err)
			}
		}()
		resp, _ := s.UploadPart(context.Background(), name, uploadId, partNumber, data, nil)
		ch <- resp
	}

	chs := make([]chan *Response, 10000)
	PartNumber := 0
	for i := 1; true; i++ {
		bytesread, err := r.Read(buffer)
		if err != nil {
			if err != io.EOF {
				return nil, nil, err
			}
			PartNumber = i
			break
		}
		chs[i] = make(chan *Response)
		go PartUpload(chs[i], context.Background(), name, uploadID, i, strings.NewReader(string(buffer[:bytesread])), nil)
	}

	for i := 1; i < PartNumber; i++ {
		resp := <-chs[i]
		// Notice one part fail can not get the etag according.
		etag := resp.Header.Get("ETag")
		optcom.Parts = append(optcom.Parts, Object{
			PartNumber: i, ETag: etag},
		)
	}

	v, resp, err := s.CompleteMultipartUpload(context.Background(), name, uploadID, optcom)

	return v, resp, err
}

问题一:
分块上传中,假如这个文件1G,partSize为1MB,那么就会分成1024块,也就是会有1024个goroutine 并发的1024个http请求,内存占用也需要有1G,这个不做控制?

问题二:
代码PartUpload的部分是有可能返回一个nil的resp的,那么在下面的for循环中不作处理么?

for i := 1; i < PartNumber; i++ {
		resp := <-chs[i]
		// Notice one part fail can not get the etag according.
		etag := resp.Header.Get("ETag")
		optcom.Parts = append(optcom.Parts, Object{
			PartNumber: i, ETag: etag},
		)
	}

既然都标注了Notice,还不做一个resp为nil的判断?这个会panic的,如果resp为nil的话

Object的上传接口不再支持reader

这个功能是非常好用的,现在封装的上传接口只能传入文件名,感觉是舍本逐末的行为,希望能恢复原来的支持reader处理

怎么设置代理呢

服务器访问不了外网,想通过代理的方式请求cos,我看java是有sdk的,go没有找到

请问这个 AppendObject 的接口何时能开放使用

cos-go-sdk-v5/object.go

Lines 477 to 506 in e8b4da1

// TODO Append 接口在优化未开放使用
//
// Append请求可以将一个文件(Object)以分块追加的方式上传至 Bucket 中。使用Append Upload的文件必须事前被设定为Appendable。
// 当Appendable的文件被执行Put Object的操作以后,文件被覆盖,属性改变为Normal。
//
// 文件属性可以在Head Object操作中被查询到,当您发起Head Object请求时,会返回自定义Header『x-cos-object-type』,该Header只有两个枚举值:Normal或者Appendable。
//
// 追加上传建议文件大小1M - 5G。如果position的值和当前Object的长度不致,COS会返回409错误。
// 如果Append一个Normal的Object,COS会返回409 ObjectNotAppendable。
//
// Appendable的文件不可以被复制,不参与版本管理,不参与生命周期管理,不可跨区域复制。
//
// 当 r 不是 bytes.Buffer/bytes.Reader/strings.Reader 时,必须指定 opt.ObjectPutHeaderOptions.ContentLength
//
// https://www.qcloud.com/document/product/436/7741
// func (s *ObjectService) Append(ctx context.Context, name string, position int, r io.Reader, opt *ObjectPutOptions) (*Response, error) {
// u := fmt.Sprintf("/%s?append&position=%d", encodeURIComponent(name), position)
// if position != 0{
// opt = nil
// }
// sendOpt := sendOptions{
// baseURL: s.client.BaseURL.BucketURL,
// uri: u,
// method: http.MethodPost,
// optHeader: opt,
// body: r,
// }
// resp, err := s.client.send(ctx, &sendOpt)
// return resp, err
// }

注释中说功能还在优化,但是我看接口文档很久都没更新了,应该已经稳定了吧,希望能尽快在sdk里支持这个接口,工作上要用到,谢谢!

md5校验

putfromfile和upload方法如何在上传过程中支持md5校验?putfromfile可以自己在头信息中添加Content-MD5支持,upload方法这样设置不生效,如何解决?

use of closed network connection

sdk version: v0.7.25
put object, 偶而会出现, write tcp 172.18.x.x -> 121.4.234.x:443 use of closed network connection
这种情况, 怎么处理, 只能业务逻辑重试 再次put 吗?

版本升级报签名错误


github.com/tencentyun/cos-go-sdk-v5 v0.7.7
升级到
github.com/tencentyun/cos-go-sdk-v5 v0.7.9
报签名错误:
403 SignatureDoesNotMatch(Message: The Signature you specified is invalid.,

Request for adding old version tags

We are developing on a old code base and it depends on version v0.7.0, and it is not compatible with the latest version v0.7.30 because of unmatched variable type.

But this version cannot be found in this repo anymore

go get: github.com/tencentyun/[email protected]: invalid version: unknown revision v0.7.0

can you add the tag back?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.