imroc / req Goto Github PK
View Code? Open in Web Editor NEWSimple Go HTTP client with Black Magic
Home Page: https://req.cool
License: MIT License
Simple Go HTTP client with Black Magic
Home Page: https://req.cool
License: MIT License
Line 71 in db28577
protobuf 这两年越来越流行了,从此人们在 json over http 之上,又多了一种选择。如果能够直接支持会更方便一些。
I've got response while compile if I refer it as a mod directly:
import "github.com/imroc/req"
cannot load github.com/imroc/req: cannot find module providing package github.com/imroc/req
for client cert and trusted ca cert
thx
Hello,
Is the http/https proxy authentication in plan?
will socks5 proxy be supported?
it would be nice if cookiejar could be supported.
best regards
demo:
header := make(http.Header)
header.Set("Accept", "application/json")
resp, _ := req.Get("https://www.baidu.com", header)
defer resp.Response().Body.Close()
请问每次发起请求后 我需要自行加上下面这行代码吗?
defer resp.Response().Body.Close()
debug模式下dumpReqHead方法中由于没有读取response内容或者关闭response,导致setRequestCancel方法中stopTimerCh一直得不到值,直到超时
使用for循环执行req.Post(),提示
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x653685]
goroutine 1 [running]:
github.com/imroc/req.(*Resp).Response(0x0, 0x37)
/root/go/src/github.com/imroc/req/resp.go:36 +0x5
main.main()
还有一个问题,如果请求的URL连接失败,需要做特殊处理吗?
还是
r.Response().Body.Close()
continue
即可。
感谢。
req.Debug = true
//打开文件句柄操作
// file, _ := os.Open(filename)
progress := func(current, total int64) {
fmt.Println(float32(current)/float32(total)*100, "%")
}
authHeader := req.Header{
"User-Agent": "Go-http-client/1.2",
"Content-Type": "image/png",
}
req.Post(tinypngPushApi,authHeader, req.File(filename),req.UploadProgress(progress))
return nil
}
2.4035301 %
POST /web/shrink HTTP/1.1
Host: tinypng.com
User-Agent: Go-http-client/1.2
Transfer-Encoding: chunked
Content-Type: multipart/form-data; boundary=bbc4ac77ebd45833cb74d9971017228148c453e4bfa38de6bc1eb77c7bc3
Accept-Encoding: gzip
--d05faa4986965b6bed833f966fd22a11adeb86dd774685c41bd33996af04
Content-Disposition: form-data; name="media"; filename="e.png"
Content-Type: application/octet-stream
--d05faa4986965b6bed833f966fd22a11adeb86dd774685c41bd33996af04--
=================================
HTTP/2.0 415 Unsupported Media Type
Content-Length: 73
Alt-Svc: clear
Content-Type: application/json
Date: Wed, 14 Aug 2019 03:14:11 GMT
Via: 1.1 google
{"error":"unsupported_media_type","message":"File type is not supported"}
authHeader := req.Header{
//"Authorization": "Basic YWRtaW46YWRtaW4=",
"Authorization": basicAuth("admin", "password"),
}
resp, _ := req.Get("http://localhost:9112", authHeader)
println(resp.String())
How to finish the basicAuth function ?
一般认为显式使用全局变量无法保证goroutine安全,虽然这对于bool型而言基本不成立,但还是建议导出函数而非全局变量。
How to cancel request?
https://github.com/imroc/req/blob/master/req.go#L156
text|xml|json|javascript|charset|java
https://github.com/imroc/req/blob/master/req.go#L295
respBody, err := ioutil.ReadAll(reader)
我要获取1个txt文件的StatusCode或Header信息,文件大概有1G多,req会强制下载Body内容写入内存.
而我不需要Body内容,只需要返回的StatusCode或Header信息.
var r *req.Resp
req.SetTimeout(10 * time.Second)
req.SetFlags(req.LstdFlags | req.Lcost) // 输出格式显示请求耗时
if config.APP.Debug {
req.Debug = true
}
if method == "get" {
r, err = req.Get("http://baidu.com", header)
} else if method == "post" {
r, err = req.Post("http://baidu.com", header, req.BodyJSON(&content))
}
if err != nil {
return nil, err
}
log.Debug("a1111", r)
输出的日志里面有请求方法、请求URL、请求参数、response。但是没有看到请求耗时。
Hi,
i want to upload an file to oboom com via API. And req dont set the correct content length header.
I get the following message: "missing or invalid Content-Length header"
I debug it and content length it always 0.
Is it possible to set correct header?
如果是一个图片的话, 可以支持toBase64()就更好了。
是不是考虑加个连接池?
I have a function which calls resp, err := req.Post(url, req.BodyJSON(requestJson))
for each row in CSV to ingest data. When one of the requests fails (for genuine reason), the subsequent post call (most of the time only 1st call after the failed request) throws EOF even though there is no issues with data. This is not consistent though.
But, while testing I saw if we use req.New()
and use this to post then the above error is not reproducible.
Is this expect? and also are there any implications if we use req.New()
for each request instead of req.Post()
method?
Hello,
I found the req does use the default json decode for req.ToJSON, is it possible to support https://github.com/json-iterator/go which is more tolerant?
--- POST http://localhost:8080/api?data1=data1&data2=data2
body content
it seems no way to send request like this.
if r.Cost() > 3 * time.Second { // 检查耗时
log.Println("WARN: slow request:", r)
}
if r.Cost() / 1e6 > 3 * time.Microsecond { // 检查耗时
log.Println("WARN: slow request:", r)
}
请查看
type OnLineUserParam struct {
Start int `json:"start"`
Limit int `json:"limit"`
Sort string `json:"sort"`
Dir string `json:"dir"`
Grpid string `json:"grpid"`
Recflag string `json:"recflag"`
Keystr string `json:"keystr"`
Type string `json:"type"`
}
像上面的结构体 要如何转换成你的Param?
code := "sdfgsdgdfgjskgjakdfgl;sfgksljkglsklsdgklsdfg"
How to set Timeout for specify request and not affect global settings?
你传参数的方式很好,只是没有具体研究过这种方式(v.(type))是否有些性能影响。
在工作中也在用这个package, 谈下我的个人想法:
在传实体(json,xml)时,能否通过header去判断而不是用 BodyJSON等方式
req.Post(url, req.BodyJSON(&foo))
如:
//数据实体
type ReqData struct {
v interface{}
}
//在外边用的时候,传入header
req.Header{
"Accept": "application/json", //可以写一些公共的,大家直接用就可以了
"Content-Type": "application/json",
}
//在POST提交的方法中,通过判断header去完成不同的序列化
//同时返回也可以通过header去判断完成反序列化,不过相应的结构会有些调整
这个库有没有可以设置不跟随重定向的选项啊,我访问不想跟随302跳转
It could be pretty useful I think
I try to make a request with query params. When i define params
with map[string]interface{}
variable type, req is not sending query params.
Here is the detail
var params map[string]interface{}
params = req.QueryParam{
"key": "value",
}
req.Get("https://something.com/path/", params)
But it's work well when params
variable type is auto assigned.
params := req.QueryParam{
"key": "value",
}
req.Get("https://something.com/path/", params)
thank u.
here is my code, hope it could be helpful.
type RetryableClient struct {
Client *http.Client
Retryable *Retryable
}
type Retryable struct {
RetryableStatus []int
RetryerTime time.Duration
RetryerCount int
Attempt int
Enable bool
}
func NewRetryClient() *RetryableClient {
return &RetryableClient{
Client: &http.Client{
Timeout: 2 * time.Second,
},
Retryable: &Retryable{
RetryableStatus: []int{
http.StatusBadRequest, http.StatusInternalServerError,
http.StatusBadGateway, http.StatusRequestTimeout,
},
RetryerTime: time.Second,
RetryerCount: 5,
Attempt: 0,
Enable: true,
},
}
}
func (rr *RetryableClient) Disable() {
rr.Retryable.Enable = false
}
func (rr *RetryableClient) SetTimeout(timeout time.Duration) {
rr.Client.Timeout = timeout
}
func (rr *RetryableClient) SetRetryStatus(statusCode ...int) {
var validStatusCode []int
for _, code := range statusCode {
if len(http.StatusText(code)) != 0 {
validStatusCode = append(validStatusCode, code)
}
}
rr.Retryable.RetryableStatus = validStatusCode
}
func (rr *RetryableClient) Do(req *http.Request) (*http.Response, []byte, error) {
var (
resp *http.Response
body []byte
err error
)
for {
resp, err = rr.Client.Do(req)
if err != nil {
revel.WARN.Printf("do request error: %v Attempt count: %v Request: %v", err, rr.Retryable.Attempt+1, req)
if rr.Retryable.Enable && rr.Retryable.Attempt < rr.Retryable.RetryerCount {
rr.Retryable.Attempt++
continue
} else {
return resp, body, err
}
}
defer resp.Body.Close()
if rr.Retryable.Enable && rr.Retryable.Attempt < rr.Retryable.RetryerCount && contains(resp.StatusCode, rr.Retryable.RetryableStatus) {
rr.Retryable.Attempt++
time.Sleep(rr.Retryable.RetryerTime)
revel.WARN.Printf("Status code: %v Attempt count: %v Request: %v", resp.StatusCode, rr.Retryable.Attempt, req)
continue
} else {
resp.Header.Set("Retry-Count", strconv.Itoa(rr.Retryable.Attempt))
}
body, err = ioutil.ReadAll(resp.Body)
if err != nil {
revel.WARN.Printf("read body from response failed: %v Attempt count: %v Request: %v", err, rr.Retryable.Attempt+1, req)
if rr.Retryable.Enable && rr.Retryable.Attempt < rr.Retryable.RetryerCount {
rr.Retryable.Attempt++
continue
} else {
return resp, body, err
}
}
break
}
if resp.StatusCode != http.StatusOK {
return resp, body, errors.New("bad response status code :" + strconv.Itoa(resp.StatusCode))
}
return resp, body, err
}
func contains(respStatus int, statuses []int) bool {
for _, status := range statuses {
if status == respStatus {
return true
}
}
return false
}
I have a few services in my project.
So I wanna set Debug for one of them.
What do you think of inject debug flag in the Req client?
But it seems hard for compatibility with current implementation.
I mean, if we have set one of them, what to do?
Now I need to change this value before and after each request, it doesn't thread-safe.
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x45d9f48]
goroutine 124 [running]:
github.com/imroc/req.(*Resp).dumpReqHead(0xc00024a580, 0xc00058b3e8)
/Users/xh/go/pkg/mod/github.com/imroc/[email protected]/dump.go:117 +0x128
github.com/imroc/req.(*Resp).dumpRequest(0xc00024a580, 0xc00058b3e8)
/Users/xh/go/pkg/mod/github.com/imroc/[email protected]/dump.go:103 +0x18f
github.com/imroc/req.(*Resp).Dump(0xc00024a580, 0x4d92d16, 0x10)
/Users/xh/go/pkg/mod/github.com/imroc/[email protected]/dump.go:206 +0xc8
github.com/imroc/req.(*Req).Do(0xc0001889a0, 0x4d89712, 0x4, 0xc0000d0000, 0x44, 0xc0005edc48, 0x4, 0x4, 0xc0005edaa8, 0x403ea3b, ...)
/Users/xh/go/pkg/mod/github.com/imroc/[email protected]/req.go:355 +0xafe
github.com/imroc/req.(*Req).Post(...)
/Users/xh/go/pkg/mod/github.com/imroc/[email protected]/req.go:621
......
调用代码
post, err := r.Post(domain, header, req.FileUpload{
File: file,
FileName: filename,
FieldName: "file",
}, req.UploadProgress(progress),ctx)
if err != nil {
return nil, err
}
跟踪内存发现,已经释放了r.req, 但是debug模式还在调用r.req.URL,
改成非debug模式或者关闭ctx就无问题
全路径情况没有跳转可以正常获得cookie。
有301 或者任意重定向之后的set-cookie获取不到。
有什么办法可以解决嘛
是不是可以考虑给这个包添一些类似viper的操作 这样获取回json数据后直接可以操作里面的值。
例如
`package main
import (
"fmt"
"github.com/ruolinn/req"
)
/*
{
"id": 1,
"name": "Leanne Graham",
"username": "Bret",
"email": "[email protected]",
"address": {
"street": "Kulas Light",
"suite": "Apt. 556",
"city": "Gwenborough",
"zipcode": "92998-3874",
"geo": {
"lat": "-37.3159",
"lng": "81.1496"
}
},
"phone": "1-770-736-8031 x56442",
"website": "hildegard.org",
"company": {
"name": "Romaguera-Crona",
"catchPhrase": "Multi-layered client-server neural-net",
"bs": "harness real-time e-markets"
}
}
*/
func main() {
rsp, _ := req.Get("https://jsonplaceholder.typicode.com/users/1")
fmt.Println(rsp.GetFloat64("address.geo.lng"))
}
`
我fork作者的库有过修改, 不知道这样合不合法 因为也是自己用,如果有冒犯的作者 我这边后面删除。
goroutine 77828 [select]:
net/http.(*persistConn).roundTrip(0xc00030ac60, 0xc0005ec9f0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:2498 +0x756
net/http.(*Transport).roundTrip(0xc0001da280, 0xc0015b1d00, 0xf8, 0xc0013a2701, 0xc0015b1d00)
/usr/local/go/src/net/http/transport.go:565 +0xad9
net/http.(*Transport).RoundTrip(0xc0001da280, 0xc0015b1d00, 0xc0001da280, 0xbfc4b6ce03456f8c, 0x7ff4564d55e)
/usr/local/go/src/net/http/roundtrip.go:17 +0x35
net/http.send(0xc0015b1c00, 0xe4d440, 0xc0001da280, 0xbfc4b6ce03456f8c, 0x7ff4564d55e, 0x1746a60, 0xc001a6c318, 0xbfc4b6ce03456f8c, 0x1, 0x0)
/usr/local/go/src/net/http/client.go:252 +0x43e
net/http.(*Client).send(0xc0002be6f0, 0xc0015b1c00, 0xbfc4b6ce03456f8c, 0x7ff4564d55e, 0x1746a60, 0xc001a6c318, 0x0, 0x1, 0x11)
/usr/local/go/src/net/http/client.go:176 +0xfa
net/http.(*Client).do(0xc0002be6f0, 0xc0015b1c00, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:699 +0x44a
net/http.(*Client).Do(...)
/usr/local/go/src/net/http/client.go:567
github.com/imroc/req.(*Req).Do(0xc00009b350, 0xd163d6, 0x3, 0xd3a72d, 0x29, 0x0, 0x0, 0x0, 0x469e1c, 0x40, ...)
/root/go/pkg/mod/github.com/imroc/[email protected]/req.go:323 +0xa0e
github.com/imroc/req.(*Req).Get(...)
/root/go/pkg/mod/github.com/imroc/[email protected]/req.go:612
github.com/imroc/req.Get(0xd3a72d, 0x29, 0x0, 0x0, 0x0, 0x1, 0x11, 0xc0004b17f0)
/root/go/pkg/mod/github.com/imroc/[email protected]/req.go:647 +0x7e
reptile/utils.GetProxy(0xc000ed6e60, 0x4)
/root/go/src/reptile/utils/get_proxy.go:12 +0x4b
reptile/utils.GetActiveProxy(0xc000ed6e40, 0x24, 0xc001060270, 0xca6e01, 0x0)
/root/go/src/reptile/utils/set_proxy.go:26 +0x53
reptile/vio.GetCacheClient(0xc00116d740, 0x12, 0x9f90a0)
/root/go/src/reptile/vio/index.go:389 +0x300
reptile/vio.FindNumberViolation(0x0, 0x0, 0xc0010dc8a0, 0x10, 0x1, 0x0, 0xc0010dc8a0, 0x10, 0x0)
/root/go/src/reptile/vio/index.go:480 +0x1106
reptile/controllers.(*VioController).FindNumberViolation(0xc000566a50)
/root/go/src/reptile/controllers/vio_controller.go:551 +0x368
reflect.Value.call(0xd10d40, 0xc000566a50, 0x3a13, 0xd16fe8, 0x4, 0x17729a0, 0x0, 0x0, 0x4b8277, 0xd10d40, ...)
/usr/local/go/src/reflect/value.go:460 +0x8ab
reflect.Value.Call(0xd10d40, 0xc000566a50, 0x3a13, 0x17729a0, 0x0, 0x0, 0x17729a0, 0x0, 0x0)
/usr/local/go/src/reflect/value.go:321 +0xb4
github.com/astaxie/beego.(*ControllerRegister).ServeHTTP(0xc00014a000, 0xe5d040, 0xc000113dc0, 0xc0016a6300)
/root/go/pkg/mod/github.com/astaxie/[email protected]/router.go:853 +0x147c
net/http.serverHandler.ServeHTTP(0xc0001121c0, 0xe5d040, 0xc000113dc0, 0xc0016a6300)
/usr/local/go/src/net/http/server.go:2807 +0xa3
net/http.(*conn).serve(0xc0002a0140, 0xe5f7c0, 0xc00089c3c0)
/usr/local/go/src/net/http/server.go:1895 +0x86c
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2933 +0x35c
我需要从struct转为req.Param{},但是req.Param{}不支持struct中json tag。
struct param {
UserId *string json:"userId"
}
reqParam := req.Param{}
for k, v := range param {
reqParam[k] = v
}
fmt.Println(reqParam) //reqParam是{"UserId": xxx} 需要的是{"userId": xxx}
My proxy ip can used .but when use SetProxyUrl
,the result is malformed HTTP response "<html>"
the test code:
// 测试代理
func TestX5(t *testing.T) {
header := req.Header{
"Accept-Encoding": "gzip, deflate, br",
"Accept": "*/*",
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36",
}
req.SetProxyUrl("http://27.203.219.181:8060")
req.Debug = true
r, err := req.Get("https://www.cnblogs.com/", header)
if err != nil {
t.Error(err)
}
t.Log(r.String())
}
result:
Get https://www.cnblogs.com/: malformed HTTP response "<html>"
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x68 pc=0x12a78bc]
use branch : master
I am trying to upload a file to my server, but when I try to upload the file I get this:
error--> Post http://127.0.0.1:2047/ctrl/upload: net/http: HTTP/1.x transport connection broken: http: ContentLength=431 with Body length 667
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x68 pc=0x6bbc2c]
This is the client code:
package ctrl
import (
"fmt"
"os"
"github.com/imroc/req"
"strings"
"strconv"
)
func UploadFile(file_up string){
//get file info
fileInfo, err := os.Stat(file_up)
if err!=nil{
fmt.Println(err)
}
file, _ := os.Open(file_up)
upload:= req.FileUpload{
File: file,
FieldName: "file",
FileName: fileInfo.Name(),
}
//set header
header := req.Header{
"Content-Type": "multipart/form-data",
"Authorization": "BEARER "+readKey(),
"Content-Length": strconv.FormatInt(fileInfo.Size(),10), //convert bytes to string
}
defer file.Close()
//set progress
progress := func(current, total int64) {
fmt.Println(float32(current)/float32(total)*100, "%")
}
r,err:=req.Post("http://127.0.0.1:2047/ctrl/upload",upload,header,req.UploadProgress(progress))
if err != nil {
fmt.Println("error--> ",err)
}
fmt.Println(r.String())
resp := r.Response()
r.Response().Body.Close()
fmt.Println(resp.StatusCode)
}
The line that causes the problem is:
"Content-Length": strconv.FormatInt(fileInfo.Size(),10),
when I remove that line everything works fine, but the server log tells me this for each file uploaded
Content-Length = -1
and I need real Content-Length of each file for the record in the db
some case need this feature
使用时往往会有这样的场景,就是为当前模块创建一个独立的http client, 并且可以初始化一些默认参数
比方说:
defaultHeader := make(http.Header)
defaultHeader .Set("APPID", "xxxx")
r := req.New("https://api.github.com", defaultHeader)
res, _ := r.get("/users")
like “userIds=1&userIds=2&userIds=3”
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.