Giter Site home page Giter Site logo

go-httpclient's Introduction

go-httpclient

Travis godoc License Go Report Card Coverage Status

Advanced HTTP client for golang.

Features

  • Chainable API
  • Direct file upload
  • Timeout
  • HTTP Proxy
  • Cookie
  • GZIP
  • Redirect Policy
  • Cancel(with context)

Installation

go get github.com/ddliu/go-httpclient

Quick Start

package main

import (
    "github.com/ddliu/go-httpclient"
)

func main() {
    httpclient.Defaults(httpclient.Map {
        httpclient.OPT_USERAGENT: "my awsome httpclient",
        "Accept-Language": "en-us",
    })

    res, err := httpclient.Get("http://google.com/search", map[string]string{
        "q": "news",
    })

    println(res.StatusCode, err)
}

Usage

Setup

Use httpclient.Defaults to setup default behaviors of the HTTP client.

httpclient.Defaults(httpclient.Map {
    httpclient.OPT_USERAGENT: "my awsome httpclient",
    "Accept-Language": "en-us",
})

The OPT_XXX options define basic behaviours of this client, other values are default request headers of this request. They are shared between different HTTP requests.

Sending Request

// get
httpclient.Get("http://httpbin.org/get", map[string]string{
    "q": "news",
})

// get with url.Values
httpclient.Get("http://httpbin.org/get", url.Values{
    "q": []string{"news", "today"}
})

// post
httpclient.Post("http://httpbin.org/post", map[string]string {
    "name": "value"
})

// post file(multipart)
httpclient.Post("http://httpbin.org/multipart", map[string]string {
    "@file": "/tmp/hello.pdf",
})

// put json
httpclient.PutJson("http://httpbin.org/put", 
`{
    "name": "hello",
}`)

// delete
httpclient.Delete("http://httpbin.org/delete")

// options
httpclient.Options("http://httpbin.org")

// head
httpclient.Head("http://httpbin.org/get")

Customize Request

Before you start a new HTTP request with Get or Post method, you can specify temporary options, headers or cookies for current request.

httpclient.
    WithHeader("User-Agent", "Super Robot").
    WithHeader("custom-header", "value").
    WithHeaders(map[string]string {
        "another-header": "another-value",
        "and-another-header": "another-value",
    }).
    WithOption(httpclient.OPT_TIMEOUT, 60).
    WithCookie(&http.Cookie{
        Name: "uid",
        Value: "123",
    }).
    Get("http://github.com")

Response

The httpclient.Response is a thin wrap of http.Response.

// traditional
res, err := httpclient.Get("http://google.com")
bodyBytes, err := ioutil.ReadAll(res.Body)
res.Body.Close()

// ToString
res, err = httpclient.Get("http://google.com")
bodyString, err := res.ToString()

// ReadAll
res, err = httpclient.Get("http://google.com")
bodyBytes, err := res.ReadAll()

Handle Cookies

url := "http://github.com"
httpclient.
    WithCookie(&http.Cookie{
        Name: "uid",
        Value: "123",
    }).
    Get(url)

for _, cookie := range httpclient.Cookies() {
    fmt.Println(cookie.Name, cookie.Value)
}

for k, v := range httpclient.CookieValues() {
    fmt.Println(k, v)
}

fmt.Println(httpclient.CookieValue("uid"))

Concurrent Safe

If you want to start many requests concurrently, remember to call the Begin method when you begin:

go func() {
    httpclient.
        Begin().
        WithHeader("Req-A", "a").
        Get("http://google.com")
}()
go func() {
    httpclient.
        Begin().
        WithHeader("Req-B", "b").
        Get("http://google.com")
}()

Error Checking

You can use httpclient.IsTimeoutError to check for timeout error:

res, err := httpclient.Get("http://google.com")
if httpclient.IsTimeoutError(err) {
    // do something
}

Full Example

See examples/main.go

Options

Available options as below:

  • OPT_FOLLOWLOCATION: TRUE to follow any "Location: " header that the server sends as part of the HTTP header. Default to true.
  • OPT_CONNECTTIMEOUT: The number of seconds or interval (with time.Duration) to wait while trying to connect. Use 0 to wait indefinitely.
  • OPT_CONNECTTIMEOUT_MS: The number of milliseconds to wait while trying to connect. Use 0 to wait indefinitely.
  • OPT_MAXREDIRS: The maximum amount of HTTP redirections to follow. Use this option alongside OPT_FOLLOWLOCATION.
  • OPT_PROXYTYPE: Specify the proxy type. Valid options are PROXY_HTTP, PROXY_SOCKS4, PROXY_SOCKS5, PROXY_SOCKS4A. Only PROXY_HTTP is supported currently.
  • OPT_TIMEOUT: The maximum number of seconds or interval (with time.Duration) to allow httpclient functions to execute.
  • OPT_TIMEOUT_MS: The maximum number of milliseconds to allow httpclient functions to execute.
  • OPT_COOKIEJAR: Set to true to enable the default cookiejar, or you can set to a http.CookieJar instance to use a customized jar. Default to true.
  • OPT_INTERFACE: TODO
  • OPT_PROXY: Proxy host and port(127.0.0.1:1080).
  • OPT_REFERER: The Referer header of the request.
  • OPT_USERAGENT: The User-Agent header of the request. Default to "go-httpclient v{{VERSION}}".
  • OPT_REDIRECT_POLICY: Function to check redirect.
  • OPT_PROXY_FUNC: Function to specify proxy.
  • OPT_UNSAFE_TLS: Set to true to disable TLS certificate checking.
  • OPT_DEBUG: Print request info.
  • OPT_CONTEXT: Set context.context (can be used to cancel request).
  • OPT_BEFORE_REQUEST_FUNC: Function to call before request is sent, option should be type func(*http.Client, *http.Request).

Seperate Clients

By using the httpclient.Get, httpclient.Post methods etc, you are using a default shared HTTP client.

If you need more than one client in a single programme. Just create and use them seperately.

c1 := httpclient.NewHttpClient().Defaults(httpclient.Map {
    httpclient.OPT_USERAGENT: "browser1",
})

c1.Get("http://google.com/")

c2 := httpclient.NewHttpClient().Defaults(httpclient.Map {
    httpclient.OPT_USERAGENT: "browser2",
})

c2.Get("http://google.com/")

go-httpclient's People

Contributors

aglyzov avatar artnez avatar bitwalker avatar brainm avatar daveyarwood avatar ddliu avatar freexploit avatar msaf1980 avatar schmich avatar wrj898 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

go-httpclient's Issues

What request parameters can be customized?

Are you able to set the following? If so is there a recommended way?

  • HTTP Version HTTP/0.9 , HTTP/1 , HTTP2
  • Host Header
  • Path

Also, does it support concurrent requests? I see you can create multiple requests but I want to create something similar to channels to handle a bunch of requests to separate hosts?

how set OPT_PROXY

hi i try to set OPT_PROXY.by this code
proxy:="127.0.0.1:8080"
cli:=httpclient.NewHttpClient()
cli.WithOption(httpclient.OPT_PROXYTYPE,httpclient.PROXY_HTTP)
cli.WithOption(httpclient.OPT_PROXY,proxy)

this not work...Please help me

post 怎么传数组?

为什么参数都是 map[string]string? 用 url.Values 不是更好吗? 这样,数组也没有问题了

请教个问题?

c.
WithHeader("User-Agent", "Super Robot").
WithHeader("custom-header", "value").
WithHeaders(map[string]string {
"another-header": "another-value",
"and-another-header": "another-value",
}).
WithOption(httpclent.OPT_TIMEOUT, 60).
WithCookie(&http.Cookie{
Name: "uid",
Value: "123",
}).
Get("http://github.com", nil)
上面添加的header和cookie是只对 Get("http://github.com", nil)这一次请求有效,还是对后面的请求都有效呢?

Add PatchJson as a convenience method

Just ran into having to resort to:

body, jsonerr := json.Marshal(map[string]interface{}{
...
})
...
result, err := httpclient.Do(
	"PATCH",
	uri,
	nil,
	bytes.NewReader(body),
)

instead of simply doing PatchJson just like PostJson.

对于defaultClient同时使用并发和非并发方式会导致panic

在一个工程里如果 pkg A 使用了非并发的方式,而 pkg B 使用了并发的方式,会导致panic。前提是都使用 defaultClient。

panic 信息:

fatal error: sync: unlock of unlocked mutex

goroutine 69 [running]:
runtime.throw(0x158603d, 0x1e)
        /usr/local/go/src/runtime/panic.go:774 +0x72 fp=0xc0002cd978 sp=0xc0002cd948 pc=0x1030fb2
sync.throw(0x158603d, 0x1e)
        /usr/local/go/src/runtime/panic.go:760 +0x35 fp=0xc0002cd998 sp=0xc0002cd978 pc=0x1030f35
sync.(*Mutex).unlockSlow(0xc0000ba2a8, 0xffffffffffffffff)
        /usr/local/go/src/sync/mutex.go:196 +0xd6 fp=0xc0002cd9c0 sp=0xc0002cd998 pc=0x106e0b6
sync.(*Mutex).Unlock(...)
        /usr/local/go/src/sync/mutex.go:190
github.com/ddliu/go-httpclient.(*HttpClient).reset(0x19718c0)
        /Users/liuteng/go/pkg/mod/github.com/ddliu/[email protected]/httpclient.go:480 +0x86 fp=0xc0002cd9e0 sp=0xc0002cd9c0 pc=0x132acf6
github.com/ddliu/go-httpclient.(*HttpClient).Do(0x19718c0, 0x157b146, 0x3, 0xc0001b2000, 0x9b, 0xc000160480, 0x0, 0x0, 0xd, 0x1932640, ...)
        /Users/liuteng/go/pkg/mod/github.com/ddliu/[email protected]/httpclient.go:584 +0x1a0 fp=0xc0002cdb38 sp=0xc0002cd9e0 pc=0x132b510

panic 原因:
withLock 为 true,非并发方式也会调用 Unlock 导致 panic。

// Reset the client state so that other requests can begin.
func (this *HttpClient) reset() {
	this.oneTimeOptions = nil
	this.oneTimeHeaders = nil
	this.oneTimeCookies = nil
	this.reuseTransport = true
	this.reuseJar = true

	// nil means the Begin has not been called, asume requests are not
	// concurrent.
	if this.withLock {
		this.lock.Unlock()
	}
}

即使在 Unlock 后将 withLock 置为 false,也不能解决问题,似乎只能将并发方式与非并发的 client 分开,或者每次都创建新的 client?

404的错误

如果访问一个不存在的链接,会不会导致没有body,导致错误呢
// Read response body into string.
func (this *Response) ToString() (string, error) {
defer this.Body.Close()
bytes, err := ioutil.ReadAll(this.Body)
if err != nil {
return "", err
}

return string(bytes), nil

}

I/o timeout

默认客户端设置了超时时间20秒,调用外部接口error返回超时,前后记录时间实际只花了40毫秒

InsecureSkipVerify

How to use InsecureSkipVerify transport option to prevent "x509: cannot validate certificate..." error?

socket: too many open files

using httpclient.Begin() in multithreading environment(some threads runing or sleeping long time, but we keep the total number of threads below the threshold, less than 30), but the socket open files is increasing all the time. using command ls -p {pid} | wc -l, we can see the number is increasing slow but never reducing。When we ran the program two days,socket open files reached 65539 then start to report errors “socket: too many open files”。

this may caused by reusing the Transport of golang http client, and this problem should be easy to reproduce。the code like this:

	for i := 0; i < w.ThreadNum; i++ {
		go w.doTest()
		time.Sleep(10 * time.Millisecond)
	}

cookie helper feature

can you add save cookie to file and load from file support~, it's useful when use cookie auth api.

添加header不成功

res, _ := c.
WithHeader("Accept-Language", "en-us").
WithCookie(&http.Cookie{
Name: "name",
Value: "github",
}).
WithHeader("Referer", "http://www.baidu.com").
Get(SERVER, nil)

fmt.Println(c.Headers)

添加header不成功啊,例子程序里面,你在最后打印c.headers 是空的

README.md

for _, cookie := range c.Cookies() {
fmt.Println(c.Name, c.Value)
}

fix:
for _, cookie := range c.Cookies() {
fmt.Println(cookie.Name, cookie.Value)
}

关于无法携带 TLS 证书这件事

目前似乎无法构造出携带 TLS 证书的请求。

原生 golang 代码类似这样:

func NewClient() *http.Client {
    certs, err := tls.LoadX509KeyPair("cert.pem", "key.pem")
    if err != nil {
        fmt.Println(err.Error())
        return nil
    }

    rootCa, err := ioutil.ReadFile("rootca.pem")
    if err != nil {
        fmt.Println(err.Error())
        return nil
    }

    pool := x509.NewCertPool()
    pool.AppendCertsFromPEM(rootCa)

    return &http.Client{
        Timeout: 15 * time.Second,
        Transport: &http.Transport{
            TLSClientConfig: &tls.Config{
                RootCAs:      pool,
                Certificates: []tls.Certificate{certs},
            },
        },
    }
}

Post throw panic when content-type is not kv

Post method transfer params by called toUrlValues,but the payload of the request is a string and content-type is text/plain,toUrlValues will throw panic.

What to do in above scenario?

panic stack:
panic: Invalid value [recovered]
panic: Invalid value

goroutine 89 [running]:
testing.tRunner.func1(0xc000336100)
/usr/local/go/src/testing/testing.go:874 +0x3a3
panic(0x143e500, 0x15664c0)
/usr/local/go/src/runtime/panic.go:679 +0x1b2
github.com/ddliu/go-httpclient.toUrlValues(0x143e500, 0xc000189e40, 0xc000189d38)
/Users/liuteng/go/pkg/mod/github.com/ddliu/[email protected]/util.go:160 +0x2bc
github.com/ddliu/go-httpclient.(*HttpClient).Post(0x184e9a0, 0xc0003ac060, 0x56, 0x143e500, 0xc000189e40, 0x1, 0x56, 0x104c3b7)
/Users/liuteng/go/pkg/mod/github.com/ddliu/[email protected]/httpclient.go:661 +0x3f

测试过协成了么?

go function testPostUrl();
当你的HTTP 来测试go 的一直httpcleint 请求的时候 。 就呵呵了!

defaultClient concurrency is not safe

package main
import (
	"github.com/ddliu/go-httpclient"
	"time"
)

func main() {
	for i := 0; i <20;i++ {
		go do()
	}

	time.Sleep(1*time.Second)
}

func do()  {
	httpclient.
		WithHeader("1","1").
		WithHeader("2","2").
		WithHeader("3","3").
		WithHeader("4","4").Get("google.com")
}

fatal error: concurrent map writes

goroutine 20 [running]:
runtime.throw(0x122da43, 0x15)
C:/Program Files/Go/src/runtime/panic.go:1117 +0x79 fp=0xc000055ed0 sp=0xc000055ea0 pc=0xf99419
runtime.mapassign_faststr(0x11e16c0, 0xc00001e030, 0x1228229, 0x1, 0x0)
C:/Program Files/Go/src/runtime/map_faststr.go:211 +0x411 fp=0xc000055f38 sp=0xc000055ed0 pc=0xf74e11
github.com/ddliu/go-httpclient.(*HttpClient).WithHeader(...)

keep-alive

how would you do a keep alive in concurrency environment?
I have a worker pool that request alltime the same url (with differend data).

hasOption

func hasOption(opt int, options []int) bool {
for _, v := range options {
if opt == v { //if opt != v { ????
return true
}
}

return false

}

Basic Auth utility feature

Hi,

This library reminds me of how much I love Python's requests. One thing that I miss is a utility to pass a (username, password) pair easily. Something around these lines:

httpclient.WithBasicAuth(username, password).Get(url)

Do you think it's worth implementing? I can submit a PR if you like it.

报告两个错误

c := httpclient.NewHttpClient(httpclient.Map {
"opt_useragent": USERAGENT,
"opt_timeout": TIMEOUT,
"Accept-Encoding": "gzip,deflate,sdch",
})
这样初始化后,
fmt.println(c.header["User-Agent"]) 打印没结果
c.WithHeader("Referer", "http://google.com")
fmt.Println(c.Headers["Referer"])
打印没结果的

deadlock when using Begin() with PostJson()

Problem

Refer to the doc, httpclient provides a concurrent-safe as https://github.com/ddliu/go-httpclient#concurrent-safe described,
However there are corner cases where deadlock exists when using httpclient.Begin().PostJson().

How to Reproduce

func Test_httpclient(t *testing.T) {
	type Node struct {
		Name string `json:"name"`
		Next *Node  `json:"next"`
	}
	n1 := Node{Name: "1", Next: nil}
	n2 := Node{Name: "2", Next: &n1}
	n1.Next = &n2
        // send an object that can't be marshalled.
	_, err := httpclient.Begin().PostJson("xxx", n2)
	if err != nil {
		fmt.Println(err.Error())
	}
        // block here as Begin() can not require the lock.
	httpclient.Begin().Get("xxx")
	fmt.Println("ok")
}

By reading the code further, there are also some other function returned without calling reset() to release the lock, such as

go-httpclient/httpclient.go

Lines 729 to 754 in d779327

func (this *HttpClient) PostMultipart(url string, params interface{}) (
*Response, error) {
body := &bytes.Buffer{}
writer := multipart.NewWriter(body)
paramsValues := toUrlValues(params)
// check files
for k, v := range paramsValues {
for _, vv := range v {
// is file
if k[0] == '@' {
err := addFormFile(writer, k[1:], vv)
if err != nil {
return nil, err
}
} else {
writer.WriteField(k, vv)
}
}
}
headers := make(map[string]string)
headers["Content-Type"] = writer.FormDataContentType()
err := writer.Close()
if err != nil {
return nil, err
.

Concurrency error

When i tried start multiple requests i had error unlock of unlocked mutex


import (
	"fmt"
	"github.com/ddliu/go-httpclient"
	"io/ioutil"
	"log"
	"time"
)
func main() {
	httpclient.Defaults(httpclient.Map {
		httpclient.OPT_CONNECTTIMEOUT: 10,
	})
	start := time.Now()
	//income := request("url1")
	const size = 10

	array := [size] chan response{}
	for i := 0; i < size; i++ {
		array[i] = request("https://url1")
	}

	income2 := request("https://url1")
	income3 := request("http:/url1")

	response2 := <-income2
	response3 := <-income3

	for i := 0; i < size; i++ {
		r := <-array[i]
		fmt.Printf("RESULT -- %v \n", r)
	}

	fmt.Printf("RESULT -- %v \n", response2)
	fmt.Printf("RESULT -- %v \n", response3)

	timeTrack(start, "execution")
}
func request(url string) chan response {

	out := make(chan response)

	go func() {
		res, err := httpclient.Begin().Get(url)
		if err != nil {
			fmt.Println(err)
			result := response{0, nil, &err}

			out <- result
		} else {
			fmt.Println(res.StatusCode, err)

			bodyBytes, _ := ioutil.ReadAll(res.Body)

			body := string(bodyBytes)

			out <- response{res.StatusCode, &body, nil}
		}

	}()
	return out
}

func timeTrack(start time.Time, name string) {
	elapsed := time.Since(start)
	log.Printf("%s took %s", name, elapsed)
}
type response struct {
	StatusCode int
	Body *string
	Err *error
}

Stack trace is

fatal error: sync: unlock of unlocked mutex
fatal error: sync: inconsistent mutex state
goroutine 36 [running]:
runtime.throw(0x12eb01a, 0x1e)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.