go-resty/resty

Simple HTTP and REST client library for Go

resty's Issues

Convenient Query string (SetQueryString) support for Request

Currently go-resty supports query string via SetQueryParam and SetQueryParams at Client and Request. However I believe in-addition to those methods. It would be better to have more comfortable experience with SetQueryString in the Request level.

resp1, err1 := resty.R().
            SetQueryString("Param1=Param1Vaule&Param2=Param2 Value&Param3=Param3 Value").
            Get("http://httpbin.org/get")

example of automatic unmarshalling?

example of automatic unmarshalling?

Add support for automatic json unmarshalling using map on response

Can you please bring automatic JSON unmarshalling using map on response like struct?

I'm trying to do something like the following:

resp, err := resty.R().SetResult(map[string]interface{}{}).Get(url)
result := resp.Result().(*map[string]interface{})
fmt.Printf("%+v", result["message"])

Support socks5 proxies

Seeing resty allows proxies to be set by URL which involves parsing and identifying its scheme, most people would expect that it supports other proxy schemes such as socks5. However, when I try to set socks5 proxy, It connects to the socks5 proxy host by using a http proxy model.

The reason why I need this is typically Tor listens only for socks5 connections. I know of no project using golang that currently supports socks5 proxy, but is there any way this could be built into the resty?

Make Request.execute public

There are cases where we have to programmatically decide which http method to use when invoking an API. While we can create a map of handler functions to call the various put/post/get/delete/options functions on a Request, it would be simpler if we could just pass the desired method to Execute directly, i.e:

resp, err := resty.R().Execute(resty.POST, url)

vs what I have to do right now:

type handler func(*resty.Request, string) (*resty.Response, error)

func get(r *resty.Request, url string) (*resty.Response, error) {
  return r.Get(url)
}

func post(r *resty.Request, url string) (*resty.Response, error) {
  return r.Post(url)
}

handlers := make(map[string]handler)
handlers[resty.GET] = get
handlers[resty.POST] = post
// etc

resp, err := handlers[resty.POST](resty.R(), url)

Enable proper retrying of POST requests

It looks like retries on POST requests do not resend the request body. Some would suggest that POSTs should not be retried, but in my case the request is idempotent, so it's fine. It should be noted that the log indicates the retried attempt contains the post body, but it's not actually there.

redirect with cookies and headers

like other http libs for golang
when redirect cookies disappear

Implement Conditional Retry from PR#37

Implement @dmiller113 contribution from PR #37 into resty main stream.

Allow file uploads with io.Reader and a name

Hey, first of all: thanks for this wonderful library! I've been using it for quite some time now, and it works like a charm.

There's one thing I'm missing, though: being able to upload files (as multipart) using an io.Reader and a name for the file. The scenario is, for example, something like this:

I have a database of many small image files which are stored as BLOBs along with their original file name (or file type). I want to upload some of them to some endpoints, I know the server will check the extension of the file name to determine if I really send images. It'd be great if I could directly "pipe" the image from the DB along with its name (or a made up name) into resty. It's a pain to create temporary files for every upload I need (imagine the DB running on another machine)...

Other scenarios might include receiving files over the network - I don't want to store a copy locally, I just want to pipe them through.

Would it be possible to get that? I'm thinking of something like

func (r *Request) SetFileReader(param, fileName string, reader io.Reader) *Request

What do you think?

Add IsProxySet() method

Add IsProxySet() method, it's help find out client instance has proxy set.

Add custom timeout per client basis + FastHTTP support

So I'm looking for something like

myresty := resty.New()
req, err := myresty.R().
            SetHeader("User-Agent", UserAgent).
            SetProxy(proxyUrl).
            SetTimeout(duration)
            Get(link)

Also wanted to ask if it's possible to use fasthttp instead of net/http. From what I read it is significantly more performant:

Refer: https://medium.com/@valyala/net-http-client-has-the-following-additional-limitations-318ac870ce9d#.502tduhmn

Refer: https://medium.com/@nate510/don-t-use-go-s-default-http-client-4804cb19f779#.pla7aw7gz

Thanks!

Add support for io.Reader in SetBody

Enhancement idea brought from #21 for adding io.Reader support in resty.R().SetBody().

Add an option to support meta redirection

This is not a standard yet but it'd be great if Resty support redirection using meta refresh such as:

<meta http-equiv="refresh" content="0; url=/index.jsp">

resty.FollowMetaRedirects(true)

Most websites follow the HTTP standard, which contains declarations for standard redirection method, but some don't. However, this is not so important because it's often not the case and can cause performance issues.

Bring proxy ability for resty

Proxy support underlying resty client transport

SetQueryParams supports map[string]interface{} maybe?

Something like this...

resty.R().
    SetQueryParams(map[string]interface{}{
        "name": "xxx",
        "age": 10,
    })

also
SetQueryParam(param string, value interface{})

Disable warnings possible?

I get this continously:

RESTY 2016/04/13 16:28:18 WARNING - Using Basic Auth in HTTP mode is not secure.

It would be nice if it was (perhaps it already is!) possible to disable warnings.

Thank you

for such wonderfully commented code. Feel free to close this :)

Context support

It would be nice to have a Request.SetContext method to set context for request cancellation like it is done in httpDo function described in https://blog.golang.org/context.

Proposal: Response Body default JSON Unmarshal to provided golang struct or map

Appreciate the default behavior of Request SetBody(interface{}) to "Automatic marshalling for JSON and XML content type, if it is struct or map." Would suggest the same automatic unmarshalling for [Reponse Body()](https://godoc.org/github.com/go-resty/resty#Response.Body() when specified with an interface{} argument.

Add HTTP2 support for resty

Add HTTP2 support for resty :)

stop redirect

can resty stop redirect without getting error ?! i tried custom redirectpolicy but still nil continue the redirect and error panic and not returning with response

Support custom Transport in the go-resty

Bring support for Transport in the go-resty

resty.SetTransport(...)

Save http response into file - similar to curl -o flag

Bring ability to save HTTP response into file.

Client Level feature:

• Setting Directory path for saving response

Request Level feature:

• Setting output filename for request. Absolute and Relative path option.
  • For relative path option, if output directory path is not set at client level then default 'go-resty' directory will be created at current working directory

Bulk Request feature

Bring bulk request feature as a batteries for go-resty. For example: very useful for Upload/Download, batch processing, etc.

DATA RACE in resty.(*Client).execute()

Hello,
I am use resty in multi-goroutine and case the DATA RACE

test.go
`package main

import (
"github.com/go-resty/resty"
"sync"
)

func main() {
w := sync.WaitGroup{}
for i := 0; i < 50; i++ {
w.Add(1)
go func() {
defer w.Done()
resty.R().Get("http://httpbin.org/get")
}()
}
w.Wait()
} go run -race test.go`

Persist Cookies into file in JSON format

Ability to write cookies for given url into file (may be JSON or XML) from cookiejar.

httpClient.Do is protected by a mutex?

Because of the locking, there can only be one outgoing call, which is really bad to the performance. If I understand it correctly,

• if req1 has proxyURL set but req2 doesn't, then req2 will be sent with req1's proxy. Is this expected?
• if req1 and req2 both have proxyURL set, then req1 will be sent with req1's proxy and req2 will be sent with req2's proxy. This works. But the overhead of this is to not have more than 1 outgoing calls.
• if Unlock() is moved up before httpClient.Do. For the above case, there is no guarantee on which proxy will be used, which is not good.

I hope this per request proxy is an optional feature. Providing it is fine but without sacrificing the normal use case.

Code ref:
https://github.com/go-resty/resty/blob/master/client.go#L634-L647

	c.mutex.Lock()

	if req.proxyURL != nil {
		c.transport.Proxy = http.ProxyURL(req.proxyURL)
	} else if c.proxyURL != nil {
		c.transport.Proxy = http.ProxyURL(c.proxyURL)
	}

	req.Time = time.Now()
	c.httpClient.Transport = c.transport

	resp, err := c.httpClient.Do(req.RawRequest)

        c.mutex.Unlock()

Nil Pointer access on timeout using retry

When a request interrupts due to a client timeout and when using a retry condition function then the underlying resty.Response.RawResponse is nil. Which is pretty obvious because the remote haven't sent any. But one has to make a nil check on the RawResponse in RetryConditionFuncs because functions like resty.Response.StatusCode() access directly the underlying RawResponses fields.

SetMultiValueFormData

Hello,

It would be nice to have SetMultiValueFormData which should work in the same way as SetMultiValueQueryParams but for FormData:

q, err := query.Values(struct {}{})

for p, v := range q {
    for _, pv := range v {
        client.FormData.Add(p, pv)
    }
}

q, err := query.Values(struct {}{})

client.SetMultiValueFormData(q)

Bring RequestBody support for DELETE request

As per latest RFC 7231 explicitly permits entity body/payload in DELETE request and RFC 2616 become obsolete.

Planning to support it!

OnBeforeRequest does not change Request.RawRequest attributes

Trying to manipulate *resty.Request fields inside an OnBeforeRequest function does not affect the actual request, which is finally executed:

https://github.com/go-resty/resty/blob/master/client.go#L627-L645

for _, f := range c.beforeRequest {
		err = f(c, req)
		if err != nil {
			return nil, err
		}
	}

	c.mutex.Lock()

	if req.proxyURL != nil {
		c.transport.Proxy = http.ProxyURL(req.proxyURL)
	} else if c.proxyURL != nil {
		c.transport.Proxy = http.ProxyURL(c.proxyURL)
	}

	req.Time = time.Now()
	c.httpClient.Transport = c.transport

	resp, err := c.httpClient.Do(req.RawRequest)

All OnBeforeRequest functions will be executed but they have no effect when they call methods directly on *resty.Request because the RawRequest stays untouched until real request execution.

For example setting an auth token header inside an OnBeforeRequest function won't work with

	httpClient.OnBeforeRequest(func(c *resty.Client, r *resty.Request) error {
		r.SetAuthToken("bla")
                return nil
        })

As a workaround one has to manipulate directly the r.RawRequest struct.

Implement extensible and multiple redirect policies capability

• RedirectPolicy and RedirectPolicyFunc approach for extensible redirect policy definition. User can define N number of policies as they need
• Design and structure SetRedirectPolicy method, so it can accept multiple redirect policies as variadic arguments
• Bring new policy called 'Host check' redirect policy

Add following features described in the description for TLS client config

Improving TLS client config via easy to use methods

• Loading client root PEM
• Loading client certificate as cert input

apply golint, errchk, go-metalinter code quality

Only breaking change is HTTP methods-

• GET => MethodGet
• POST => MethodPost
• PUT => MethodPut
• DELETE => MethodDelete
• PATCH => MethodPatch
• HEAD => MethodHead
• OPTIONS => MethodOptions

Lost repsonse and no error return

Someone api will cost N ms.

If we SetTimeout M ms (N< M < 2N) to Client and we call this api twice, the server side will receive three request and send three response.

In client side, user only see 2 api call but 3 api call be sent actually.

This issue will be duplicated in go 1.7.3, here is the code.

I found a bad implement of SetTimeout and fix it.

Confusion between `set` and `add` methods

Hi,

I noticed an unexpected behavior for most of the set methods. This is from request.go:

func (r *Request) SetFormData(data map[string]string) *Request {
	for k, v := range data {
		r.FormData.Add(k, v)
	}
	return r
}

Most of the set methods are actually add which is misleading and prevents from overriding existing keys.
I don't know whether it is a design choice or not.

My suggestion would be to have both set and add methods (I have the changes locally). It would make things cleaner imo.

However, I am also aware that for functions such as SetQueryString it might be possible to have multiple times the same key.
It would be possible for the set functions to make them smarter and in case of multiple occurrences of one key, to first call set (which would erase the previously stored key) then call add.
Though I am not sure it would really be better.

Also, changing the current set methods to internally use a set call instead of an add call will break compatibility with code using the current behavior.

What is your opinion on the topic ?

Unable to set custom `http.RoundTripper`

At the moment it is not possible to set custom http.RoundTripper as a transport in resty since its transport field wants exactly http.Transport struct. On the contrary, golang's stdlib client allows us to use any RoundTripper - https://golang.org/src/net/http/client.go?s=1933:4055#L46

Providing custom RoundTripper can be usable in testing, i.e. one could hijack transport to be provided by some mocking library (https://github.com/dnaeon/go-vcr) or when there is a need to use custom transport for whatever reasons like connecting to weird places with weird protocols or using transport with more features than golang's one provides (https://github.com/pkulak/simpletransport).

Resty is currently relying on http.Transport to provide a way for controlling proxies, tls and timeouts.

We could use type assertion in those places to check if RoundTripper we have is an http.Transport and do nothing in case it is not (or returning error). We could also mention it in the documentation that those things are controlled by underlying transport so SetProxy, SetTimeout and SetTLSClientConfig won't function in case custom non http.Transport transport used.

I can provide a pull request if you find the idea to be valid.

API streamlining (Minor breaking changes)

For API consistency and progressing toward v1.0 goal, following changes taking place in v0.5-

• Response.Body turns to Response.Body() method
• Response.ReceivedAt turns to Response.ReceivedAt() method

Form Data Does Not Allow Multi-Valued Parameters

SetFormData currently only accepts map[string]string but the HTTP form data really supports a model of map[string][]string (a map of string to list-of-strings).

No DNS failover possible with resty

Hi!

I have a daemon, which polls some API endpoints. If the API endpoint switch traffic with DNS (changing the IP), then resty will not do a DNS lookup to get the new IP.
I use the default client use the following code:

urlString := fmt.Sprintf("https://api.endpoint.foo.com/events/%v", ID) // DNS name changed
resp, err := resty.R().
    SetHeader("Content-Type", "application/json").
    SetBody(body).
    SetAuthToken(cli.AccessToken).
    Put(urlString)

Maybe I can get resty to lookup DNS somehow, but I did not find it, yet.

Don't set the transport's proxy settings to nil by default

In client.go, at this location, there's this code:

    if req.proxyURL != nil {
        c.transport.Proxy = http.ProxyURL(req.proxyURL)
    } else if c.proxyURL != nil {
        c.transport.Proxy = http.ProxyURL(c.proxyURL)
    } else {
        c.transport.Proxy = nil
    }

The final else clause should be removed. If the caller provides a transport, it may already have a proxy configured, there's no reason to set the transport's proxy to nil.

Add per request Proxy Support

Add per request proxy support, preserving client level proxy setting value.

Add support for Request Retry

It would be nice to have request try with interval, no of attempts, etc.

SetBody() does not support array of models

Hi
I am a team member of Swagger CodeGen, we use the resty API, we recently ran into issue when setting body with an array of models, is it a known issue that resty does not allow passing array of objects in the body? below is the test exception:

--- FAIL: TestCreateUsersWithArrayInput (0.00s)
panic: interface conversion: interface is *[]petstore.User, not []uint8 [recovered]
    panic: interface conversion: interface is *[]petstore.User, not []uint8

goroutine 18 [running]:
testing.tRunner.func1(0xc8200f6090)
    /usr/local/go/src/testing/testing.go:450 +0x171
github.com/go-resty/resty.DetectContentType(0x2e9320, 0xc8200fe140, 0x0, 0x0)
    /Users/xx/go/src/github.com/go-resty/resty/client.go:780 +0xd6
github.com/go-resty/resty.handleRequestBody(0xc8200bc000, 0xc82010a000, 0x0, 0x0)
    /Users/xx/go/src/github.com/go-resty/resty/middleware.go:295 +0xc4
github.com/go-resty/resty.parseRequestBody(0xc8200bc000, 0xc82010a000, 0x0, 0x0)
    /Users/xx/go/src/github.com/go-resty/resty/middleware.go:107 +0x245
github.com/go-resty/resty.(*Client).execute(0xc8200bc000, 0xc82010a000, 0x0, 0x0, 0x0)
    /Users/xx/go/src/github.com/go-resty/resty/client.go:579 +0xc7
github.com/go-resty/resty.(*Request).Execute(0xc82010a000, 0x40ab28, 0x4, 0xc8200f2240, 0x32, 0x4, 0x0, 0x0)
    /Users/xx/go/src/github.com/go-resty/resty/request.go:394 +0x105
github.com/go-resty/resty.(*Request).Post(0xc82010a000, 0xc8200f2240, 0x32, 0x4

FYI, with the same code, single object works perfectly.

Add support for WaitTime and MaxWaitTime for retries

Currently client does not expose a way to configure wait times for retry, only conditions and retry count are configurable -

resty/request.go

Line 430 in 90b37fd

}, Retries(r.client.RetryCount), RetryConditions(r.client.RetryConditions))

We need to make wait times configurable as well, default 100ms wait time is too little in my case.

I can provide a pull request for this if the issue is valid

Support trailing backslashes in URLs

Many REST APIs still include different behavior for addresses with a trailing slash vs without. For example:

GET /api/v1/dog

might possibly be different from

GET /api/v1/dog/

Current behavior is to trim the trailing backslash. I propose to include trailing backslashes and leave it to the user to specify the verbose path.

Resty not binding for a non 200 response

Hi,

Thanks for the awesome library!

I have a struct that I'm filling with a JSON response from an API endpoint using SetResult. When the endpoint returns a 200 (success status), this struct is filled in just fine. However, when it returns say a 4XX or 5XX error message this struct is empty even though the response body itself has proper JSON.

Add an option for user-defined middleware after the built-in middlware, to modify the RawRequest object

I wanted to use resty for requests to an AWS endpoint that requires requests to be signed with AWS credentials. A library already exists for this (https://github.com/smartystreets/go-aws-auth), however using that one requires access to the http.Request object directly. Currently, user defined middleware is run before the built-in middleware, which makes sense when modifying the resty.Request object. However for resty to be compatible with go-aws-auth, I need middleware that is processed after the built-in middleware, so that the RawRequest object is available.

Support pointer and non-pointer for SetResult and SetError methods

SetResult:

// Note: Result object can be pointer or non-pointer.
resty.R().SetResult(&AuthToken{})
// OR
resty.R().SetResult(AuthToken{})

// Accessing a result value
response.Result().(*AuthToken)

SetError:

// Note: Error object can be pointer or non-pointer.
resty.R().SetError(&AuthError{})
// OR
resty.R().SetError(AuthError{})

// Accessing a error value
response.Error().(*AuthError)

SetQueryParams does not support "Multi" inputs

Does Resty support Multiple params? I am passing any array of statuses for query param, i am expecting the querystring is
?status=pending&status=approved, but what i got from the debugger is only appending the first value,(status=pending)
here is my code

    var status = []string {"pending","approved"}
    for _, value := range status {
        resty.R().SetQueryParam("status",value)
    }

fyi, i also tried SetQueryParams, but since it is a dictionary, you can't have multiple values, so it won't work either. I know the work around is converting status array to a comma separated strings, but I prefer to set multiple parameters.

are you going to add this feature in the near future?

cc @wing328