levigross/grequests

A Go "clone" of the great and famous Requests library

GRequests

A Go "clone" of the great and famous Requests library

License

GRequests is licensed under the Apache License, Version 2.0. See LICENSE for the full license text

Features

• Responses can be serialized into JSON and XML
• Easy file uploads
• Easy file downloads
• Support for the following HTTP verbs GET, HEAD, POST, PUT, DELETE, PATCH, OPTIONS

Install

go get -u github.com/levigross/grequests

Usage

import "github.com/levigross/grequests"

Basic Examples

Basic GET request:

resp, err := grequests.Get("http://httpbin.org/get", nil)
// You can modify the request by passing an optional RequestOptions struct

if err != nil {
	log.Fatalln("Unable to make request: ", err)
}

fmt.Println(resp.String())
// {
//   "args": {},
//   "headers": {
//     "Accept": "*/*",
//     "Host": "httpbin.org",

If an error occurs all of the other properties and methods of a Response will be nil

Quirks

Request Quirks

When passing parameters to be added to a URL, if the URL has existing parameters that contradict with what has been passed within Params – Params will be the "source of authority" and overwrite the contradicting URL parameter.

Lets see how it works...

ro := &RequestOptions{
	Params: map[string]string{"Hello": "Goodbye"},
}
Get("http://httpbin.org/get?Hello=World", ro)
// The URL is now http://httpbin.org/get?Hello=Goodbye

Response Quirks

Order matters! This is because grequests.Response is implemented as an io.ReadCloser which proxies the http.Response.Body io.ReadCloser interface. It also includes an internal buffer for use in Response.String() and Response.Bytes().

Here are a list of methods that consume the http.Response.Body io.ReadCloser interface.

• Response.JSON
• Response.XML
• Response.DownloadToFile
• Response.Close
• Response.Read

The following methods make use of an internal byte buffer

• Response.String
• Response.Bytes

In the code below, once the file is downloaded – the Response struct no longer has access to the request bytes

response := Get("http://some-wonderful-file.txt", nil)

if err := response.DownloadToFile("randomFile"); err != nil {
	log.Println("Unable to download file: ", err)
}

// At this point the .String and .Bytes method will return empty responses

response.Bytes() == nil // true
response.String() == "" // true

But if we were to call response.Bytes() or response.String() first, every operation will succeed until the internal buffer is cleared:

response := Get("http://some-wonderful-file.txt", nil)

// This call to .Bytes caches the request bytes in an internal byte buffer – which can be used again and again until it is cleared
response.Bytes() == `file-bytes`
response.String() == "file-string"

// This will work because it will use the internal byte buffer
if err := resp.DownloadToFile("randomFile"); err != nil {
	log.Println("Unable to download file: ", err)
}

// Now if we clear the internal buffer....
response.ClearInternalBuffer()

// At this point the .String and .Bytes method will return empty responses

response.Bytes() == nil // true
response.String() == "" // true

grequests's Issues

whole request cycle timeout

feature request

Sorry for the inconvenience!

Clean up sessions

grequests how to disable Redirect

I thought it would be like this.

grequests.RequestOptions{
	RedirectLimit: -1,
}

But this code does not disable redirect,please tell me how to disable redirect, thanks~~~

Add net/context support

Https? [Help Wanted]

Is there any way to send requests with https instead of http or is that generally "impossible" in go?

fasthttp instead of net/http

Hi, this is a question.

I wanted to ask if it's possible to use fasthttp instead of net/http. From what I read it is significantly more performant:

Refer: https://medium.com/@valyala/net-http-client-has-the-following-additional-limitations-318ac870ce9d#.502tduhmn

Refer: https://medium.com/@nate510/don-t-use-go-s-default-http-client-4804cb19f779#.pla7aw7gz

Thanks for your package, been using it daily at work.

Make doRegularRequest public

Requests allows you to create requests using the request function, instead of having to call the wrapper for the http method. I am trying to achieve the same functionality in grequests, but doRegularRequest is private.

Response not getting cached when using .Bytes() or .String() methods

I read the response quirks on the readme but have so far been unable to get a struct populated using the response.JSON() method.

I'm using the latest version of grequests with Go 1.11. Here is the chunk of code in question:

        resp, _ := session.Post(loginapi, ro)

	fmt.Println(resp.Bytes())
	fmt.Println(resp.String()) // S1
	fmt.Println(resp.StatusCode)

	var r = new(APIResponse)

	fmt.Println(resp.JSON(&r))
	fmt.Println(resp.String()) // S2

In this scenario, S1 will print the response string and S2 will be empty. If I remove the resp.JSON() line everything works fine. Even if I try to call resp.JSON() first to populate before any other response method calls, the struct comes back nil.

UserAgent and other headers ignored after first redirect

Look like problem in doFollowingRedirects()

...
        if redirect != 0 {
            nreq := new(Request)
            nreq.Method = ireq.Method
            if ireq.Method == "POST" || ireq.Method == "PUT" {
                nreq.Method = "GET"
            }
            nreq.Header = make(Header)
            nreq.URL, err = base.Parse(urlStr)
            if err != nil {
                break
            }
            if len(via) > 0 {
                // Add the Referer header.
                lastReq := via[len(via)-1]
                if ref := refererForURL(lastReq.URL, nreq.URL); ref != "" {
                    nreq.Header.Set("Referer", ref)
                }
...

There is only Referer header is set.

Session requests not passing session request options

Hello!

When implementing a library/app in another language, it's usually called a port, not a clone!

UserAgent setting is being ignored in CONNECT requests.

I am trying to get an URL through a proxy and with a custom user agent:

    ro := &grequests.RequestOptions{
      RequestTimeout: s.httpTimeout,
      UserAgent:      uaString, 
      Proxies: map[string]*url.URL{
        "http":  proxyOne,  
        "https": proxyTwo, 
      },  
    }
    resp, err := grequests.Get("http://someurl.com", ro)

But if I look at a network traffic dump - I see two types of requests:

1. GET requests with my custom UA.
2. CONNECT requests with Go-http-client/1.1 UA.

Could you please fix UA handling in CONNECT calls? Thanks.

proposal:use custom cookieJar implementation

now grequest use the official implmentation of cookieJar (net/http/cookiejar)

I think this one lacks some functions, such as exporting/importing all cookies of a session.

But it's no way to use custom cookiejar at the moment.

maybe it's better to use our own cookiejar implementation to add some features, or allow user to override the default ones.

Need docs to claim that all timeouts will not change during sessions

some options in request options, such as all timeout settings, is actually bind to http.Client.
in doRegularRequest, we create client each time if timeout is not zero.
but in doSessionRequest, the client is created in NewSession().
so if user pass a RequestOptions with timeout to session.Get(), it will not change client's timeout.

I think current timeout implementation is good,but it's better to add some docs about this behavior.

EOF happens when sending a lot fo requests

when i send a lot of requests, EOF happens. how could i close old connections before the results received by new connection?

Proxy Example

Can you please give me an example of how to set a proxy for use with grequests?

Thank you.

how to set proxy after session created?

session := grequests.NewSession(nil)        // session.HTTPClient.Transport initialized
session.RequestOptions.Proxies = proxie  // here will not rebuild HTTPClient.Transport, so proxy does not work

Support of SOCKS proxies

Hello, is there a SOCKS-proxy support? I see that I can use HTTP/S proxies. When will you implement the support of SOCKS4/5?

grequests.addRedirectFunctionality() data race

golang version

go1.11.1

code

package main

import (
	"github.com/levigross/grequests"
	"sync"
)

func main(){
	wg := new(sync.WaitGroup)
	for i:=0;i <= 100;i ++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			grequests.Get("www.baidu.com", &grequests.RequestOptions{
				Params: map[string]string{
					"helllo": "world",
				},
			})
		}()
	}

	wg.Wait()
	return

}

go run --race main.go shows:

WARNING: DATA RACE

Read at 0x000000a2b0b0 by goroutine 9:
github.com/levigross/grequests.addRedirectFunctionality()
C:/Users/xxx/go/src/github.com/levigross/grequests/utils.go:55 +0x4e

Previous write at 0x000000a2b0b0 by goroutine 6:
github.com/levigross/grequests.addRedirectFunctionality()
C:/Users/xxx/go/src/github.com/levigross/grequests/utils.go:58 +0xc7

Check this out please!

Query string not being constructed

When sending a GET, with Data map[string]string set in the RequestOptions{}, the documentation in the package states following:

	// Data is a map of key values that will eventually convert into the
	// query string of a GET request or the body of a POST request.

Using .Get() on the *Session and specifying the options in Data, the content-type in the http header is set to Content-Type:[application/x-www-form-urlencoded]. This causes the request to be interpreted as a POST. Either the code needs to be changed, or the documentation updated to state that Data should be used for POSTs and QueryStruct should be used for GETs.

Repeated addition of HTTP headers

opt  := &grequests.RequestOptions{}
	proxyURL, err := url.Parse("http://127.0.0.1:8080")
	//body := strings.NewReader("username=admin&password=password&Login=Login")

	opt.Headers = map[string]string{
		"test":"test",
	}

	opt.Proxies = map[string]*url.URL{proxyURL.Scheme: proxyURL}

	//opt.RequestBody = body
	opt.UseCookieJar = true
	opt.DisableCompression = false
	opt.Data =  map[string]string{"username":"admin","password":"password","Login":"Login"}

	resp,err := grequests.Post ("http://192.168.95.4/aaa/login.php",opt)
	fmt.Println(err)
	//fmt.Println(resp.String())
	fmt.Println(resp.StatusCode)

After login, the corresponding 302 status code of the web server，When grequests continues to make requests, the “test” header in the HTTP request header repeats
http requests

GET /aaa/index.php HTTP/1.1
Host: 192.168.95.4
User-Agent: GRequests/0.10
Content-Type: application/x-www-form-urlencoded
Content-Type: application/x-www-form-urlencoded
Cookie: security=high; PHPSESSID=90949a1f57fa20a9e948f589e17a6786
Referer: http://192.168.95.4/aaa/login.php
Test: test
Test: test
Accept-Encoding: gzip, deflate
Connection: close

The reason is that the function "addRedirectFunctionality" in the utils. go file uses req. Header. Add (k, v) ，
Modified to req. Header.Set (k, v) resolves the problem of duplication of HTTP request headers

My English is very poor

`too many open files` error

resp, err := grequests.Post(url, options)
if err != nil || resp.StatusCode != 200 {
    ...
    return
}

above is my code, and after about 2 weeks, I could get an error as socket: too many open files. I don't know whether it due to the response unclosed. resp only used to get StatusCode, no resp.JSON or any other usages.

Should i must close the connection? like defer resp.Close(). Or what should I do to avoid this socket error?

http proxy setting does not work when try to access websites with https

        proxyURL, err := url.Parse("http://127.0.0.1:8080") // Proxy URL
	if err != nil {
		log.Panicln(err)
	}

	resp, err := grequests.Get("http://www.google.com",
		&grequests.RequestOptions{Proxies: map[string]*url.URL{proxyURL.Scheme: proxyURL}})

	if err != nil {
		log.Println(err)
	}

	if resp.Ok != true {
		log.Println("Request did not return OK")
	}

	log.Println(resp)

The code above will work as expected.

But, if you replace http://www.google.com to https://www.google.com, it will not work, it access google directly without the proxy you set!

How to NOT follow redirection

In python, I can do this
requests.get('http://github.com', allow_redirects=False)
this call returns immediately without follow any redirection
How can I do this in Go? I tried to set RedirectLimit to -1 in RequestOptions but its not worked

Why can't i add Transfer-Encoding to header

	body := strings.NewReader(`xxxxxxxx`)
	opt := grequests.RequestOptions{
		Headers: map[string]string{
			"Content-Type":"multipart/form-data; boundary=xxoo",
			"Transfer-Encoding":"chunked",
		},
		RequestBody:body,
	}

The content of the request initiated by grequests is as follows

POST / HTTP/1.1
Host: xxxx.com
User-Agent: GRequests/0.10
Content-Length: 284
Content-Type: multipart/form-data; boundary=xxoo
Cookie: PHPSESSID=gjj4afltersilid2qhq7f9fcl1
Accept-Encoding: gzip, deflate
Connection: close

xxoo

Transfer-Encoding not found in the request

My question is how to add "Transfer-Encoding" to the header.

Add a finalizer to the response to ensure that it gets closed and GCed

Make use of url.Values and autoconvert the current type to it

how post a byte date ?

such as post /api
but post data is text , not key=value ,i cannot use response.Data beacuse this is map[string]interface{}.

proxy doesn't work as parameter in session request

working well as follow:

var ourl = "http://baidu.com"
prox,_ := url.Parse("http://219.135.164.245:3128")
fmt.Println("prox:", prox)

session := grequests.NewSession(&grequests.RequestOptions{
	Proxies:map[string]*url.URL{prox.Scheme:prox},
})
rsp,err := session.Get(ourl, nil)

fmt.Println(err, rsp.String())

not working as follow:

var ourl = "http://baidu.com"
prox,_ := url.Parse("http://219.135.164.245:3128")
fmt.Println("prox:", prox)

session := grequests.NewSession(nil)
rsp,err := session.Get(ourl, &grequests.RequestOptions{
	Proxies:map[string]*url.URL{prox.Scheme:prox},
})

fmt.Println(err, rsp.String())

how can I mock http service with h2non/gock and levigross/grequests

Hi,

how can I mock http service with h2non/gock and levigross/grequests?

Best regards,
Stéphane

Why does not the Req method of Session be provided?

This project is convenient for simply requesting and response handling, but it's tricky if you want to hook the request will be sent or the response we received.

Here are my suggestions:

• Add a Req method for Session
• Export buildRequest and send request in other place, like requests in Python
• Export buildResponse for external response packging

Release a newer version of lib

I am considering using this library in a new project at my company, and have made a generic wrapper over this library using the Req method of the library, which is committed in master but unavailable when installed using glide. I have already submitted the test cases for the same and raised a PR for it.

Enable HTTP request tracing

duplicate query keys not possible with GET requests

Firstly, great package! Thanks for making this.

It seems that it's not possible to create a GET request, with parameters such as "http://localhost/file?tags=a&tags=b" since Params is a map.

See here:

ro := &RequestOptions{
	Params: map[string]string{"Hello": "Goodbye"},
}
Get("http://httpbin.org/get?Hello=World", ro)

would you accept a pull request to fix this?

Thanks,
Greg

Question: override DNS resolution?

For healthchecking, it's often useful to override the DNS provided IP of a URL, so that you can query a specific backend, while leaving things like SNI headers, etc intact.

Is there a way to do this with grequests?

error handling

Is there a reason why this library doesn't provide the "go way" of error handling ?

resp := grequests.Get("http://httpbin.org/xml", nil)

if resp.Error != nil {
    log.Fatalln("Unable to make request", resp.Error)
}

VS

resp, err := grequests.Get("http://httpbin.org/xml", nil)

if err != nil {
    log.Fatalln("Unable to make request", err)
}

RequestOptions cookie improvement

Please rework RequestOptions to take
[]*http.Cookie
instead of
[]http.Cookie
because http.Response.Cookies() returns []*http.Cookie

Grequests should not replace empty UserAgent with it's own

As much as I like Grequests and I truly want more people to find about it, forcing default User-Agent is not the way.

grequests/request.go

Line 558 in fd07961

req.Header.Set("User-Agent", localUserAgent)

Many platform will change behavior based on the User-Agent. Some of their WAF (web application firewall) will try to single out requests originating from the same IP, User-Agent, etc. There are situation where an empty User-Agent needs to be sent, literally User-Agent: and Grequests is preventing that from happening.

So I would suggest two solutions.

1. Create custom struct that can be set to empty, e.g.

// NullString represents a string that may be null.
type NullString struct {
	String string
	Valid  bool // Valid is true if String is not NULL
}

and if Valid is true then use the String provided in the struct.

2. Remove the check.

Session didn't get Cookies

s := grequests.NewSession(&grequests.RequestOptions{
		Headers: map[string]string{
			"Host":                      "202.192.18.32:8080",
			"User-Agent":                "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:51.0) Gecko/20100101 Firefox/51.0",
			"Accept":                    "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
			"Accept-Language":           "zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3",
			"Accept-Encoding":           "gzip, deflate",
			"Connection":                "keep-alive",
			"Upgrade-Insecure-Requests": "1",
			"Cache-Control":             "max-age=0",
		},
	})


	inf, err := s.Get(verifyUrl, nil)
	fmt.Print(s.RequestOptions.Cookies)
        fmt.Print(inf.RawResponse.Cookies())

[][JSESSIONID=3B606F518EBFD1011B57A56F8EAC8D76; Path=/selfservice]

in this case,I can't login some website and I found the Session didn't contain the Cookie.
Maybe I am worry,can you tell me? thanks!

RequestOptions UseCookieJar seems not in use

It seems that RequestOptions UseCookieJar parameter is specified but not used.

func BuildHTTPClient always use a cookie jar

Create Issue template

Do not overwrite client.CheckRedirect

grequests always set client.ChekcRedirect.

https://github.com/levigross/grequests/blob/0.9/utils.go#L55

But I want to use my client.CheckRedirect function in a test code.

client := http.Client{
    CheckRedirect: function() error {
        return errors.New("cancel redirection")
    },
}
r, err := grequests.Get("http://example.com/redirect", &grequests.RequestOptions{
    HTTPClient: &client,
})

Please support this feature?

response url convienience function

Hi, could you add a URL function to the Response struct, struggled with figuring out how to access final URL after series of redirect at first.

func (r *Response) URL() string {
return r.RawResponse.URL.String()
}

This would provide a much more obvious and simple way of accessing that final URL.

Build more modular API

test

There's sth. wrong with Proxies that use req.URL.Scheme.

If you want to use "http://localhost:8888" to proxy your application "https://baidu.com", but in "requess.go" that code use proxy environment.

I don't known why the proxies shoud be a map, and I want to known send a request why should need more proxies url.

Because I could code this

I think this code will let grequests wrong, Because it doesn't known how to select proxy, there has two proxy server.

I want to pull requet, but my English is so bad. I hope this can help other people who is using grequests.

And, I think ssl agent should be support in proxies .

ro := &grequests.RequestOptions{
   Proxies: []interface{"user-agent": "xx", pUrl: "http://xx:8080"}
}

Because if don't set ssl user agent, the remote serve may be check ssl user-aget

Some option to log everything

Please add some option to log everything (url, headers, request and response body)

Large response getting cut off.

simple_server.zip
Reading the response of a GET request using .String() arbitrarily cuts off the full response when the response is quite large (+5MB). The issue appears to occur when reading the internal buffer from the response object; the full response string is not returned.

Using http.Get() works fine:

Attached is a standalone project that reproduces the issue. The attached project has a simple server that returns a hard coded json string:

src/simple_server

There are also two clients:

src/grequests_client (reproduces the issue)
src/http_client (works fine)

The error occurs in grequests_client when trying to decode the json response: 'use of closed network connection'. But inspecting the json response using resp.String() or resp.Byte() shows that only part of the json response is returned.

proposal: allow custom request body

Atm it is no way to pass string, []byte, or even io.Reader as request body.

Async requests

Not sure how to make many requests in parallel as the python library does