levigross / grequests Goto Github PK
View Code? Open in Web Editor NEWA Go "clone" of the great and famous Requests library
License: Apache License 2.0
A Go "clone" of the great and famous Requests library
License: Apache License 2.0
When implementing a library/app in another language, it's usually called a port, not a clone!
Not sure how to make many requests in parallel as the python library does
Look like problem in doFollowingRedirects()
...
if redirect != 0 {
nreq := new(Request)
nreq.Method = ireq.Method
if ireq.Method == "POST" || ireq.Method == "PUT" {
nreq.Method = "GET"
}
nreq.Header = make(Header)
nreq.URL, err = base.Parse(urlStr)
if err != nil {
break
}
if len(via) > 0 {
// Add the Referer header.
lastReq := via[len(via)-1]
if ref := refererForURL(lastReq.URL, nreq.URL); ref != "" {
nreq.Header.Set("Referer", ref)
}
...
There is only Referer
header is set.
body := strings.NewReader(`xxxxxxxx`)
opt := grequests.RequestOptions{
Headers: map[string]string{
"Content-Type":"multipart/form-data; boundary=xxoo",
"Transfer-Encoding":"chunked",
},
RequestBody:body,
}
The content of the request initiated by grequests is as follows
POST / HTTP/1.1
Host: xxxx.com
User-Agent: GRequests/0.10
Content-Length: 284
Content-Type: multipart/form-data; boundary=xxoo
Cookie: PHPSESSID=gjj4afltersilid2qhq7f9fcl1
Accept-Encoding: gzip, deflate
Connection: close
xxoo
Transfer-Encoding not found in the request
My question is how to add "Transfer-Encoding" to the header.
s := grequests.NewSession(&grequests.RequestOptions{
Headers: map[string]string{
"Host": "202.192.18.32:8080",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:51.0) Gecko/20100101 Firefox/51.0",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language": "zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3",
"Accept-Encoding": "gzip, deflate",
"Connection": "keep-alive",
"Upgrade-Insecure-Requests": "1",
"Cache-Control": "max-age=0",
},
})
inf, err := s.Get(verifyUrl, nil)
fmt.Print(s.RequestOptions.Cookies)
fmt.Print(inf.RawResponse.Cookies())
[][JSESSIONID=3B606F518EBFD1011B57A56F8EAC8D76; Path=/selfservice]
in this case,I can't login some website and I found the Session didn't contain the Cookie.
Maybe I am worry,can you tell me? thanks!
when i send a lot of requests, EOF happens. how could i close old connections before the results received by new connection?
such as post /api
but post data is text , not key=value ,i cannot use response.Data beacuse this is map[string]interface{}.
session := grequests.NewSession(nil) // session.HTTPClient.Transport initialized
session.RequestOptions.Proxies = proxie // here will not rebuild HTTPClient.Transport, so proxy does not work
Atm it is no way to pass string, []byte, or even io.Reader as request body.
Hello, is there a SOCKS-proxy support? I see that I can use HTTP/S proxies. When will you implement the support of SOCKS4/5?
some options in request options, such as all timeout settings, is actually bind to http.Client.
in doRegularRequest, we create client each time if timeout is not zero.
but in doSessionRequest, the client is created in NewSession().
so if user pass a RequestOptions with timeout to session.Get(), it will not change client's timeout.
I think current timeout implementation is good,but it's better to add some docs about this behavior.
Hi, could you add a URL function to the Response struct, struggled with figuring out how to access final URL after series of redirect at first.
func (r *Response) URL() string {
return r.RawResponse.URL.String()
}
This would provide a much more obvious and simple way of accessing that final URL.
opt := &grequests.RequestOptions{}
proxyURL, err := url.Parse("http://127.0.0.1:8080")
//body := strings.NewReader("username=admin&password=password&Login=Login")
opt.Headers = map[string]string{
"test":"test",
}
opt.Proxies = map[string]*url.URL{proxyURL.Scheme: proxyURL}
//opt.RequestBody = body
opt.UseCookieJar = true
opt.DisableCompression = false
opt.Data = map[string]string{"username":"admin","password":"password","Login":"Login"}
resp,err := grequests.Post ("http://192.168.95.4/aaa/login.php",opt)
fmt.Println(err)
//fmt.Println(resp.String())
fmt.Println(resp.StatusCode)
After login, the corresponding 302 status code of the web server,When grequests continues to make requests, the “test” header in the HTTP request header repeats
http requests
GET /aaa/index.php HTTP/1.1
Host: 192.168.95.4
User-Agent: GRequests/0.10
Content-Type: application/x-www-form-urlencoded
Content-Type: application/x-www-form-urlencoded
Cookie: security=high; PHPSESSID=90949a1f57fa20a9e948f589e17a6786
Referer: http://192.168.95.4/aaa/login.php
Test: test
Test: test
Accept-Encoding: gzip, deflate
Connection: close
The reason is that the function "addRedirectFunctionality" in the utils. go file uses req. Header. Add (k, v) ,
Modified to req. Header.Set (k, v) resolves the problem of duplication of HTTP request headers
My English is very poor
Is there any way to send requests with https instead of http or is that generally "impossible" in go?
In python, I can do this
requests.get('http://github.com', allow_redirects=False)
this call returns immediately without follow any redirection
How can I do this in Go? I tried to set RedirectLimit
to -1
in RequestOptions
but its not worked
Firstly, great package! Thanks for making this.
It seems that it's not possible to create a GET request, with parameters such as "http://localhost/file?tags=a&tags=b" since Params is a map.
See here:
ro := &RequestOptions{
Params: map[string]string{"Hello": "Goodbye"},
}
Get("http://httpbin.org/get?Hello=World", ro)
would you accept a pull request to fix this?
Thanks,
Greg
Please add some option to log everything (url, headers, request and response body)
Hi, this is a question.
I wanted to ask if it's possible to use fasthttp instead of net/http. From what I read it is significantly more performant:
Refer: https://medium.com/@nate510/don-t-use-go-s-default-http-client-4804cb19f779#.pla7aw7gz
Thanks for your package, been using it daily at work.
I am considering using this library in a new project at my company, and have made a generic wrapper over this library using the Req
method of the library, which is committed in master but unavailable when installed using glide
. I have already submitted the test cases for the same and raised a PR for it.
now grequest use the official implmentation of cookieJar (net/http/cookiejar)
I think this one lacks some functions, such as exporting/importing all cookies of a session.
But it's no way to use custom cookiejar at the moment.
maybe it's better to use our own cookiejar implementation to add some features, or allow user to override the default ones.
I read the response quirks on the readme but have so far been unable to get a struct populated using the response.JSON() method.
I'm using the latest version of grequests with Go 1.11. Here is the chunk of code in question:
resp, _ := session.Post(loginapi, ro)
fmt.Println(resp.Bytes())
fmt.Println(resp.String()) // S1
fmt.Println(resp.StatusCode)
var r = new(APIResponse)
fmt.Println(resp.JSON(&r))
fmt.Println(resp.String()) // S2
In this scenario, S1 will print the response string and S2 will be empty. If I remove the resp.JSON()
line everything works fine. Even if I try to call resp.JSON() first to populate before any other response method calls, the struct comes back nil.
go1.11.1
package main
import (
"github.com/levigross/grequests"
"sync"
)
func main(){
wg := new(sync.WaitGroup)
for i:=0;i <= 100;i ++ {
wg.Add(1)
go func() {
defer wg.Done()
grequests.Get("www.baidu.com", &grequests.RequestOptions{
Params: map[string]string{
"helllo": "world",
},
})
}()
}
wg.Wait()
return
}
go run --race main.go shows:
WARNING: DATA RACE
Read at 0x000000a2b0b0 by goroutine 9:
github.com/levigross/grequests.addRedirectFunctionality()
C:/Users/xxx/go/src/github.com/levigross/grequests/utils.go:55 +0x4ePrevious write at 0x000000a2b0b0 by goroutine 6:
github.com/levigross/grequests.addRedirectFunctionality()
C:/Users/xxx/go/src/github.com/levigross/grequests/utils.go:58 +0xc7
Check this out please!
For healthchecking, it's often useful to override the DNS provided IP of a URL, so that you can query a specific backend, while leaving things like SNI headers, etc intact.
Is there a way to do this with grequests?
Can you please give me an example of how to set a proxy for use with grequests?
Thank you.
This project is convenient for simply requesting and response handling, but it's tricky if you want to hook the request will be sent or the response we received.
Here are my suggestions:
Req
method for Session
buildRequest
and send request in other place, like requests
in PythonbuildResponse
for external response packgingI am trying to get an URL through a proxy and with a custom user agent:
ro := &grequests.RequestOptions{
RequestTimeout: s.httpTimeout,
UserAgent: uaString,
Proxies: map[string]*url.URL{
"http": proxyOne,
"https": proxyTwo,
},
}
resp, err := grequests.Get("http://someurl.com", ro)
But if I look at a network traffic dump - I see two types of requests:
Go-http-client/1.1
UA.Could you please fix UA handling in CONNECT calls? Thanks.
When sending a GET
, with Data map[string]string
set in the RequestOptions{}
, the documentation in the package states following:
// Data is a map of key values that will eventually convert into the
// query string of a GET request or the body of a POST request.
Using .Get()
on the *Session
and specifying the options in Data
, the content-type
in the http header is set to Content-Type:[application/x-www-form-urlencoded]
. This causes the request to be interpreted as a POST
. Either the code needs to be changed, or the documentation updated to state that Data
should be used for POST
s and QueryStruct
should be used for GET
s.
As much as I like Grequests and I truly want more people to find about it, forcing default User-Agent is not the way.
Line 558 in fd07961
Many platform will change behavior based on the User-Agent. Some of their WAF (web application firewall) will try to single out requests originating from the same IP, User-Agent, etc. There are situation where an empty User-Agent needs to be sent, literally User-Agent:
and Grequests is preventing that from happening.
So I would suggest two solutions.
// NullString represents a string that may be null.
type NullString struct {
String string
Valid bool // Valid is true if String is not NULL
}
and if Valid
is true
then use the String
provided in the struct.
working well as follow:
var ourl = "http://baidu.com"
prox,_ := url.Parse("http://219.135.164.245:3128")
fmt.Println("prox:", prox)
session := grequests.NewSession(&grequests.RequestOptions{
Proxies:map[string]*url.URL{prox.Scheme:prox},
})
rsp,err := session.Get(ourl, nil)
fmt.Println(err, rsp.String())
not working as follow:
var ourl = "http://baidu.com"
prox,_ := url.Parse("http://219.135.164.245:3128")
fmt.Println("prox:", prox)
session := grequests.NewSession(nil)
rsp,err := session.Get(ourl, &grequests.RequestOptions{
Proxies:map[string]*url.URL{prox.Scheme:prox},
})
fmt.Println(err, rsp.String())
simple_server.zip
Reading the response of a GET request using .String() arbitrarily cuts off the full response when the response is quite large (+5MB). The issue appears to occur when reading the internal buffer from the response object; the full response string is not returned.
Using http.Get() works fine:
Attached is a standalone project that reproduces the issue. The attached project has a simple server that returns a hard coded json string:
src/simple_server
There are also two clients:
src/grequests_client (reproduces the issue)
src/http_client (works fine)
The error occurs in grequests_client when trying to decode the json response: 'use of closed network connection'. But inspecting the json response using resp.String() or resp.Byte() shows that only part of the json response is returned.
grequests always set client.ChekcRedirect.
https://github.com/levigross/grequests/blob/0.9/utils.go#L55
But I want to use my client.CheckRedirect function in a test code.
client := http.Client{
CheckRedirect: function() error {
return errors.New("cancel redirection")
},
}
r, err := grequests.Get("http://example.com/redirect", &grequests.RequestOptions{
HTTPClient: &client,
})
Please support this feature?
I thought it would be like this.
grequests.RequestOptions{
RedirectLimit: -1,
}
But this code does not disable redirect,please tell me how to disable redirect, thanks~~~
Please rework RequestOptions to take
[]*http.Cookie
instead of
[]http.Cookie
because http.Response.Cookies()
returns []*http.Cookie
Is there a reason why this library doesn't provide the "go way" of error handling ?
resp := grequests.Get("http://httpbin.org/xml", nil)
if resp.Error != nil {
log.Fatalln("Unable to make request", resp.Error)
}
VS
resp, err := grequests.Get("http://httpbin.org/xml", nil)
if err != nil {
log.Fatalln("Unable to make request", err)
}
feature request
Sorry for the inconvenience!
proxyURL, err := url.Parse("http://127.0.0.1:8080") // Proxy URL
if err != nil {
log.Panicln(err)
}
resp, err := grequests.Get("http://www.google.com",
&grequests.RequestOptions{Proxies: map[string]*url.URL{proxyURL.Scheme: proxyURL}})
if err != nil {
log.Println(err)
}
if resp.Ok != true {
log.Println("Request did not return OK")
}
log.Println(resp)
The code above will work as expected.
But, if you replace http://www.google.com
to https://www.google.com
, it will not work, it access google directly without the proxy you set!
It seems that RequestOptions UseCookieJar parameter is specified but not used.
func BuildHTTPClient always use a cookie jar
Requests allows you to create requests using the request function, instead of having to call the wrapper for the http method. I am trying to achieve the same functionality in grequests, but doRegularRequest is private.
If you want to use "http://localhost:8888" to proxy your application "https://baidu.com", but in "requess.go" that code use proxy environment.
I don't known why the proxies shoud be a map, and I want to known send a request why should need more proxies url.
I think this code will let grequests wrong, Because it doesn't known how to select proxy, there has two proxy server.
I want to pull requet, but my English is so bad. I hope this can help other people who is using grequests.
And, I think ssl agent should be support in proxies .
ro := &grequests.RequestOptions{
Proxies: []interface{"user-agent": "xx", pUrl: "http://xx:8080"}
}
Because if don't set ssl user agent, the remote serve may be check ssl user-aget
resp, err := grequests.Post(url, options)
if err != nil || resp.StatusCode != 200 {
...
return
}
above is my code, and after about 2 weeks, I could get an error as socket: too many open files
. I don't know whether it due to the response unclosed. resp
only used to get StatusCode, no resp.JSON
or any other usages.
Should i must close the connection? like defer resp.Close(). Or what should I do to avoid this socket error?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.