xinsnake / go-http-digest-auth-client Goto Github PK
View Code? Open in Web Editor NEWGolang Http Digest Authentication Client
License: BSD 2-Clause "Simplified" License
Golang Http Digest Authentication Client
License: BSD 2-Clause "Simplified" License
I think the TCP connections are not reused with the current code:
package main
import (
dac "github.com/xinsnake/go-http-digest-auth-client"
"io"
"io/ioutil"
)
const ENDPOINT = "https://foobar.com"
const USERNAME = "Foo"
const PASSWORD = "Bar"
const DBPATH = "Baz"
func main() {
// create a new digest authentication request
dr := dac.NewRequest(USERNAME, PASSWORD, "GET", ENDPOINT+"v1/documents/?uri="+DBPATH, "")
resp, _ := dr.Execute()
io.Copy(ioutil.Discard, resp.Body)
resp.Body.Close()
dr.UpdateRequest(USERNAME, PASSWORD, "GET", ENDPOINT+"v1/documents/?uri="+DBPATH, "")
resp, _ = dr.Execute()
io.Copy(ioutil.Discard, resp.Body)
resp.Body.Close()
dr = dac.NewRequest(USERNAME, PASSWORD, "GET", ENDPOINT+"v1/documents/?uri="+DBPATH, "")
resp, _ = dr.Execute()
io.Copy(ioutil.Discard, resp.Body)
resp.Body.Close()
}
I patched the client according to: https://blog.golang.com/http-tracing
diff --git a/digest_auth_client.go b/digest_auth_client.go
index a2e69c4..9f0f7ba 100644
--- a/digest_auth_client.go
+++ b/digest_auth_client.go
@@ -4,7 +4,9 @@ import (
"bytes"
"crypto/tls"
"fmt"
+ "log"
"net/http"
+ "net/http/httptrace"
"time"
)
@@ -106,6 +108,12 @@ func (dr *DigestRequest) Execute() (resp *http.Response, err error) {
return nil, err
}
req.Header = dr.Header
+ trace := &httptrace.ClientTrace{
+ GotConn: func(connInfo httptrace.GotConnInfo) {
+ log.Printf("Got Conn: %+v\n", connInfo)
+ },
+ }
+ req = req.WithContext(httptrace.WithClientTrace(req.Context(), trace))
client := dr.getHTTPClient()
@@ -169,6 +177,13 @@ func (dr *DigestRequest) executeRequest(authString string) (resp *http.Response,
req.Header = dr.Header
req.Header.Add("Authorization", authString)
+ trace := &httptrace.ClientTrace{
+ GotConn: func(connInfo httptrace.GotConnInfo) {
+ log.Printf("Got Conn: %+v\n", connInfo)
+ },
+ }
+ req = req.WithContext(httptrace.WithClientTrace(req.Context(), trace))
+
client := dr.getHTTPClient()
return client.Do(req)
}
And here's what I got:
Go/src/test via 🐹 v1.14.4 took 6s
[N] ➜ go run main.go
2020/07/03 15:08:58 Got Conn: {Conn:0xc000080380 Reused:false WasIdle:false IdleTime:0s}
2020/07/03 15:08:58 Got Conn: {Conn:0xc000600e00 Reused:false WasIdle:false IdleTime:0s}
2020/07/03 15:09:00 Got Conn: {Conn:0xc000601180 Reused:false WasIdle:false IdleTime:0s}
2020/07/03 15:09:02 Got Conn: {Conn:0xc000601880 Reused:false WasIdle:false IdleTime:0s}
2020/07/03 15:09:02 Got Conn: {Conn:0xc000580380 Reused:false WasIdle:false IdleTime:0s}
I believe the problem is twofold:
https://golang.org/src/net/http/transport.go?h=http.Transport
// By default, Transport caches connections for future re-use.
// This may leave many open connections when accessing many hosts.
// This behavior can be managed using Transport's CloseIdleConnections method
// and the MaxIdleConnsPerHost and DisableKeepAlives fields.
//
// Transports should be reused instead of created as needed.
// Transports are safe for concurrent use by multiple goroutines.
By applying the following patch:
diff --git a/digest_auth_client.go b/digest_auth_client.go
index a2e69c4..13ff3c3 100644
--- a/digest_auth_client.go
+++ b/digest_auth_client.go
@@ -54,9 +56,9 @@ func (dr *DigestRequest) getHTTPClient() *http.Client {
return &http.Client{
Timeout: 30 * time.Second,
- Transport: &http.Transport{
- TLSClientConfig: &tlsConfig,
- },
+ //Transport: &http.Transport{
+ //TLSClientConfig: &tlsConfig,
+ //},
}
}
we get the following:
Go/src/test via 🐹 v1.14.4 took 7s
[N] ➜ go run main.go
2020/07/03 15:13:06 Got Conn: {Conn:0xc000105880 Reused:false WasIdle:false IdleTime:0s}
2020/07/03 15:13:06 Got Conn: {Conn:0xc000104700 Reused:false WasIdle:false IdleTime:0s}
2020/07/03 15:13:06 Got Conn: {Conn:0xc000104700 Reused:true WasIdle:true IdleTime:221.992µs}
2020/07/03 15:13:06 Got Conn: {Conn:0xc000104700 Reused:true WasIdle:true IdleTime:115.194µs}
2020/07/03 15:13:06 Got Conn: {Conn:0xc000053500 Reused:false WasIdle:false IdleTime:0s}
diff --git a/digest_auth_client.go b/digest_auth_client.go
index a2e69c4..d97ad9a 100644
--- a/digest_auth_client.go
+++ b/digest_auth_client.go
@@ -4,7 +4,11 @@ import (
"bytes"
"crypto/tls"
"fmt"
+ "io"
+ "io/ioutil"
"net/http"
"time"
)
@@ -129,6 +139,7 @@ func (dr *DigestRequest) executeNewDigest(resp *http.Response) (resp2 *http.Resp
)
// body not required for authentication, closing
+ io.Copy(ioutil.Discard, resp.Body)
resp.Body.Close()
if waString = resp.Header.Get("WWW-Authenticate"); waString == "" {
Then we get:
Go/src/test via 🐹 v1.14.4
[N] ➜ go run main.go
2020/07/03 15:17:59 Got Conn: {Conn:0xc000080380 Reused:false WasIdle:false IdleTime:0s}
2020/07/03 15:17:59 Got Conn: {Conn:0xc000080380 Reused:true WasIdle:true IdleTime:91.744µs}
2020/07/03 15:18:01 Got Conn: {Conn:0xc000080380 Reused:true WasIdle:true IdleTime:117.822µs}
2020/07/03 15:18:01 Got Conn: {Conn:0xc000080380 Reused:true WasIdle:true IdleTime:79.115µs}
2020/07/03 15:18:01 Got Conn: {Conn:0xc000080380 Reused:true WasIdle:true IdleTime:193.787µs}
The problem with hanging connections is that under a heavy load, the client can run out of TCP connections, since they're all in a TIME-WAIT state.
I think problem 2 is easy to solve (simply discard the resp.Body)
I have no idea how to solve problem 1 while still keeping a configurable client, though.
Sorry for that long text, and thank you very much for your work on this useful library :)
response body not close , whitch created in func (dr *DigestRequest) Execute() (resp *http.Response, err error)
user only can get and close the response body created in func (dr *DigestRequest) executeRequest(authString string) (*http.Response, error), but the response body created in func (dr *DigestRequest) Execute() (resp *http.Response, err error) is not close anywhere.
i think can add resp.Body.Close() after line 102 in file digest_auth_client.go, is it right
The last 2 releases were tagged without the v
in front: 0.5.0
and 0.6.0
. As a result, these 2 releases are not discoverable via go modules. See https://pkg.go.dev/github.com/xinsnake/go-http-digest-auth-client?tab=versions for the list of discoverable releases.
The 0.5.0
and 0.6.0
releases should be retagged to v0.5.0
and v0.6.0
respectively.
It would be nice if your package can provide a http.RoundTripper
implementation similar to https://godoc.org/github.com/bobziuchkovski/digest
This would allow your package to be used with a http.Client
which can then be used with the ctxhttp
package.
Hi @xinsnake,
thanks for this neat library! I'm using it to grab some pictures from an IP Camera which requires the digest based authentication.
The only wish I have is to be able to set a shorter timeout. Currently the timeout is hard coded to 30 seconds.
Maybe an implementation using the context package would make sense. Context.WithTimeout
could be an adequate solution.
need add code in digest_auth_client.go line: 94:
// 修复header问题
if req.Header != nil {
dr.Header = req.Header
}
Hello,
I am trying to authenticate on a server that uses digest authentication. After installing your plugin, i still receive 401.
uri := "http://localhost:18082/json_rpc"
fmt.Println("-----")
dr := dac.NewRequest("mihai", "mihai", "POST", uri, "")
response1, err := dr.Execute()
fmt.Println(response1)
fmt.Println("-----")
This is the server response:
HTTP/1.1 401 Unauthorized
Server: Epee-based
Content-Length: 98
Content-Type: text/html
Last-Modified: Tue, 02 Jan 2018 14:20:12 GMT
Accept-Ranges: bytes
WWW-authenticate:Digest qop="auth",algorithm=MD5,realm="monero-rpc",nonce="PfLdSGoqjOOSMw5bmJOv/g==",stale=false
WWW-authenticate:Digest qop="auth",algorithm=MD5-sess,realm="monero-rpc",nonce="PfLdSGoqjOOSMw5bmJOv/g==",stale=false
PS: I have also tried with http.Request
What am i doing wrong?
Thanks
There's been around 10 new commits since the last release. Do you think it's possible to tag a new release?
Hello,
I'm using the library but the hardcoded timeout of 30" is too small for my use-case (I'm downloading very large files).
What do you think about making the value configurable?It could be added to DigestRequest
and DigestTransport
so that it's picked up by the paths that execute the requests eventually.
If that sounds good, I could prepare a patch. Thanks!
P.S. This is probably related to #5.
test string: "Digest realm="e29072b1b78a1e4d7bfdec11", domain="::", qop="auth", nonce="nonce", opaque="", algorithm="MD5", stale="FALSE"", regex error when param has empty string
Using pprof i noticed alot of hanging goroutines and in prometheus i could see that filedescriptors was accumulating. Traced the leak down to be from this package, switched to github.com/rkl-/digest instead and the leak is now gone. Thought i would let you know though.
I'm trying to do an HTTP POST using this library with the second approach but http headers are not recognized.
transport := dig.NewTransport(username, password)
req, err := http.NewRequest("POST", fullUri, someJsonData)
req.Header.Set("Content-Type", "application/json")
resp, _ := transport.RoundTrip(req)
Tried to do a POST on to https://requestb.in and Content-Type header is indeed not received.
Tried to use another library but this exact same code worked.
I haven't looked into the pull requests but this PR #6 might have already solved this problem.
I'm trying to call an external api in loop using go concurrency at one time nearly 16k but I'm getting error
net/http: request canceled (Client.Timeout exceeded while awaiting headers)
here is my impletenrtin
`
var wg sync.WaitGroup
wg.Add(16367)
// fmt.Println("Running for loop…")
for results.Next() {
var tag StationData
// for each row, scan the result into our tag composite object
err = results.Scan(&tag.ID, &tag.Station)
if err != nil {
panic(err.Error()) // proper error handling instead of panic in your app
}
id := tag.ID
stationname := tag.Station
go func(id int) {
defer wg.Done()
time.Sleep(time.Millisecond * 500)
t := dac.NewTransport("username", "password")
req, err := http.NewRequest("GET", "urlsomttthh", nil)
if err != nil {
log.Fatalln(err)
}
response, err := t.RoundTrip(req)
if err != nil {
fmt.Printf("somthing missing from data %s\n", err)
} else {
data, _ := ioutil.ReadAll(response.Body)
fmt.Println(string(data))
}
}(id)
// datasprin := "somthing" + tag.Station
// and then print out the tag's Name attribute
// log.Printf(datasprin)
}
wg.Wait()
// fmt.Println("Finished for loop")
fmt.Printf("total time %s", time.Now().Sub(start))
In func NewRequest, could create parameter with headers.
I had a problem with an api, I had to send Content-Type. I made the following change:
func (dr *DigestRequest) UpdateRequest(username, password, method, uri, body string) *DigestRequest {
dr.Body = body
dr.Method = method
dr.Password = password
dr.URI = uri
dr.Username = username
dr.Header = make(map[string][]string)
//HERE
dr.Header["Content-Type"] = []string{"application/json"}
return dr
}
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.