elazarl/goproxy

An HTTP proxy library for Go

goproxy's Issues

Mitm'ed HTTPS responses are not automatically deflated

I'm trying to get the HTTPS MITM to work, and it looks like HTTPS responses are not automatically deflated before they are handed to the handlers.

Examples don't work

Some examples, like noRedditAtWorktime.go, don't work in my system.

The proxy do works, but it allows me to access reddit. What is wrong?

I have a Fedora 19 with go 1.1.2.

Https/NAT support

Is there any way to add transparent HTTPS (without CONNECT) handling?

I try to export goproxy.handleHttps and force it to listener without much success

Parent proxies?

It would be helpful if goproxy allowed to set a parent proxy.

Using atomic.AddInt64 with the field ProxyHttpServer.sess crashes on i386

This is apparently a known bug:

http://golang.org/src/pkg/sync/atomic/doc.go#L41

Moving the sess field to the top of the struct fixes the issue.

Thanks,
James

Basic Example Does Nothing

Hi.

Thanks for creating this library.

I'm looking for a basic example of using goproxy. The current basic example does nothing except publishing an empty page on the given port.

What I'm looking for is an example that actually proxies a hostname and port to another hostname and port, like; reddit.com:80 to localhost:8080

Best

Azer

Logging via context doesnt output anything

I'm just experimenting, getting myself familiar with goproxy. I was exploring one of the examples (the hijacking example) because I want to write a proxy that has access to the connection to the remote server (as I need to read the client request to determine the changes to make to the request to the remote server) and wanted to output some simple info from the request. However it appears that I can't seem to get anything logged to stdout using the proxy context. Could anyone more familiar with goproxy steer me in the right direction, or is this a bug with goproxy itself?

Code below:

package main

import (
    "bufio"
    "log"
    "math/rand"
    "net"
    "net/http"
    "regexp"
    "time"

    "github.com/elazarl/goproxy"
)

func orPanic(err error) {
    if err != nil {
        panic(err)
    }
}

func main() {
    rand.Seed(time.Now().UTC().UnixNano())

    proxy := goproxy.NewProxyHttpServer()
    proxy.OnRequest(goproxy.ReqHostMatches(regexp.MustCompile("^.*baidu.com$"))).HandleConnect(goproxy.AlwaysReject)
    proxy.OnRequest(goproxy.ReqHostMatches(regexp.MustCompile("^.*$"))).HandleConnect(goproxy.AlwaysMitm)

    // enable curl -p for all hosts on port 80
    proxy.OnRequest(goproxy.ReqHostMatches(regexp.MustCompile("^.*:80$"))).HijackConnect(
        func(req *http.Request, client net.Conn, ctx *goproxy.ProxyCtx) {
            defer func() {
                if e := recover(); e != nil {
                    ctx.Logf("error connecting to remote: %v", e)
                    client.Write([]byte("HTTP/1.1 500 Cannot reach destination\r\n\r\n"))
                }
                client.Close()
            }()

            // - Create a clientBuffer that is a reader/writer from the client connection
            // - Create a connection to the remote server
            // - Create remoteBuffer is a reader/writer from the new remote connection
            clientBuf := bufio.NewReadWriter(bufio.NewReader(client), bufio.NewWriter(client))
            remote, err := net.Dial("tcp", req.URL.Host)
            orPanic(err)
            remoteBuf := bufio.NewReadWriter(bufio.NewReader(remote), bufio.NewWriter(remote))

            id := randomString(10)
            ctx.Warnf("TEST")
            ctx.Warnf("-- ID %s --------------\n", id)

            for { // loop indefinately reading from the client buffer (until the buffer is empty?)

                ctx.Warnf("-- BGN %s --------------\n", id)

                // - Create a http.Request by reading from the clientBuffer
                req, err := http.ReadRequest(clientBuf.Reader)
                orPanic(err)

                // Write to remote and log at the same time!
                //io.MultiWriter(remoteBuf, )

                ctx.Warnf("URL: %s\n", req.URL)

                // - Write the request to the remoteBuffer and flush
                orPanic(req.Write(remoteBuf))
                orPanic(remoteBuf.Flush())

                // - Create a http.Response from the remoteBuffer
                resp, err := http.ReadResponse(remoteBuf.Reader, req)
                orPanic(err)

                // - Write the response to the clientBuffer and flush
                orPanic(resp.Write(clientBuf.Writer))
                orPanic(clientBuf.Flush())

                ctx.Warnf("-- END %s --------------\n", id)
            }
        })

    proxy.Verbose = true
    log.Fatal(http.ListenAndServe(":8080", proxy))
}

func randomString(l int) string {
    bytes := make([]byte, l)
    for i := 0; i < l; i++ {
        bytes[i] = byte(randInt(65, 90))
    }
    return string(bytes)
}

func randInt(min int, max int) int {
    return min + rand.Intn(max-min)
}

Memory leak

Compile examples on windows 7. Open 5-7 sites and look Task manager, process usage 200-400 Mb ram and increased with the opening of new sites.

Example StripSSL ?

Hello,
Really like this library!
Wondering about the StripSSL example?
If I'm understanding it correctly, it should redirect https traffic http traffic?
If yes it seems like its not working for me, I do get the Mitm certificate warning though.
What I'd like to achieve is to block all http/https traffic while showing an html or text content like the reddit example
Is it possible? I was able to do what I wanted on http requests but not https.

Syntax error in example proxy code

Hi Elazarl

Thanks a ton for goproxy. I am running go 1.2 and when I copy the following example, I get a syntax error

proxy.OnRequest().DoFunc(
    func(r *http.Request,ctx *goproxy.ProxyCtx)(*http.Request,*http.Response) {
        r.Header.Set("X-GoProxy","yxorPoG-X")
        return r,nil
    }
)

Error: ./proxy.go:17: syntax error: unexpected semicolon or newline, expecting )

I fixed by placing the }) on the same line.

proxy.OnRequest().DoFunc(
    func(r *http.Request,ctx *goproxy.ProxyCtx)(*http.Request,*http.Response) {
        r.Header.Set("X-GoProxy","yxorPoG-X")
        return r,nil
    })

You might want to update the README :)

Thanks a lot for this library!

query strings not passed forward through HTTPS when AlwaysMitm in effect

When I run goproxy with AlwaysMitm and make a client request like https://example.com?foo=bar it seems that goproxy does not include the query string with the proxied request. The verbose output reports

Sending request GET https://example.com:443

When I view web server access logs (on my own server, not example.com :) I also see that the query string did not get through.

I see this behavior with goproxy/examples/eavesdropper.

Convert https -> http

Is it possible to let goproxy terminate the ssl connection and "convert" it from https to http ? If yes, how could it be done ?

Multiple HandleConnect doesn't seem to work

When I do:

    proxy.OnRequest(goproxy.ReqHostIs("sitea.com:443")).HandleConnect(goproxy.AlwaysMitm)
    proxy.OnRequest(goproxy.ReqHostIs("siteb.com:443")).HandleConnect(goproxy.AlwaysMitm)

Only one of them works (I think the later one wins).

Getting and Modifying Request Values

To modify/get form fields I have to use r.ParseForm to populate the values in r.Form/r.PostForm.
However, doing this causes panic in goproxy

proxy.OnRequest().DoFunc(func(r *http.Request, ctx *goproxy.ProxyCtx) (*http.Request, *http.Response) {
        r.ParseForm()
        // hopefully the ability to modify by doing r.Form.Set("somefield")
        return r, nil
    })

Results in a panic.

goroutine 102 [running]:
net/http.func·011()
    /usr/local/go/src/pkg/net/http/server.go:1100 +0xb7
runtime.panic(0x2e3e60, 0x4db4c4)
    /usr/local/go/src/pkg/runtime/panic.c:248 +0x18d
github.com/elazarl/goproxy.(*ProxyCtx).Charset(0xc208124460, 0x0, 0x0)
    /Users/danny/Dev/go/src/github.com/elazarl/goproxy/ctx.go:82 +0x34
github.com/elazarl/goproxy/ext/html.func·002(0x0, 0xc208124460, 0xc2081a8000)
    /Users/danny/Dev/go/src/github.com/elazarl/goproxy/ext/html/html.go:52 +0x3e

Using GZIP to compress response of proxy

Hi,

I want to use goproxy to write a customer proxy which can gzip response. I want this because I want to use proxy in crawler. When I use crawler to crawl webpage through proxy, I want proxy to gzip webpages to save bandwidth between crawler and proxy server.

Is there an easy way to to this?

Thanks

Bad Request from wtfpod.com

I've only seen this on http://www.wtfpod.com/ but presumably it could show up elsewhere too.

Accessing that site through goproxy results in the following response:

$ http_proxy=localhost:8070 curl -i http://www.wtfpod.com/

HTTP/1.1 400 Bad Request
Content-Type: text/html
Date: Thu, 07 Feb 2013 16:49:06 GMT
Server: Apache/2.2.21 (Unix) mod_ssl/2.2.21 OpenSSL/0.9.8m DAV/2 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635
Transfer-Encoding: chunked
X-Powered-By: PHP/5.2.17

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<!--< html xmlns="http://www.w3.org/1999/xhtml">-->
<head>
<title>HTTP Error 400</title>
</head>
<body>
<h1>Error 400</h1>
<p>We're sorry, but we could not fulfill your request for
/ on this server.</p>
<p>Your proxy server sent an invalid request. Please contact the proxy server administrator to have this problem fixed.</p>
<p>Your technical support key is: <strong>4c76-b1b2-b783-0251</strong></p>
<p>You can use this key to <a href="http://www.ioerror.us/bb2-support-key?key=4c76-b1b2-b783-0251">fix this problem yourself</a>.</p>
<p>If you are unable to fix the problem yourself, please contact <a href="mailto:[email protected]">martin at celisdesign.com</a> and be sure to provide the technical support key shown above.</p>

Reproducible with a bare bones goproxy:

package main

import (
    "github.com/elazarl/goproxy"
    "log"
    "net/http"
)

func main() {
    proxy := goproxy.NewProxyHttpServer()
    proxy.Verbose = true

    log.Fatal(http.ListenAndServe(":8070", proxy))
}

Deployment on GAE

Excuse my ignorance on golang. I am very new to this.

How do you think the possibility and potential problems of running goproxy on Google App Engine?

errors regarding signing

github.com/elazarl/goproxy

../src/github.com/elazarl/goproxy/signer.go:62: template.IPAddresses undefined (type x509.Certificate has no field or method IPAddresses)

0 package main 1 2 import ( 3 "github.com/elazarl/goproxy" 4 "log" 5 "net/http" 6 ) 7 8 func main() { 9 proxy := goproxy.NewProxyHttpServer() 10 proxy.Verbose = true 11 log.Fatal(http.ListenAndServe(":8080", proxy)) 12 }

Frequent warnings related to connection close handling

Hi,

I am doing a small example of logging HTTPS Connect requests and during the execution the library throws warnings similar to:

2014/05/04 20:44:40 [248] WARN: Error copying to client WSARecv tcp 127.0.0.1:8080: use of closed network connection
2014/05/04 20:44:48 [245] WARN: Error copying to client WSARecv tcp 127.0.0.1:8080: use of closed network connection
2014/05/04 20:47:12 [018] WARN: Error copying to client WSARecv tcp 127.0.0.1:8080: An existing connection was forcibly closed by the remote host.

Looks like connection is already closed.

Do not work with https://www.facebook.com

Hi, folks

I try to use goproxy to visit https://www.facebook.com, I only got some errors. I checked go proxy with well with https://github.com.

Do you have any idea?

2014/06/28 04:47:55 [055] WARN: Error copying to client read tcp 111.199.82.183:62495: use of closed network connection
2014/06/28 04:47:55 [056] WARN: Error copying to client read tcp 111.199.82.183:62499: use of closed network connection
2014/06/28 04:47:55 [059] WARN: Error copying to client read tcp 111.199.82.183:62504: use of closed network connection
2014/06/28 04:47:55 [054] WARN: Error copying to client read tcp 111.199.82.183:62494: use of closed network connection
2014/06/28 04:47:55 [053] WARN: Error copying to client read tcp 111.199.82.183:62493: use of closed network connection
2014/06/28 04:47:55 [058] WARN: Error copying to client read tcp 111.199.82.183:62502: use of closed network connection
2014/06/28 04:47:56 [057] WARN: Error copying to client read tcp 111.199.82.183:62500: use of closed network connection
2014/06/28 04:47:56 [064] INFO: Running 0 CONNECT handlers
2014/06/28 04:47:56 [064] INFO: Accepting CONNECT to www.facebook.com:443
2014/06/28 04:47:56 [064] WARN: Error copying to client use of closed network connection
2014/06/28 04:47:56 [065] INFO: Running 0 CONNECT handlers
2014/06/28 04:47:56 [065] INFO: Accepting CONNECT to www.facebook.com:443
2014/06/28 04:47:56 [065] WARN: Error copying to client use of closed network connection
2014/06/28 04:47:56 [063] WARN: Error copying to client read tcp 111.199.82.183:62520: use of closed network connection
2014/06/28 04:47:56 [062] WARN: Error copying to client read tcp 111.199.82.183:62521: use of closed network connection
2014/06/28 04:47:57 [061] WARN: Error copying to client read tcp 111.199.82.183:62510: connection reset by peer
2014/06/28 04:47:57 [061] WARN: Error copying to client read tcp 54.236.179.219:443: use of closed network connection
2014/06/28 04:47:57 [052] WARN: Error copying to client read tcp 111.199.82.183:62489: use of closed network connection

Not able to start on Win 7 with 1.1Beta

Hi Elazarl

I wrote about not being able to run goproxy on stackoverflow recently
(I use latest Beta of Go)

The error is:

goproxy-master\goproxy-master\examples\basic>go run main.go
2013/04/22 10:41:34 http: panic serving 127.0.0.1:54214: runtime error:
 invalid memory address or nil pointer dereference
goroutine 4 [running]:
net/http.func·007()
        C:/Users/ADMINI~1/AppData/Local/Temp/2/bindist012168375/go/src/pkg/net/h
ttp/server.go:1022 +0x9e
sync/atomic.AddUint64()
        C:/Users/ADMINI~1/AppData/Local/Temp/2/bindist012168375/go/src/pkg/sync/
atomic/asm_386.s:69 +0xc
github.com/elazarl/goproxy.(*ProxyHttpServer).ServeHTTP(0x11ada1c0, 0x11abcfa0,
0x11ada440, 0x11a4f5b0)
        C:/Go/src/pkg/github.com/elazarl/goproxy/proxy.go:81 +0xb4
        net/http.serverHandler.ServeHTTP(0x11a4ea50, 0x11abcfa0, 0x11ada440, 0x11a4f5b0)

        C:/Users/ADMINI~1/AppData/Local/Temp/2/bindist012168375/go/src/pkg/net/h
ttp/server.go:1517 +0x120
net/http.(*conn).serve(0x11ad8410)
        C:/Users/ADMINI~1/AppData/Local/Temp/2/bindist012168375/go/src/pkg/net/h
ttp/server.go:1096 +0x652
created by net/http.(*Server).Serve
        C:/Users/ADMINI~1/AppData/Local/Temp/2/bindist012168375/go/src/pkg/net/h
ttp/server.go:1564 +0x241```

Infinite recursion upon a `GET /`

Goproxy recurses indefinitely when the address it's bound to is simply entered into a browser. The resulting log with Verbose = true:

2014/06/17 19:31:50 [000] INFO: Got request / 127.0.0.1:8080 GET /
2014/06/17 19:31:50 [000] INFO: Sending request GET http://127.0.0.1:8080/
2014/06/17 19:31:50 [001] INFO: Got request / 127.0.0.1:8080 GET /
2014/06/17 19:31:50 [001] INFO: Sending request GET http://127.0.0.1:8080/
2014/06/17 19:31:50 [002] INFO: Got request / 127.0.0.1:8080 GET /
2014/06/17 19:31:50 [002] INFO: Sending request GET http://127.0.0.1:8080/
2014/06/17 19:31:50 [003] INFO: Got request / 127.0.0.1:8080 GET /
2014/06/17 19:31:50 [003] INFO: Sending request GET http://127.0.0.1:8080/
[and so on...]

This is obviously not the way goproxy is supposed to used, but it opens a trivial attack vector: with a simple GET request it hogs up memory and open file limit. I'm happy to contribute the fix, but I'm not quite sure what accurate way there is to determine if the request is routed to the proxy itself.

sample for auth

hi... i try to write a proxy server by your goproxy is working great . but i can not find sample for http auth for goproxy .please help me for writing it.
english is not my speech language . i am sorry for bad writing

Create/Overwrite header

Hi, I was wondering if you have time creating an example on creating (or over-writing) the header passed to the client on both http and https? This would be useful on cases like modifying the Access-Control-Allow-Origin header. Thanks!

https mitm - Google

First of all thank you for putting together this project.

I'm trying the eavesdropper example and I can't get Google to return search results in https. I'm using Firefox. Note that mitmproxy (http://mitmproxy.org/doc/howmitmproxy.html) is working fine.

No play radio

Hi!
Compile you example "Basic", and open site http://www.the-radio.ru/online-1188-w96 , music dont play :(

Use: Go 1.0.3, Ubuntu 12.04

regretting after overflow makes no sense

Hi,

It seems image decode is giving this error.

2014/10/07 15:38:50 http: panic serving 192.168.1.78:3446: regretting after overflow makes no sense
goroutine 95 [running]:
net/http.func011()
/usr/local/go/src/pkg/net/http/server.go:1100 +0xb7
runtime.panic(0x6a7b20, 0xc2088004c0)
/usr/local/go/src/pkg/runtime/panic.c:248 +0x18d
github.com/elazarl/goproxy/regretable.(_RegretableReader).Regret(0xc2089ae500)
/home/tsgan/go/src/github.com/elazarl/goproxy/regretable/regretreader.go:49 +0x70
github.com/elazarl/goproxy/ext/image.func001(0xc20809e240, 0xc2089ae370, 0x800baf538)
/home/tsgan/go/src/github.com/elazarl/goproxy/ext/image/image.go:38 +0x20c
github.com/elazarl/goproxy.FuncRespHandler.Handle(0xc2081ef610, 0xc20809e240, 0xc2089ae370, 0xc2081f0000)
/home/tsgan/go/src/github.com/elazarl/goproxy/actions.go:35 +0x36
github.com/elazarl/goproxy.func016(0xc20809e240, 0xc2089ae370, 0x13)
/home/tsgan/go/src/github.com/elazarl/goproxy/dispatcher.go:279 +0x23f
github.com/elazarl/goproxy.FuncRespHandler.Handle(0xc2081b12e0, 0xc20809e240, 0xc2089ae370, 0x2)
/home/tsgan/go/src/github.com/elazarl/goproxy/actions.go:35 +0x36
github.com/elazarl/goproxy.(_ProxyHttpServer).filterResponse(0xc208054280, 0xc20809e240, 0xc2089ae370, 0xc20809e240)
/home/tsgan/go/src/github.com/elazarl/goproxy/proxy.go:69 +0xb3
github.com/elazarl/goproxy.(_ProxyHttpServer).ServeHTTP(0xc208054280, 0x800bb6d20, 0xc208998280, 0xc208027ba0)
/home/tsgan/go/src/github.com/elazarl/goproxy/proxy.go:123 +0x6e7
net/http.serverHandler.ServeHTTP(0xc2080045a0, 0x800bb6d20, 0xc208998280, 0xc208027ba0)
/usr/local/go/src/pkg/net/http/server.go:1673 +0x19f
net/http.(_conn).serve(0xc208054580)
/usr/local/go/src/pkg/net/http/server.go:1174 +0xa7e
created by net/http.(*Server).Serve
/usr/local/go/src/pkg/net/http/server.go:1721 +0x313

doc error

In your doc：
proxy := goproxy.NewProxyHttpServer()
proxy.OnRequest(..conditions..).Do(..requesthandler..)
proxy.OnRequest(..conditions..).DoFunc(..requesthandlerFunction..)
proxy.OnReponse(..conditions..).Do(..responesHandler..)
proxy.OnReponse(..conditions..).DoFunc(..responesHandlerFunction..)
http.ListenAndServe(":8080", proxy)

OnReponse <--or--> OnResponse

Removing the Content-Length header breaks HEAD requests

Hi,

We've started using your proxy as the basis of an internal proxy server and I have found a bug. The proxy deletes the Content-Length header on all responses on the basis that the http library should be able to work it out again from the body, however HEAD requests do contain the Content-Length header but do not contain the body so the library cannot determine the correct Content-Length in this case.

In our copy I have just removed this line and that works for the way we are using the library but I can see that would not be ideal in all scenarios.

You can reproduce this by using Windows Update with the proxy (which uses HEAD requests).

Thanks,
James

Status code on unresolvable hosts.

If we try to access through goproxy host which doesn't have DNS record goproxy returns 500 (Internal Server Error) HTTP status code, while squid-cache returns 502 (Bad Gateway) HTTP status code. Shouldn't goproxy simulate squid's behavior?

$ curl -v --proxy 127.0.0.1:3129  --url http://no.exists/
* About to connect() to proxy 127.0.0.1 port 3129 (#0)
*   Trying 127.0.0.1...
* Adding handle: conn: 0x1d71bd0
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x1d71bd0) send_pipe: 1, recv_pipe: 0
* Connected to 127.0.0.1 (127.0.0.1) port 3129 (#0)
> GET http://no.exists/ HTTP/1.1
> User-Agent: curl/7.32.0
> Host: no.exists
> Accept: */*
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 500 Internal Server Error
< Content-Type: text/plain; charset=utf-8
< Date: Tue, 20 May 2014 13:13:02 GMT
< Content-Length: 41
< 
dial tcp: lookup no.exists: no such host
* Connection #0 to host 127.0.0.1 left intact
$ 

Dialing errors during CONNECT request result in hung connection

When issuing a CONNECT request, if the dialer returns an error, ProxyHttpServer.handleHttps returns without issuing a response. But because it's already hijacked the connection, the connection never gets responded to or closed:

evan@caron:~$ cat test.go
package main

import (
    "github.com/elazarl/goproxy"
    "log"
    "net/http"
)

func main() {
    proxy := goproxy.NewProxyHttpServer()
    proxy.Verbose = true
    log.Fatal(http.ListenAndServe(":8080", proxy))
}
evan@caron:~$ go build test.go
evan@caron:~$ ./test
2014/01/28 12:16:56 [001] INFO: Running 0 CONNECT handlers

(Then in another terminal:)

evan@caron:~$ curl -v --proxy localhost:8080 https://alksdjflkajdsf
* Rebuilt URL to: https://alksdjflkajdsf/
* Adding handle: conn: 0x2393bd0
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x2393bd0) send_pipe: 1, recv_pipe: 0
* About to connect() to proxy localhost port 8080 (#0)
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 8080 (#0)
* Establish HTTP proxy tunnel to alksdjflkajdsf:443
> CONNECT alksdjflkajdsf:443 HTTP/1.1
> Host: alksdjflkajdsf:443
> User-Agent: curl/7.32.0
> Proxy-Connection: Keep-Alive
> 

[hangs indefinitely]

I think you either want to close the connection or return a 50x response. I'd lean towards the former, I think - it feels more representative of a transport error - but my usage doesn't depend on the exact behavior.

unexpected EOF

when get http://autoproxy-gfwlist.googlecode.com/svn/trunk/gfwlist.txt through goproxy
,it gives unexpected EOF

Why the following code doesn't work?

Here are the code:

package main

import (
"github.com/elazarl/goproxy"
"log"
"net/http"
)

func main() {
proxy := goproxy.NewProxyHttpServer()

proxy.OnResponse().DoFunc(func(r *http.Response, ctx *goproxy.ProxyCtx)*http.Response{
  println(ctx.Req.Host,"->",r.Header.Get("Content-Type"))
  return r
})

log.Fatalln(http.ListenAndServe(":8080", proxy))

}

MITM of https requests doesn't allow %2F in the URL

I use:
proxy.OnRequest().HandleConnect(goproxy.AlwaysMitm)

then, try to browse https://review.openstack.org but some pages break because of %2F in URLs like:
https://review.openstack.org/projects/openstack%2Fnova/config

I assume these are being proxied into requests for https://review.openstack.org/projects/openstack/nova/config - which is a 404

Feature Request: ad block list and url history

Great project!
Any chance that it is implemented with something like ad block list and url history? So people could use it as a very useful personal proxy server.
Thanks.

goproxy and transparent mode

First of all, many thanks for so useful software!

It generally works ok, but there is issue with transparent mode (i.e. with -A PREROUTING -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 8080) -- response is always nil, so i need to use burp->gorpoxy chain. Is there any way to fix it?

Chuncked encoding doesn't seem to work correctly in Chrome

Potentially related to #9.

I accidentally commented on #11. After further testing this seems to work in Safari, so I'm assuming it's chrome specific. I also found this which might be related.

The original bug report:

Thanks for the fix! It mostly works now, when the content-type is text/html. (I couldn't get the UrlIs filter to work with HTTPS, but other stuff like ContentTypeIs and DstHostIs works.)

However, I have an application/x-javascript file that I'm tampering the body, and it doesn't work.

One of these two things would happen:

Case 1. The server didn't send any response headers, but prepended these to the response body:

0

HTTP/1.0 200 OK
Accept-Ranges: bytes
Age: 1082
Cache-Control: max-age=315360000
Connection: keep-alive
Content-Length: 685580
Content-Type: application/x-javascript
Date: Wed, 03 Jul 2013 23:24:56 GMT
Expires: Thu, 31 Dec 2037 23:55:55 GMT
Last-Modified: Fri, 28 Jun 2013 17:30:08 GMT
Server: nginx
Transfer-Encoding: chunked
Vary: Accept-Encoding
Via: 1.0 4e181fe94ccef6b90a51121fd522cf07.cloudfront.net (CloudFront)
X-Amz-Cf-Id: 1KQu6HedzBf1NkS15WCAAsF4SZAi74VDVkux9GzBjZqqZaENHK6NmQ==
X-Cache: Hit from cloudfront

8000
// Actual respinse body begins here

In this case, the browser just keeps spinning and wait for the end on the response indefinitely.

Case 2. The server sends the response headers correctly, prepended this to my response body:

8000
// Actual respinse body begins here

I'm not 100% sure if this is due to the content type or if it's something specific to this particular file. Either way, it looks like goproxy is inserting some newlines where it's not supposed to and causing the response to break.

MITM https problem

I'm using code from #9

When I try to access https://www.yahoo.com, browser only shows "Get me out of here" button.

Log says:
...
2014/10/04 23:39:16 [012] INFO: Running 1 CONNECT handlers
2014/10/04 23:39:16 [012] INFO: handler: &{2 } www.yahoo.com:443
2014/10/04 23:39:16 [012] INFO: Assuming CONNECT is TLS, mitm proxying it
...

For other sites like hotmail.com browser shows "I understand the risks" and let me Add exception and go further.
How to solve above problem when accessing https://www.yahoo.com?
Please let me know.

thanks,

Feature Request: Add support for GZIP

Feature Request: Add module to proxy to enable all web related content to be automatically gzipped. Optionally add support for HTML, Javascript, and CSS to be minified.

Websocket does not work.

Even though secure websocket works. Test it by going to http://www.websocket.org/echo.html.

Relevant lines from log when trying it:

2014/03/26 22:24:27 [104] INFO: Running 0 CONNECT handlers
2014/03/26 22:24:27 [104] INFO: Accepting CONNECT to echo.websocket.org:80
2014/03/26 22:24:44 [105] INFO: Running 0 CONNECT handlers
2014/03/26 22:24:45 [105] INFO: Accepting CONNECT to echo.websocket.org:443
2014/03/26 22:24:47 [106] INFO: Got request / ocsp.godaddy.com POST http://ocsp.godaddy.com/
2014/03/26 22:24:47 [106] INFO: Sending request POST http://ocsp.godaddy.com/
2014/03/26 22:24:48 [106] INFO: Received response 200 OK
2014/03/26 22:24:48 [106] INFO: Copying response to client 200 OK [200]
2014/03/26 22:24:48 [106] INFO: Copied 1923 bytes to client error=<nil>
2014/03/26 22:24:49 [107] INFO: Got request /img/tls-lock.png www.websocket.org GET http://www.websocket.org/img/tls-lock.png
2014/03/26 22:24:49 [107] INFO: Sending request GET http://www.websocket.org/img/tls-lock.png

The request to tls-lock.png is made after websocket is successfully connected. [I think the request to godaddy is unrelated, I am not sure what is going on, as only firefox is using the proxy, and there are no other tabs open, who is making request to godaddy?]

Another attempt:

2014/03/26 22:32:42 [125] INFO: Running 0 CONNECT handlers
2014/03/26 22:32:42 [125] INFO: Accepting CONNECT to echo.websocket.org:80
2014/03/26 22:32:47 [126] INFO: Running 0 CONNECT handlers
2014/03/26 22:32:49 [126] INFO: Accepting CONNECT to echo.websocket.org:443

There is no logging once connect is made it seems. In this case, I sent a few packets back and forth on secure websocket connection and closed the connection afterwards.

Fails to build

The latest commit fails on go get with:

# github.com/elazarl/goproxy
../../elazarl/goproxy/https.go:132: undefined: newChunkedWriter

Latest version at time of filing this is:

4d91534

Use HTTP 1.1 for ConnectAccept case

Would it be possible to use the more efficient HTTP 1.1 while in the ConnectAccept case in handleHttps()? The code currently uses HTTP 1.0 to open the upstream connection.
The performance for 1.1 is substantially faster due to connection re-use, and it's quite perceptible when RTTs between the proxy and the origin server are large.

Thanks,
Siddharth

external proxy is not enabled by default

In a intranet, it would have to access external web site via proxy.
goproxy does not enable external proxy be default from environment.

I did a quick search and find DefaultTransport is not used in the code.
DefaultTransport will use ProxyFromEnvironment

Rejecting a CONNECT request with an html message?

I am trying to Reject CONNECT requests via

proxy.OnRequest().HandleConnectFunc(func(host string, ctx *goproxy.ProxyCtx) (*goproxy.ConnectAction, string) {
            ctx.Resp = goproxy.NewResponse(ctx.Req,
                    goproxy.ContentTypeHtml, http.StatusForbidden,
                    "<br><br><center><h1>Address is blocked.</h1></center>")
            return goproxy.RejectConnect, host
}

But this doesn't seem to work, browser shows the connection problem page instead of the html.

Cannot read TLS response from mitm'd server EOF

I use examples\sslstrip,Error when I access github.com
Cannot read TLS response from mitm'd server EOF

examples\basic,don't support https

support http connect method

Hi Elazarl,
Do you think this case is valid use case or not?
I meet a http request, the method is connect.
goproxy will fail to modify the response.
here is the demo code:

import (
    "net/http"
    "github.com/elazarl/goproxy"
    "github.com/elazarl/goproxy/ext/html"
)
func main() {
    proxy := goproxy.NewProxyHttpServer()
    proxy.Verbose = true
    proxy.OnResponse(goproxy_html.IsHtml).Do(goproxy_html.HandleString(func(s string, ctx *goproxy.ProxyCtx) string {
        return s + "<script>alert(1)</script>"
    }))
    http.ListenAndServe(":7000", proxy)
}

curl -v --proxy localhost:7000 http://www.baidu.com
will get modified page and it should pop up an alert dialog if you view it in browser.
curl -v -p --proxy localhost:7000 http://www.baidu.com
will not get modified page and popup dialog will not show.

auth

consider support username&password?

Customizing dialing in HTTPS proxy

I'm more than happy to put a PR together, but wanted to get a sense of your thoughts before I wrote it up.

It's possible to override how dialing works for HTTP proxies by setting proxy.Tr.Dial. This is useful for e.g. setting custom connect timeouts.

There's no equivalent functionality for https proxying, because it just calls net.Dial directly, and I've run into clients that don't set a timeout while waiting for the 200 from the CONNECT method (Ruby's Net::HTTP has this problem)

Does it make sense to look at Tr.Dial from the https code? Do you have any thoughts on alternative ways to allow overriding the dialer behavior?

runtime error when resp is banned by China GFW

http: panic serving 127.0.0.1:46899: runtime error: invalid memory address or nil pointer dereference
goroutine 1 [running]:
net/http.func·007()
/usr/local/go/src/pkg/net/http/server.go:1022 +0xac
github.com/elazarl/goproxy.func·012(0x0, 0xc200109320, 0x0)
/home/zhangpy/Applications/gopath/src/github.com/elazarl/goproxy/dispatcher.go:144 +0x30

---

simple fix by checking resp in dispatcher.go 144

func ContentTypeIs(typ string, types ...string) RespCondition {
    types = append(types, typ)
    return RespConditionFunc(func(resp *http.Response, ctx *ProxyCtx) bool {
        if resp == nil {
            fmt.Println("url is banned: ", ctx.Req.URL )
            return false
        }
        contentType := resp.Header.Get("Content-Type")
...

clarification on httpdump example

I'm using httpdump.go as a reference implementation for a simple forward proxy I'm cooking up that will intercept responses and push them into another system.

I need to send something like the Meta struct down a channel when the response is fully consumed by the downstream client, but more importantly when the body has been consumed by TeeReadCloser.

When LogMeta executes here:
https://github.com/elazarl/goproxy/blob/master/examples/httpdump/httpdump.go#L140
...am I reading it correctly that NewTeeReader hasn't had a chance necessarily to flush the response stream to the FileStream?

Would I need to add a call into TeeReadCloser.Close() ?