lox/httpcache

An RFC7234 compliant golang http.Handler for caching HTTP responses

httpcache's Issues

Enable Sourcegraph

I want to use Sourcegraph code search and code review with httpcache. A project maintainer needs to enable it to set up a webhook so the code is up-to-date there.

Could you please enable httpcache on @sourcegraph by going to https://sourcegraph.com/github.com/lox/httpcache and clicking on Settings? (It should only take 15 seconds.)

Thank you!

http: proxy error: unsupported protocol scheme ""

I have my proxy settings to only use httpcache for HTTP (not HTTPS) but I get this error when things like Dropbox try to use HTTPS:

2014/11/14 11:26:11 http: proxy error: unsupported protocol scheme ""
2014/11/14 11:26:11 127.0.0.1 "CONNECT //client-lb.dropbox.com:443 HTTP/1.1" (Internal Server Error) 0 ←[33;1mSKIP←[0m 0

I suspect it's because it's using // instead of specifying a scheme? Is this something httpcache should be able to handle?

No LICENSE

I didn't see a LICENSE file or appropriate headers in the files. Maybe BSD, Apache or MIT? :-)

Don't treat no-cache as no-store

https://httpwg.github.io/specs/rfc7234.html#rfc.section.5.2.2.2

no-cache in a response means you can still store on your proxy, you just cannot serve it without revalidation. We should store the content but immediately expired. Then revalidate it upon request.

I can work on a patch for this if you would like.

Update tests for newer testify lib

testify has been updated to report failures for assert.Equal when the types are different. This can be fixed like so:

diff --git a/spec_test.go b/spec_test.go
index 700cd6c..e656c74 100644
--- a/spec_test.go
+++ b/spec_test.go
@@ -304,7 +304,7 @@ func TestSpecCacheControlMaxStale(t *testing.T) {
        upstream.timeTravel(time.Second * 90)
        r3 := client.get("/")
        assert.Equal(t, "MISS", r3.cacheStatus)
-       assert.Equal(t, 0, r3.age)
+       assert.Equal(t, time.Duration(0), r3.age)
 }

 func TestSpecValidatingStaleResponsesUnchanged(t *testing.T) {
@@ -334,7 +334,7 @@ func TestSpecValidatingStaleResponsesWithNewContent(t *testing.T) {
        assert.Equal(t, http.StatusOK, r2.Code)
        assert.Equal(t, "MISS", r2.cacheStatus)
        assert.Equal(t, "brand new content", string(r2.body))
-       assert.Equal(t, 0, r2.age)
+       assert.Equal(t, time.Duration(0), r2.age)
 }

 func TestSpecValidatingStaleResponsesWithNewEtag(t *testing.T) {
@@ -458,7 +458,7 @@ func TestSpecFresheningGetWithHeadRequest(t *testing.T) {

        refreshed := client.get("/explicit")
        assert.Equal(t, "HIT", refreshed.cacheStatus)
-       assert.Equal(t, 0, refreshed.age)
+       assert.Equal(t, time.Duration(0), refreshed.age)
        assert.Equal(t, "llamas", refreshed.header.Get("X-Llamas"))
 }

Was this service designed for reverse proxy only?

Was this service designed for reverse proxy only?
I can think of a way to use it in as a forward proxy but couple things are not really sitting in my head around the design of the software.

Manual evictions

I find varnish cache BAN functionality very neat and it would be nice to be able to use it here as well.
Right now a key hash is calculated and stored which makes wildcard bans problematic.

Also there seems to be no easy way to manually do something like this:

cache.Invalidate(path) because of how keys are created (I may very well be wrong here...)

I tried using the underlying VFS (memory) but again it seems to fail on how keys are created.

Is there a way to do this currently? I have handlers in place to intercept so I think I just need a way to deterministically calculate keys.

As for wildcards any ideas would be welcome keeping another set of currently seen keys may work but seems redundant and error prone at best.

Possible Race condition

Hi, I run into following race condition. On windows the issue happens randomly but on travis ci its reproducible see logs https://travis-ci.org/hemerajs/momos/builds/266557761

==================
WARNING: DATA RACE
Write at 0x00c420160058 by goroutine 16:
  net/http.(*response).Header()
      /home/travis/.gimme/versions/go1.8.3.linux.amd64/src/net/http/server.go:1032 +0x63
  github.com/lox/httpcache.(*responseStreamer).Header()
      <autogenerated>:97 +0x69
  net/http/httputil.(*ReverseProxy).ServeHTTP()
      /home/travis/.gimme/versions/go1.8.3.linux.amd64/src/net/http/httputil/reverseproxy.go:257 +0xc9a
  github.com/lox/httpcache.(*Handler).passUpstream.func1()
      /home/travis/gopath/src/github.com/lox/httpcache/handler.go:254 +0x8b
Previous write at 0x00c420160058 by goroutine 12:
  net/http.(*response).Header()
      /home/travis/.gimme/versions/go1.8.3.linux.amd64/src/net/http/server.go:1032 +0x63
  github.com/lox/httpcache.(*Handler).passUpstream()
      /home/travis/gopath/src/github.com/lox/httpcache/handler.go:261 +0x4e6
  github.com/lox/httpcache.(*Handler).ServeHTTP()
      /home/travis/gopath/src/github.com/lox/httpcache/handler.go:97 +0xbf9
  net/http.serverHandler.ServeHTTP()
      /home/travis/.gimme/versions/go1.8.3.linux.amd64/src/net/http/server.go:2568 +0xbc
  net/http.(*conn).serve()
      /home/travis/.gimme/versions/go1.8.3.linux.amd64/src/net/http/server.go:1825 +0x71a
Goroutine 16 (running) created at:
  github.com/lox/httpcache.(*Handler).passUpstream()
      /home/travis/gopath/src/github.com/lox/httpcache/handler.go:256 +0x36a
  github.com/lox/httpcache.(*Handler).ServeHTTP()
      /home/travis/gopath/src/github.com/lox/httpcache/handler.go:97 +0xbf9
  net/http.serverHandler.ServeHTTP()
      /home/travis/.gimme/versions/go1.8.3.linux.amd64/src/net/http/server.go:2568 +0xbc
  net/http.(*conn).serve()
      /home/travis/.gimme/versions/go1.8.3.linux.amd64/src/net/http/server.go:1825 +0x71a
Goroutine 12 (running) created at:
  net/http.(*Server).Serve()
      /home/travis/.gimme/versions/go1.8.3.linux.amd64/src/net/http/server.go:2668 +0x35f
  net/http/httptest.(*Server).goServe.func1()
      /home/travis/.gimme/versions/go1.8.3.linux.amd64/src/net/http/httptest/server.go:235 +0xa2
==================

Windows

==================
WARNING: DATA RACE
Write at 0x00c04213c138 by goroutine 8:
  net/http.(*response).Header()
      C:/Go/src/net/http/server.go:1032 +0x6a
  github.com/lox/httpcache.(*Handler).passUpstream()
      E:/go/work/src/github.com/lox/httpcache/handler.go:261 +0x4ed
  github.com/lox/httpcache.(*Handler).ServeHTTP()
      E:/go/work/src/github.com/lox/httpcache/handler.go:97 +0xc00
  net/http.serverHandler.ServeHTTP()
      C:/Go/src/net/http/server.go:2568 +0xc3
  net/http.(*conn).serve()
      C:/Go/src/net/http/server.go:1825 +0x721

Previous write at 0x00c04213c138 by goroutine 20:
  net/http.(*response).Header()
      C:/Go/src/net/http/server.go:1032 +0x6a
  github.com/lox/httpcache.(*responseStreamer).Header()
      <autogenerated>:97 +0x70
  net/http/httputil.(*ReverseProxy).ServeHTTP()
      C:/Go/src/net/http/httputil/reverseproxy.go:257 +0xca1
  github.com/lox/httpcache.(*Handler).passUpstream.func1()
      E:/go/work/src/github.com/lox/httpcache/handler.go:254 +0x92

Goroutine 8 (running) created at:
  net/http.(*Server).Serve()
      C:/Go/src/net/http/server.go:2668 +0x366
  net/http.(*Server).ListenAndServe()
      C:/Go/src/net/http/server.go:2585 +0xe7
  main.main()
      E:/go/work/src/github.com/hemerajs/momos/examples/server.go:61 +0x23d

Goroutine 20 (finished) created at:
  github.com/lox/httpcache.(*Handler).passUpstream()
      E:/go/work/src/github.com/lox/httpcache/handler.go:256 +0x371
  github.com/lox/httpcache.(*Handler).ServeHTTP()
      E:/go/work/src/github.com/lox/httpcache/handler.go:97 +0xc00
  net/http.serverHandler.ServeHTTP()
      C:/Go/src/net/http/server.go:2568 +0xc3
  net/http.(*conn).serve()
      C:/Go/src/net/http/server.go:1825 +0x721

Transfer-Encoding: Chunked returns only the first chunk

Thanks for your work with httpcache - no small effort!

I have found that the Transfer-Encoding: chunked behavior is broken. When sending a request through httpcache.NewHandler(httpcche.NewMemoryCache()) with httputil.NewSingleHostReverseProxy, that the cache returns only the first chunk.

Without the cache, httputil.NewSingleHostReverseProxy properly fetches chunked content fine.

An example of chunked output from the web server I'm testing with:

Jasons-MacBook-Pro:gateway-test jfesler$ telnet localhost 4000
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
GET /cached/hello/ HTTP/1.1
Host: localhost:3000

HTTP/1.1 200 OK
Content-Type: text/plain
Date: Tue, 13 Sep 2016 23:45:55 GMT
Transfer-Encoding: chunked

2f
Counter(i=0); time is now 2016-09-13- 16:45:55

2f
Counter(i=1); time is now 2016-09-13- 16:45:56

2f
Counter(i=2); time is now 2016-09-13- 16:45:57

2f
Counter(i=3); time is now 2016-09-13- 16:45:58

2f
Counter(i=4); time is now 2016-09-13- 16:45:59

2f
Counter(i=5); time is now 2016-09-13- 16:46:00

2f
Counter(i=6); time is now 2016-09-13- 16:46:01

2f
Counter(i=7); time is now 2016-09-13- 16:46:02

2f
Counter(i=8); time is now 2016-09-13- 16:46:03

2f
Counter(i=9); time is now 2016-09-13- 16:46:04

0

And with curl, hitting that proxy directly, it works fine as well:

Jasons-MacBook-Pro:gateway-test jfesler$ curl http://localhost:4000/counter/
Counter(i=0); time is now 2016-09-13- 16:52:56
Counter(i=1); time is now 2016-09-13- 16:52:57
Counter(i=2); time is now 2016-09-13- 16:52:58
Counter(i=3); time is now 2016-09-13- 16:52:59
Counter(i=4); time is now 2016-09-13- 16:53:00
Counter(i=5); time is now 2016-09-13- 16:53:01
Counter(i=6); time is now 2016-09-13- 16:53:02
Counter(i=7); time is now 2016-09-13- 16:53:03
Counter(i=8); time is now 2016-09-13- 16:53:04
Counter(i=9); time is now 2016-09-13- 16:53:05

On the bright side, the cache at least doesn't serve me the same chunk#1 as a cache hit; it instead calls it a miss and goes to the back end again on the subsequent query.

Unfortunately, I can't provide code samples, due to my current choice of employer. Which means I can't offer a PR either :-( For that I truly apologize.

httpcache makes two requests

While using httpcache in an environment where we used a ReverseProxy implementation which supports several upstream hosts I noticed that the cache makes two requests to the upstream when cache has expired.

Fixing outreq in pull request Use outreq helped me, but two requests are still problematic.

• With multiple upstreams validation can succeed, but the second request may fail.
• Two requests, validation not being HEAD, makes it impossible to implement round robin with simple upstreams[n]%len(upstreams)

Also, we think that it is a bit strange that the code uses only differences in the headers to check whether the cache should be updated. If we respond with exactly the same headers, the cache does not seem to get updated at all.

Cached requests return empty response

Using the cli tool with memory or disk-based cache, any request that has been cached returns an empty reply.

To reproduce:

./cli -listen="0.0.0.0:8080" -dir="" -v
export http_proxy="http://127.0.0.1:8080/"
curl http://getbootstrap.com/css/
curl http://getbootstrap.com/css/

The second curl request returns no data.

Don't get it working with max-age

Hi,
II just found you library and would like to implement an reserve proxy with HTTP caching but the example in the bench_test.go dies not work. I expect that the payload is cached.

	backend := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		w.Header().Add("Cache-Control", "max-age=100000")
		fmt.Fprintf(w, "cache server payload")
		fmt.Print("Inbound\n")
	}))
	defer backend.Close()

	u, err := url.Parse(backend.URL)

	if err != nil {
		panic(err)
	}

	handler := httpcache.NewHandler(httpcache.NewMemoryCache(), httputil.NewSingleHostReverseProxy(u))
	handler.Shared = true
	cacheServer := httptest.NewServer(handler)
	defer cacheServer.Close()

	for n := 0; n < 5; n++ {
		client := http.Client{}
		resp, err := client.Get(fmt.Sprintf("%s/llamas/%d", cacheServer.URL, n))
		if err != nil {
			panic(err)
		}
		resp.Body.Close()
	}

Inbound
Inbound
Inbound
Inbound
Inbound

httpcache blocks until cacheable upstream responses are buffered into memory

The handler.passUpstream method buffers the entire upstream response into memory and then returns it to the downstream client. This has two huge issues, firstly if the response is large, it can cause memory overruns, secondly the downstream client might timeout whilst waiting.

Ideally what should happen is just the upstream headers should be read, to determine storeability and then the body should be streamed to the downstream client and the cache in parallel.

Still maintained?

Hi thanks for the great software. I see there are some open issues which are untouched since a year. Do you use httpcache in Production? Thanks.

Some content appears to cause panic

I've seen the following panic occur on retrieving context via the proxy, both in my own used of httpcache and via the cli sample (with modified listener port, e.g. 5000 instead of 80).

2016/08/11 14:16:55 [::1] "GET http://127.0.0.1:4000/notifications/recent HTTP/1.1" (OK) 32768 SKIP 131.157149ms

>> GET /notifications/recent HTTP/1.1
>> Host: localhost:8080
>> Accept: */*
>> User-Agent: curl/7.43.0

2016/08/11 14:16:56 GET /notifications/recent not in shared cache
2016/08/11 14:16:56 passing request upstream
2016/08/11 14:16:56 upstream responded headers in 129.992328ms
2016/08/11 14:16:56 resource is uncacheable

<< HTTP/1.1 200 OK
<< Cache-Control: no-store
<< Content-Type: application/atom+xml
<< Date: Thu, 11 Aug 2016 21:16:56 GMT
<< X-Cache: SKIP

2016/08/11 14:16:56 [::1] "GET http://127.0.0.1:4000/notifications/recent HTTP/1.1" (OK) 32768 SKIP 130.166404ms
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x20 pc=0x1e231b]

goroutine 25 [running]:
panic(0x365f80, 0xc82000e0c0)
    /usr/local/go/src/runtime/panic.go:481 +0x3e6
bufio.(*Writer).Write(0xc82006a7c0, 0xc820190000, 0x59d9, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:594 +0xcb
net/http.(*response).write(0xc82018e000, 0x59d9, 0xc820190000, 0x59d9, 0x8000, 0x0, 0x0, 0xc82012e301, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1237 +0x1bf
net/http.(*response).Write(0xc82018e000, 0xc820190000, 0x59d9, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1209 +0x64
github.com/lox/httpcache/httplog.(*responseWriter).Write(0xc8200da210, 0xc820190000, 0x59d9, 0x8000, 0x59d9, 0x0, 0x0)
    /Users/a045103/goeventstore/src/github.com/lox/httpcache/httplog/log.go:38 +0xcb
github.com/lox/httpcache.(*responseStreamer).Write(0xc820071920, 0xc820190000, 0x59d9, 0x8000, 0x59d9, 0x0, 0x0)
    /Users/a045103/goeventstore/src/github.com/lox/httpcache/handler.go:558 +0x9b
io.copyBuffer(0x6bb128, 0xc820071920, 0x6bb150, 0xc82012e4c0, 0xc820190000, 0x8000, 0x8000, 0x8000, 0x0, 0x0)
    /usr/local/go/src/io/io.go:382 +0x2c9
io.CopyBuffer(0x6bb128, 0xc820071920, 0x6bb150, 0xc82012e4c0, 0x0, 0x0, 0x0, 0xc82006c8f8, 0x0, 0x0)
    /usr/local/go/src/io/io.go:361 +0xe1
net/http/httputil.(*ReverseProxy).copyResponse(0xc82006a4c0, 0x6bb128, 0xc820071920, 0x6bb150, 0xc82012e4c0)
    /usr/local/go/src/net/http/httputil/reverseproxy.go:265 +0x299
net/http/httputil.(*ReverseProxy).ServeHTTP(0xc82006a4c0, 0x1309ee0, 0xc820071920, 0xc8200ec2a0)
    /usr/local/go/src/net/http/httputil/reverseproxy.go:242 +0xe2a
github.com/lox/httpcache.(*Handler).passUpstream.func1(0xc820070fc0, 0xc820071920, 0xc82018e0d0)
    /Users/a045103/goeventstore/src/github.com/lox/httpcache/handler.go:254 +0x7c
created by github.com/lox/httpcache.(*Handler).passUpstream
    /Users/a045103/goeventstore/src/github.com/lox/httpcache/handler.go:256 +0x330
exit status 2

I cannot pin down exactly what it is about the content that is causing the crash. The same server runs without error for a long time, but then certain pages being served through the cache cause it to panic almost everytime (sometimes the first fetch works).

Note the content causing the crash is not cacheable - the Cache-Control header is set to no-store.

I've attached a sample that crashes the proxy. Note that I can retrieve the content directly via curl or in a browser without error.

curl.out.zip

Environment information:

go version
go version go1.6.2 darwin/amd64

go env
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH=""
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GO15VENDOREXPERIMENT="1"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fno-common"
CXX="clang++"
CGO_ENABLED="1"

Error EOF when first request for a resource is HEAD?

It seems that httpcache logs an error while attempting to cache a resource, if the first/only request is a HEAD. The error comes from handler.go:385.

Also, this line results in a poorly formatted error - when errorf passes the args to the log function, there is no distinction between err and the elements of keys, so the function thinks that the arg for %s is missing. One workaround is

errorf(fmt.Sprintf("error %s storing resources %%#v", err), keys)