fpco/http-reverse-proxy

Reverse proxy HTTP requests, either over raw sockets or with WAI

http-reverse-proxy

Provides a simple means of reverse-proxying HTTP requests. The raw approach uses the same technique as leveraged by keter, whereas the WAI approach performs full request/response parsing via WAI and http-conduit.

Raw example

The following sets up raw reverse proxying from local port 3000 to www.example.com, port 80.

{-# LANGUAGE OverloadedStrings #-}
import Network.HTTP.ReverseProxy
import Data.Conduit.Network

main :: IO ()
main = runTCPServer (serverSettings 3000 "*") $ \appData ->
    rawProxyTo
        (\_headers -> return $ Right $ ProxyDest "www.example.com" 80)
        appData

HTTPS example to proxy bing.com

The following example sets up reverse proxying froming to www.bing.com from localhost port 3000:

import Network.HTTP.Client.TLS
import Network.HTTP.ReverseProxy
import Network.Wai
import Network.Wai.Handler.Warp (run)

main :: IO ()
main = bingExample >>= run 3000

bingExample :: IO Application
bingExample = do
  manager <- newTlsManager
  pure $
    waiProxyToSettings
      ( \request ->
          return $
            WPRModifiedRequestSecure
              ( request
                  { requestHeaders = [("Host", "www.bing.com")]
                  }
              )
              (ProxyDest "www.bing.com" 443)
      )
      defaultWaiProxySettings {wpsLogRequest = print}
      manager

After running it, you can visit http://localhost:3000 to visit the search engine's page.

http-reverse-proxy's Issues

https and change url support for 0.2 release

The https ProxyDest I get

Error connecting to gateway:
InvalidStatusLine "\NAK\ETX\NUL\NUL\STX\STX"

1. I would like to proxy
   http://localhost/index.html or https://localhost/index.html to
   https://lb1.pcs.ovh.net/v1/AUTH_a4c2ef109c996858a5c3f8e411de8538/gert/index.html

2. If possible redirection to multiple hosts for example
   https://localhost/server1/index.html -> https://server1/index.html
   https://localhost/server2/index.html -> https://server2/index.html
   https://localhost/server3/index.html -> warp staticApp

WrongRequestBodyStreamSize N 0 when proxying POST request with body

I'm encountering the same issues described in this StackOverflow question: https://stackoverflow.com/questions/50103208/haskell-reverse-proxy-issue

The problem arises when you use waiProxyTo to proxy a POST request with a non-empty body. You get WrongRequestBodyStreamSize N 0 where N is the actual size of the body. The exception is coming from the check here:

https://github.com/snoyberg/http-client/blob/fda028b633cc000d295caf1ad0bdbcde98ad5b6e/http-client/Network/HTTP/Client/Request.hs#L447

It seems to happen non-deterministically. I've tried to set up a simple repro but haven't found a simple way to trigger it outside of my more complex testing setup.

I have a solution which I'll post as a separate PR.

A bit more low-level proxy

A version of proxy (rawProxyTo) which does not look into headers, so it can work for other TCP-like protocols.

`wpsProcessBody` doesn't allow one to read the response body

Background

• I have a reverse proxying WAI application where I wish to log the response body from 'backend' applications when response status is 500 and modify the (outgoing) response body such that the client only sees "Something went wrong".
• The 'backend' applications are designed such that they send the actual exception messages in their response body with status code set to 500.

Problem

But wpsProcessBody doesn't give me the actual response body due to Response () in its type signature.

Is there a way to work around this in my reverse proxying application OR should we fix this in this package?

http-reverse-proxy-0.5.0 test suite build failure with http-conduit-2.3

As seen on the Stackage build server:

Building test suite 'test' for http-reverse-proxy-0.5.0..
[1 of 1] Compiling Main             ( test/main.hs, dist/build/test/test-tmp/Main.o )

test/main.hs:89:11: error:
    Not in scope: ‘HC.withManager’
    Perhaps you meant ‘HC.newManager’ (imported from Network.HTTP.Conduit)
    Module ‘Network.HTTP.Conduit’ does not export ‘withManager’.
   |
89 | withMan = HC.withManager . (liftIO .)
   |           ^^^^^^^^^^^^^^

I was able to reproduce this locally like so:

stack unpack http-reverse-proxy-0.5.0 && cd http-reverse-proxy-0.5.0
edit stack.yaml # add the following stack.yaml
stack test

# stack.yaml
resolver: nightly-2018-03-10
extra-deps:
- http-conduit-2.3.0

I will mark this as an expected test failure in Stackage.

Reverse proxy for http/https-requests

Hi @snoyberg! Could you help me, please?

I need to proxify particular http/https-requests to particular IP/PORT. I use http-reverse-proxy-0.4.3 and warp-tls-3.1.3. I.e.:

Firefox -> http://www.example.com/  -> http-reverse-proxy -> 123.124.125.126:80
Firefox -> https://www.example.com/ -> http-reverse-proxy -> 123.124.125.126:443

where 123.124.125.126 is NOT an IP-address of www.example.com.

I use PAC-based proxy settings for Firefox, and this is my .pac-file:

function FindProxyForURL(url,host) {
    if (shExpMatch(url, "http://www.example.com*")) {
        return "PROXY 127.0.0.1:8080";
    }
    if (shExpMatch(url, "https://www.example.com*")) {
        return "PROXY 127.0.0.1:8081";
    }
    return "DIRECT";
}

Of course, reverse-proxy for http-requests works perfectly, based on this example: https://github.com/fpco/http-reverse-proxy#raw-example:

{-# LANGUAGE OverloadedStrings #-}
import Network.HTTP.ReverseProxy
import Data.Conduit.Network

main :: IO ()
main = runTCPServer (serverSettings 8080 "*") $ \appData ->
    rawProxyTo
        (\_headers -> return $ Right $ ProxyDest "123.124.125.126" 80)
        appData

But for https-requests it's not so easy. This is my code:

{-# LANGUAGE OverloadedStrings #-}

import           Network.HTTP.ReverseProxy 
import qualified Network.Wai                    as WAI
import qualified Network.Wai.Handler.Warp       as Warp
import           Network.Wai.Handler.WarpTLS
import           Network.HTTP.Client

main :: IO ()
main = do
    manager <- newManager defaultManagerSettings
    let aProxyApp = waiProxyTo modifyRequest 
                               defaultOnExc 
                               manager
    runTLS (tlsSettings "/tmp/certificate.pem"
                        "/tmp/key.pem") { onInsecure = AllowInsecure } 
           (Warp.setPort 8081 Warp.defaultSettings) 
           aProxyApp

modifyRequest :: WAI.Request -> IO WaiProxyResponse
modifyRequest requestFromBrowser = 
    let request = requestFromBrowser { WAI.isSecure = True } in
    return $ WPRModifiedRequestSecure request $ ProxyDest "123.124.125.126" 443

Files certificate.pem and key.pem were created like this:

openssl genrsa -out key.pem 2048
openssl req -new -key key.pem -out certificate.csr
openssl x509 -req -in certificate.csr -signkey key.pem -out certificate.pem

But I've got an error:

502 Bad Gateway

How can I fix it?

Mac OS X stackage build failure

When attempting to build this package as part of my stackage configuration on Mac OS X, I consistently get the following test-suite failures:

[1 of 1] Compiling Network.HTTP.ReverseProxy ( Network/HTTP/ReverseProxy.hs, dist/build/Network/HTTP/ReverseProxy.o )
In-place registering http-reverse-proxy-0.1.0.6...
Preprocessing test suite 'test' for http-reverse-proxy-0.1.0.6...
[1 of 1] Compiling Main             ( test/main.hs, dist/build/test/test-tmp/Main.o )
Linking dist/build/test/test ...
Running 1 test suites...
Test suite test: RUNNING...

http-reverse-proxy
  - works FAILED [1]
  - deals with streaming data FAILED [2]

waiToRaw
  - works FAILED [3]
  - sends files FAILED [4]

1) http-reverse-proxy works FAILED (uncaught exception)
GHC.IO.Exception.IOException (connect: does not exist (Connection refused))

2) http-reverse-proxy deals with streaming data FAILED (uncaught exception)
GHC.IO.Exception.IOException (connect: does not exist (Connection refused))

3) waiToRaw works FAILED (uncaught exception)
GHC.IO.Exception.IOException (connect: does not exist (Connection refused))

4) waiToRaw sends files FAILED (uncaught exception)
GHC.IO.Exception.IOException (connect: does not exist (Connection refused))

Finished in 0.4223 seconds
4 examples, 4 failures
Test suite test: FAIL
Test suite logged to: dist/test/http-reverse-proxy-0.1.0.6-test.log
0 of 1 test suites (0 of 1 test cases) passed.

I wonder if its something to do with the way you set up ports or some mac default firewall config?

proxying https 443

Hi,

is it possible to act as a HTTPS server and then reverse proxy to a non-HTTPS app?

browser --- https ---> http-reverse-proxy ----http---> appWithNonHttpsPort

If yes, how to configure the certificate and key file in the http-reverse-proxy?

thanks,
Alex.

Document a rawProxyTo usage example

It would be super helpful to see what a reverse proxy server main using rawProxyTo looks like. I'm probably being dense, but after clicking around and looking at the source I'm quite confused.

http-reverse-proxy-0.5.0.1 test suite build failure

Hi! After trying to unblock http-reverse-proxy and see if there is still a build failure with GHC 8.4 I've run into the following:

http-reverse-proxy-0.5.0.1: build (lib + test)
Preprocessing library for http-reverse-proxy-0.5.0.1..
Building library for http-reverse-proxy-0.5.0.1..
Preprocessing test suite 'test' for http-reverse-proxy-0.5.0.1..
Building test suite 'test' for http-reverse-proxy-0.5.0.1..
[1 of 1] Compiling Main             ( test/main.hs, .stack-work/dist/x86_64-linux-nix/Cabal-2.2.0.1/build/test/test-tmp/Main.o )

/home/alexeyzab/code/http-reverse-proxy-0.5.0.1/test/main.hs:127:43: error:
    • No instance for (monad-control-1.0.2.3:Control.Monad.Trans.Control.MonadBaseControl
                         IO (Control.Monad.Trans.Resource.Internal.ResourceT IO))
        arising from a use of ‘timeout’
    • In the expression: timeout 1000000
      In the second argument of ‘($)’, namely
        ‘timeout 1000000
           $ do res <- HC.http req manager
                HC.responseBody res $$ await’
      In a stmt of a 'do' block:
        mbs <- runResourceT
                 $ timeout 1000000
                     $ do res <- HC.http req manager
                          HC.responseBody res $$ await
    |
127 |                     mbs <- runResourceT $ timeout 1000000 $ do
    |                                           ^^^^^^^^^^^^^^^

Progress: 1/2
--  While building custom Setup.hs for package http-reverse-proxy-0.5.0.1 using:
      /home/alexeyzab/.stack/setup-exe-cache/x86_64-linux-nix/Cabal-simple_mPHDZzAJ_2.2.0.1_ghc-8.4.2 --builddir=.stack-work/dist/x86_64-linux-nix/Cabal-2.2.0.1 build lib:http-reverse-proxy test:test --ghc-options " -ddump-hi -ddump-to-file -fdiagnostics-color=always"
    Process exited with code: ExitFailure 1

Steps to reproduce:

stack unpack http-reverse-proxy-0.5.0.1
cd http-reverse-proxy-0.5.0.1
stack init --resolver nightly-2018-04-25
stack build --resolver nightly-2018-04-25 --haddock --test --bench --no-run-benchmarks

I am going to skip the tests for this package on Stackage in the meantime.

EventSource connections

Hey!

After a long time without being able to use yesod devel, I've stopped using hsenv and tried to put yesod devel to work again. It seems to be working fine on everything but EventSources.

Using wireshark I was able to see that the app server is correctly returning the event stream. However, the devel proxy is not returning anything.

It seems that the proxy tries to gunzip the response, so I've tried to set http-conduit's rawBody to True. The error persisted.

Any ideas? If not, I may try to create a simple test case.

Reverse proxy spilling beans in response.

After replacing haproxy with http-reverse-proxy (which works much better!), I'm seeing one thing I don't like.

So the context is that I am proxying a request to gitlab to make my stateless setup seem stateful. It fails, but then the user gets to know that I proxied to gitlab! That's not a very loyal behavior as that's supposed to be my little secret!

Log to stderr if something is wrong, don't spill the beans to my customers!

curl -v -X PUT --header 'Private-Token: xxx'      "https://api.example.com/v1/config/example?content=$(cat myconf.conf | curl -Gso /dev/null -w %{url_effective} --data-urlencode @- '' | cut -c 3-)"
*   Trying 1.1.1.1...
* TCP_NODELAY set
* Connected to api.example.com (1.1.1.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=api.example.com
*  start date: Aug 28 01:30:46 2018 GMT
*  expire date: Nov 26 01:30:46 2018 GMT
*  subjectAltName: host "api.example.com" matched cert's "api.example.com"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
> PUT /v1/config/example?content=blahblahblah HTTP/1.1
> Host: api.example.com
> User-Agent: curl/7.58.0
> Accept: */*
> Private-Token: xxx
>
< HTTP/1.1 502 Bad Gateway
< Transfer-Encoding: chunked
< Date: Sat, 08 Sep 2018 06:40:39 GMT
< Server: Apache
< content-type: text/plain
<
Error connecting to gateway:

HttpExceptionRequest Request {
  host                 = "gitlab.com"
  port                 = 443
  secure               = True
  requestHeaders       = [("X-Real-IP","10.255.0.2"),("Host","gitlab.com"),("User-Agent","curl/7.58.0"),("Accept","*/*"),("Private-Token","xxx")]
  path                 = "/api/v4/projects/1231231/repository/files/myconf.conf"
  queryString          = "?branch=master&commit_message=update&content=blahblah"
  method               = "PUT"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 0
  responseTimeout      = ResponseTimeoutNone
  requestVersion       = HTTP/1.1
}
* Connection #0 to host api.example.com left intact

fixReqHeaders may create weird x-real-ip header

x-forwarded-for header may contain more than one ip with comma seperate.
The first ip is the real client ip.
But x-real-ip should only have one ip.
The fixReqHeaders function put the whole x-forwarded-for into x-real-ip.

renderHeaders assumes rawPathInfo is an absolute path

When renderHeaders creates a request it assumes rawPathInfo is a absolute path. If it is a relative path the Request-Uri part of the HTTP request will be invalid.

I don't think wai makes any guarantees about the rawPathInfo so I don't think http-reverse-proxy should either.

I think renderHeaders should use the logic in http-client http://hackage.haskell.org/package/http-client-0.5.14/docs/src/Network.HTTP.Client.Request.html#local-6989586621679091539

Here is the logic I am referring to:

            <> (case S8.uncons $ path req of
                    Just ('/', _) -> fromByteString $ path req
                    _ -> fromChar '/' <> fromByteString (path req))

install failures

At first glance this seems like an issue of classy-prelude not being pegged. I would suggest have mono-traversable rather than classy-prelude as a dependency.

Configuring http-reverse-proxy-0.1.1.6...
Building http-reverse-proxy-0.1.1.6...
Preprocessing library http-reverse-proxy-0.1.1.6...
[1 of 2] Compiling Paths_http_reverse_proxy ( dist/dist-sandbox-3b9560ea/build/autogen/Paths_http_reverse_proxy.hs, dist/dist-sandbox-3b9560ea/build/Paths_http_reverse_proxy.o )
[2 of 2] Compiling Network.HTTP.ReverseProxy ( Network/HTTP/ReverseProxy.hs, dist/dist-sandbox-3b9560ea/build/Network/HTTP/ReverseProxy.o )

Network/HTTP/ReverseProxy.hs:142:24: Not in scope: `HC.def'

Network/HTTP/ReverseProxy.hs:160:31:
    Not in scope: data constructor `HC.RequestBodySourceChunked'
    Perhaps you meant `HC.RequestBodyStreamChunked' (imported from Network.HTTP.Conduit)

Network/HTTP/ReverseProxy.hs:164:46:
    Not in scope: data constructor `HC.RequestBodySource'
    Perhaps you meant `HC.RequestBodyStream' (imported from Network.HTTP.Conduit)

Network/HTTP/ReverseProxy.hs:199:24:
    Not in scope: `empty'
    Perhaps you meant one of these:
      `S8.empty' (imported from Data.ByteString.Char8),
      `mempty' (imported from ClassyPrelude)

Network/HTTP/ReverseProxy.hs:204:32:
    Not in scope: `append'
    Perhaps you meant one of these:
      `S8.append' (imported from Data.ByteString.Char8),
      `mappend' (imported from ClassyPrelude)

Building http-reverse-proxy-0.2.1.1...
Failed to install http-reverse-proxy-0.2.1.1
Last 10 lines of the build log ( /home/docmunch/docmunch/webapp/.cabal-sandbox/logs/http-reverse-proxy-0.2.1.1.log ):

Network/HTTP/ReverseProxy.hs:196:24: Not in scope: `HC.def'

Network/HTTP/ReverseProxy.hs:222:31:
    Not in scope: data constructor `HC.RequestBodySourceChunked'
    Perhaps you meant `HC.RequestBodyStreamChunked' (imported from Network.HTTP.Conduit)

Network/HTTP/ReverseProxy.hs:226:46:
    Not in scope: data constructor `HC.RequestBodySource'
    Perhaps you meant `HC.RequestBodyStream' (imported from Network.HTTP.Conduit)

Documentation problem

The docs for waiProxyTo say about the first parameter:

How to reverse proxy. A Left result will be sent verbatim as the response, whereas Right will cause a reverse proxy.

but that first parameter doesn't contain an Either.

waiProxyTo hangs when the server responds with HTTP 103 Early Hints

HTTP 103 Early Hints is a newish HTTP feature where the server can indicate assets that the client can begin downloading immediately, before the server finishes its response.

It does this by sending 103 Early Hint followed by a Link: header.

I made a full repro here: https://github.com/codedownio/http-reverse-proxy/tree/http-103-early-hints-repro

The repro starts up a Node.js server which serves two routes, /simple and /early_hints. It also contains a tiny Haskell app which uses Warp to proxy to the Node.js server.

The /early_hints route sends early hints, and causes waiProxyTo to fail. The repro README describes how to run it and shows the output of curl -v against the different endpoints.

WAI reverse proxy eats + in query string

I have an application that passes inputs to itself via query string parameters. These inputs often contain the "+" character (they are brainfuck programs, so this is absolutely inevitable).

When I run the application standalone (or talk to it directly via the local port it listens on, e.g. http://localhost:5500/), those "+" characters get percent-encoded into %2B by the browser's handling of GET forms (which is correct and desirable), and the server correctly parses the "+" back out and feeds it back into the form. E.g., writing the program +++++++++++++++++++++++++++++++++. into the form and submitting it produces the following URL: http://localhost:5500/?program=%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B.&input=. The program is executed and reflected correctly (it prints "!", and the original program appears in the form as it was entered). Debug-tracing the program request parameter (read using param from scotty) shows that it is parsed as "+++++++++++++++++++++++++++++++++.", so we can see that everything works correctly.

However, running the application behind a WAI reverse proxy that maps incoming requests by subdomain, using a stateful lookup table maintained in an MVar by the parent application, I get different behavior. The URL still looks correct, e.g.: http://pxzfeaiiuzbmfvbb.redacted.example.org/?program=%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B%2B.&input=. Same query string. However, the reflected program appears as ., and prints as " ." in the debug log, suggesting that the query string doesn't survive the reverse proxying unchanged. I fully expect the reverse proxy to potentially change the encoding of the string - e.g., it would be valid to percent-encode spaces previously given as literal + in the URL. But what seems to be happening here is that the reverse proxy parses the request, converting %2B into + (which is correct), but then fails to reverse the decoding when passing it to the proxied child application, keeping the + as such instead of percent-encoding it. The child application then sees a literal + in the encoded query string parameter, and decodes it into a space (which is correct behavior in itself).

In short:

• Setup: use waiProxyTo to reverse-proxy requests by mapping subdomains to ports (WPRProxyDest). Payload application reads brainfuck programs from query string and reflects those in HTML.
• Expected: + characters in cleartext input are encoded as %2B in the client-side URL, presented to the proxied payload app as %2B, and decode and reflect as +.
• Observed: + characters in cleartext input are encoded as %2B in the client-side URL, but reflect as (space).

v0.4.2 build failure on GHC 7.4 due to LambdaCase

Opened an issue since I don't know if you care about 7.4...

Revision: https://hackage.haskell.org/package/http-reverse-proxy-0.4.2/revisions/

error: missing binary operator before token "("

My C pre-processor is struggling with the main test code?

GHCi, version 7.6.3: http://www.haskell.org/ghc/  :? for help
Loading package ghc-prim ... linking ... done.
Loading package integer-gmp ... linking ... done.
Loading package base ... linking ... done.

main.hs:102:0:
     error: missing binary operator before token "("
     #if MIN_VERSION_wai(1, 4, 0)
     ^
phase `C pre-processor' failed (exitcode = 1)
> 

If I remove all pre-processor stuff the code runs.

Be able to process a response's body based on its request.

Greetings,

I have a need where I got to modify a response only when its request meets some requirements. I'm keen to submit a PR, the change looks trivial. WaiProxySettings 's wpsProcessBody record field seems to be the right place.

However, that patch will introduce a breaking change. Would you agree to accept such patch?

Regards.

WPRModifiedRequest using old style rawpathInfo

According to documents of WAI on hackage, we see that WAI recommends to use pathInfo instead of rawPathInfo to modify path in a request. However, this package seems to use rawPathInfo in function WPRModifiedRequest and WPRModifiedRequestSecure when I try to modify the path info. So is it a historical issues? Is it going to be fixed to support new-style pathInfo?