Comments (8)
Here's how to reproduce. Start 2 docker containers:
docker run -p 8080:80 containous/whoami
docker run -p 27017:27017 containous/whoami
Then I would expect the following config to work:
{
"logging": {
"logs": {
"default": {
"level": "DEBUG"
}
}
},
"apps": {
"tls": {
"certificates": {
"automate": ["mongo.127.0.0.1.nip.io"]
},
"automation": {
"policies": [{
"issuer": {
"module": "internal"
},
"key_type": "rsa4096"
}
]
}
},
"layer4": {
"servers": {
"layer4-srv0": {
"listen": ["0.0.0.0:443"],
"routes": [{
"match": [{
"tls": {
"sni": ["mongo.127.0.0.1.nip.io"]
}
}
],
"handle": [{
"handler": "tls"
}, {
"handler": "proxy",
"upstreams": [{
"dial": ["localhost:27017"]
}
]
}
]
}, {
"match": [{
"tls": {
"sni": ["webserver.127.0.0.1.nip.io"]
}
}
],
"handle": [{
"handler": "tls"
},{
"handler": "proxy",
"upstreams": [{
"dial": ["127.0.0.1:8080"]
}
]
}
]
}
]
}
}
}
}
}
To test I first connected to mongo. Worked fine. (Had to use windows client, see screenshots below)
Then I tested curl over https, which failed:
❯ curl https://webserver.127.0.0.1.nip.io/
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number
A non https curl yields:
❯curl http://webserver.127.0.0.1.nip.io:8080/
Hostname: 816bb0554f2c
IP: 127.0.0.1
IP: 172.17.0.3
RemoteAddr: 172.17.0.1:44410
GET / HTTP/1.1
Host: webserver.127.0.0.1.nip.io:8080
User-Agent: curl/7.68.0
Accept: */*
My stop gap solution is proxying the non terminated connections to another port with this config:
{
"logging": {
"logs": {
"default": {
"level": "DEBUG"
}
}
},
"apps": {
"tls": {
"certificates": {
"automate": ["mongo.127.0.0.1.nip.io"]
},
"automation": {
"policies": [{
"issuer": {
"module": "internal"
},
"key_type": "rsa4096"
}
]
}
},
"http": {
"servers": {
"http-srv0": {
"listen": ["127.0.0.1:444", "127.0.0.1:80"],
"routes": [{
"match": [{
"host": ["webserver.127.0.0.1.nip.io"]
}
],
"handle": [{
"handler": "subroute",
"routes": [{
"handle": [{
"handler": "reverse_proxy",
"upstreams": [{
"dial": "127.0.0.1:8080"
}
]
}
]
}
]
}
],
"terminal": true
}
]
}
}
},
"layer4": {
"servers": {
"layer4-srv0": {
"listen": ["0.0.0.0:443"],
"routes": [{
"match": [{
"tls": {
"sni": ["mongo.127.0.0.1.nip.io"]
}
}
],
"handle": [{
"handler": "tls"
}, {
"handler": "proxy",
"upstreams": [{
"dial": ["localhost:27017"]
}
]
}
]
}, {
"match": [{
"tls": {}
}
],
"handle": [{
"handler": "proxy",
"upstreams": [{
"dial": ["127.0.0.1:444"]
}
]
}
]
}
]
}
}
}
}
}
Then curl yields:
❯ curl https://webserver.127.0.0.1.nip.io/
Hostname: 816bb0554f2c
IP: 127.0.0.1
IP: 172.17.0.3
RemoteAddr: 172.17.0.1:44474
GET / HTTP/1.1
Host: webserver.127.0.0.1.nip.io
User-Agent: curl/7.68.0
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 127.0.0.1
X-Forwarded-Proto: https
Another interesting tidbit, if you curl a mongodb port it says that you are trying to access it with http when it should be a native driver:
❯ curl http://mongo.127.0.0.1.nip.io:27017
It looks like you are trying to access MongoDB over HTTP on the native driver port.
However, when we run the same with the "working" config above, it just hangs:
❯ curl https://mongo.127.0.0.1.nip.io
[HANGS indefinitely]
Is caddy-l4 perhaps waiting for a TLS hello indefinitely when it shouldn't? (I am not very familiar with tcp programming)
from caddy-l4.
Here's a simple demonstration without docker of this "issue" (perhaps it's by design). The below script installs caddy-l4 and this tls termination tunnel: https://github.com/cmpxchg16/go-sslterminator . Both rely on caddy to generate local certs. The two tunnels will listen on :8443 (caddy) and :8444 (go-sslterminator), and should terminate ssl and then tunnel traffic to port 8080, which is running this websever: http://github.com/jpillora/serve
CRT_PATH=./storage/certificates/local/webserver.127.0.0.1.nip.io/webserver.127.0.0.1.nip.io.crt
KEY_PATH=./storage/certificates/local/webserver.127.0.0.1.nip.io/webserver.127.0.0.1.nip.io.key
#make dummy file
echo "test script run at $(date)" > now.txt
#kill anything in place
pkill -9 caddy
pkill -9 serve
rm -f ./caddy
rm -rf build
mkdir build
rm -rf storage
mkdir storage
#install caddy
git clone [email protected]:mholt/caddy-l4.git build/caddy/caddy-l4
xcaddy build --with github.com/mholt/caddy-l4=${PWD}/build/caddy/caddy-l4 --output ./caddy
chmod +x ./caddy
#install tunnel alternative
git clone [email protected]:cmpxchg16/go-sslterminator.git build/go-sslterminator
#install simple file server
go get -v github.com/jpillora/serve
serve -p 8080 . &
#run caddy
./caddy run --config config.json &
#wait for certs
while [ ! -f $CRT_PATH ]; do sleep 1; done
while [ ! -f $KEY_PATH ]; do sleep 1; done
#run go-sslterminator
go run ./build/go-sslterminator/go-sslterminator.go -b :8080 -c $CRT_PATH -k $KEY_PATH -l :8444
The config file used by caddy is simple:
{
"storage": {
"module": "file_system",
"root": "./storage"
},
"logging": {
"logs": {
"default": {
"level": "DEBUG"
}
}
},
"apps": {
"tls": {
"certificates": {
"automate": ["webserver.127.0.0.1.nip.io"]
},
"automation": {
"policies": [{
"issuer": {
"module": "internal"
},
"key_type": "rsa4096"
}
]
}
},
"layer4": {
"servers": {
"layer4-srv0": {
"listen": ["0.0.0.0:8443"],
"routes": [{
"match": [{
"tls": {
"sni": ["webserver.127.0.0.1.nip.io"]
}
}
],
"handle": [{
"handler": "tls"
}, {
"handler": "proxy",
"upstreams": [{
"dial": ["localhost:8080"]
}
]
}
]
}
]
}
}
}
}
}
Now, I would expect curling both 8443 and 8444 to have the same response (TLS terminated, and then serve the http content from :8080). That's been my experience to date in http1.1 land with haproxy.
Here's the output from caddy:
❯ curl -v https://webserver.127.0.0.1.nip.io:8443/now.txt
* Trying 127.0.0.1:8443...
* TCP_NODELAY set
* Connected to webserver.127.0.0.1.nip.io (127.0.0.1) port 8443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: [NONE]
* start date: Oct 6 21:24:49 2020 GMT
* expire date: Oct 7 09:24:49 2020 GMT
* subjectAltName: host "webserver.127.0.0.1.nip.io" matched cert's "webserver.127.0.0.1.nip.io"
* issuer: CN=Caddy Local Authority - ECC Intermediate
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fffdbf8baa0)
> GET /now.txt HTTP/2
> Host: webserver.127.0.0.1.nip.io:8443
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* http2 error: Remote peer returned unexpected data while we expected SETTINGS frame. Perhaps, peer does not support HTTP/2 properly.
* Connection #0 to host webserver.127.0.0.1.nip.io left intact
curl: (16) Error in the HTTP2 framing layer
Here's the output from go-sslterminator:
❯ curl -v https://webserver.127.0.0.1.nip.io:8444/now.txt
* Trying 127.0.0.1:8444...
* TCP_NODELAY set
* Connected to webserver.127.0.0.1.nip.io (127.0.0.1) port 8444 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: [NONE]
* start date: Oct 6 21:24:49 2020 GMT
* expire date: Oct 7 09:24:49 2020 GMT
* subjectAltName: host "webserver.127.0.0.1.nip.io" matched cert's "webserver.127.0.0.1.nip.io"
* issuer: CN=Caddy Local Authority - ECC Intermediate
* SSL certificate verify ok.
> GET /now.txt HTTP/1.1
> Host: webserver.127.0.0.1.nip.io:8444
> User-Agent: curl/7.68.0
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Content-Length: 48
< Content-Type: text/plain; charset=utf-8
< Last-Modified: Tue, 06 Oct 2020 21:34:23 GMT
< Date: Tue, 06 Oct 2020 21:34:28 GMT
<
test script run at Tue Oct 6 17:34:23 EDT 2020
* Connection #0 to host webserver.127.0.0.1.nip.io left intact
Looking at the diff I can see that the difference is probably the "server accepted to use h2" part. I am not very familiar with http2, but I am assuming it's because caddy is using http2 and the file server above is not, like you suggested earlier.
I looked through the docs to see if there is a way to disable http2 globally in caddy, but can't find it. Although I'm not sure if that would apply given that this is an add-on.
I then saw your comment and can confirm it works correctly when disabling http2 in my curl command:
❯ curl -v --http1.1 https://webserver.127.0.0.1.nip.io:8443/now.txt
* Trying 127.0.0.1:8443...
* TCP_NODELAY set
* Connected to webserver.127.0.0.1.nip.io (127.0.0.1) port 8443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: [NONE]
* start date: Oct 6 21:49:23 2020 GMT
* expire date: Oct 7 09:49:23 2020 GMT
* subjectAltName: host "webserver.127.0.0.1.nip.io" matched cert's "webserver.127.0.0.1.nip.io"
* issuer: CN=Caddy Local Authority - ECC Intermediate
* SSL certificate verify ok.
> GET /now.txt HTTP/1.1
> Host: webserver.127.0.0.1.nip.io:8443
> User-Agent: curl/7.68.0
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Content-Length: 48
< Content-Type: text/plain; charset=utf-8
< Last-Modified: Tue, 06 Oct 2020 21:52:42 GMT
< Date: Tue, 06 Oct 2020 21:52:42 GMT
<
test script run at Tue Oct 6 17:49:09 EDT 2020
* Connection #0 to host webserver.127.0.0.1.nip.io left intact
Perhaps this is by design, but it'd be great to have one of the following options (in order of my personal preference):
- Detect if backend can't support http2 and disable if so (I'm going to assume this is a big undertaking, and a no-go)
- Have a special "http" "https" handler that will terminate ssl before moving on the "proxy" handler
- Disable http2 on a per upstream basis
- Disable http2 globally
For now I think I can just force clients to use http1.1, or create a local tunnel to an http server in my caddy file as a work around. For example, I think this configuration performs TLS termination in the Http app, which can handle http2, making this a non-issue even though the back end server doesn't.
{
"storage": {
"module": "file_system",
"root": "./storage"
},
"logging": {
"logs": {
"default": {
"level": "DEBUG"
}
}
},
"apps": {
"tls": {
"certificates": {
"automate": ["webserver.127.0.0.1.nip.io"]
},
"automation": {
"policies": [{
"issuer": {
"module": "internal"
},
"key_type": "rsa4096"
}
]
}
},
"http": {
"servers": {
"server0": {
"listen": ["localhost:8445"],
"routes": [{
"match": [{
"host": ["webserver.127.0.0.1.nip.io"]
}
],
"handle": [{
"handler": "reverse_proxy",
"transport": {
"protocol": "http"
},
"upstreams": [{
"dial": "localhost:8080"
}
]
}
]
}
]
}
}
},
"layer4": {
"servers": {
"layer4-srv0": {
"listen": ["0.0.0.0:8443"],
"routes": [{
"match": [{
"tls": {
"sni": ["webserver.127.0.0.1.nip.io"]
}
}
],
"handle": [{
"handler": "proxy",
"upstreams": [{
"dial": ["localhost:8445"]
}
]
}
]
}
]
}
}
}
}
}
Here's the curl for the above config:
❯ curl -v https://webserver.127.0.0.1.nip.io:8443/now.txt
* Trying 127.0.0.1:8443...
* TCP_NODELAY set
* Connected to webserver.127.0.0.1.nip.io (127.0.0.1) port 8443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: [NONE]
* start date: Oct 6 21:49:23 2020 GMT
* expire date: Oct 7 09:49:23 2020 GMT
* subjectAltName: host "webserver.127.0.0.1.nip.io" matched cert's "webserver.127.0.0.1.nip.io"
* issuer: CN=Caddy Local Authority - ECC Intermediate
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fffb7681aa0)
> GET /now.txt HTTP/2
> Host: webserver.127.0.0.1.nip.io:8443
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 200
< accept-ranges: bytes
< content-type: text/plain; charset=utf-8
< date: Tue, 06 Oct 2020 22:13:49 GMT
< last-modified: Tue, 06 Oct 2020 22:13:49 GMT
< server: Caddy
< content-length: 48
<
test script run at Tue Oct 6 17:49:09 EDT 2020
* Connection #0 to host webserver.127.0.0.1.nip.io left intact
Hope this insight helps, or if it's all by design, feel free to close this!
from caddy-l4.
Figured out a stopgap solution. I can tunnel the TLS connection to an http app running on another port with the proxy handler. It'd be great if we could skip this step, perhaps with an "https" handler.
from caddy-l4.
Hmm good q, lemme look into that tomorrow!
Can you post which curl request yields the error? I'll try to reproduce it.
from caddy-l4.
Also, FWIW here's what I get in chrome on https://webserver.127.0.0.1
from caddy-l4.
Thanks for the detailed instructions!
I've been having trouble reproducing this locally. (For one, I don't/can't use Docker, so I have my own upstreams that I proxy to.)
I haven't been able to get this error:
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number
What version of curl and openssl do you have?
In fact, your config -- as far as I can tell -- works fine for me. When I curl -v "https://localhost:5443"
(again, I set up my own local backend) I got an error about HTTP/2 framing, then I remembered that my backend doesn't use HTTP/2 (it is a plaintext HTTP server, which doesn't support H2C), so I used the --http1.1
flag with curl and then everything worked fine.
What is the output if you use curl -v
, and what are the server-side logs?
from caddy-l4.
Great writeup, thank you!
Looking at the diff I can see that the difference is probably the "server accepted to use h2" part. I am not very familiar with http2, but I am assuming it's because caddy is using http2 and the file server above is not, like you suggested earlier.
Indeed; Caddy's default TLS configuration sets the following values in the NextProtos field as ALPN:
var defaultALPN = []string{"h2", "http/1.1"}
Server ALPN basically tells the client connecting to a TLS server what (application-layer) protocols the server can speak after TLS is negotiated. As you can see we enable h2 by default because that works with most services. However, older or less capable software won't support HTTP/2.
Since Project Conncept is a layer 4 proxy, it terminates the transport / TLS, but it does not interpret application packets like HTTP. For that, you'd of course use Caddy's HTTP proxy which is already built-in: it can read an HTTP/2 request from the client, decode it, then re-encode it as HTTP/1.1 for the upstream backend that may not support HTTP/2. And this negotiation happens automatically.
But with a layer 4 proxy, it just terminates TLS and then will shuttle raw bytes across the wire, so if it negotiated HTTP/2 with the client, the client will send HTTP/2 packets which get proxied, uninterpreted, to the backend.
I think the solution in your case might be to remove "h2" from the ALPN values. This is done in the l4 module the same way it is in the HTTP server, that is, with a TLS connection policy: https://caddyserver.com/docs/json/apps/http/servers/tls_connection_policies/#alpn -- except, for l4, you'd do it in the connection_policies
field:
{
"handler": "tls",
"connection_policies": [
{"alpn": ["http/1.1"]}
}
}
Give that a shot and see if that works better.
(Note that the structure of each connection policy and the features that are available to you are the exact same as those in the HTTP server, I just can't link to docs for this app because it's not public yet. So basically, this means you can customize the TLS server in other ways too, like filtering policies by ServerName, or changing other things about the TLS connection.)
from caddy-l4.
Closing due to inactivity. Assuming resolved, I guess.
from caddy-l4.
Related Issues (20)
- Switch to `go-proxyproto` HOT 4
- "Connection refused" when proxying UDP port HOT 2
- How to match on StartTLS for proxying Postgres? HOT 9
- Can't build the docker image HOT 1
- High CPU usage HOT 11
- how to graceful restart or shutdown HOT 2
- PR#141 Broke Matchers HOT 4
- How to match TLS SNI with proxy_protocol? HOT 2
- http active health check HOT 1
- [FEATURE]: Add support for access logs HOT 4
- http/2 corruption HOT 2
- Non blocking matchers & matching timeout breaking l4 as a listener wrapper HOT 13
- Collection of interesting configs HOT 1
- Feature request: `invoke` handler HOT 2
- Regression: PROXY protocol line sent only when first data arrives from client HOT 7
- Accept connections with an without PROXY protocol on the same port HOT 1
- Is it possible to use caddy's modules inside L4 routes? HOT 12
- SIGHUP for config reload? HOT 5
- Caddyfile does not recognize `layer4` global option HOT 8
- Cannot proxy to Mariadb instance HOT 8
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from caddy-l4.