Giter Site home page Giter Site logo

Https Termination about caddy-l4 HOT 8 CLOSED

mholt avatar mholt commented on September 28, 2024 1
Https Termination

from caddy-l4.

Comments (8)

regbo avatar regbo commented on September 28, 2024 1

Here's how to reproduce. Start 2 docker containers:

docker run -p 8080:80 containous/whoami
docker run -p 27017:27017 containous/whoami

Then I would expect the following config to work:

{
	"logging": {
		"logs": {
			"default": {
				"level": "DEBUG"
			}
		}
	},
	"apps": {
		"tls": {
			"certificates": {
				"automate": ["mongo.127.0.0.1.nip.io"]
			},
			"automation": {
				"policies": [{
						"issuer": {
							"module": "internal"
						},
						"key_type": "rsa4096"
					}
				]
			}
		},
		"layer4": {
			"servers": {
				"layer4-srv0": {
					"listen": ["0.0.0.0:443"],
					"routes": [{
							"match": [{
									"tls": {
										"sni": ["mongo.127.0.0.1.nip.io"]
									}
								}
							],
							"handle": [{
									"handler": "tls"
								}, {
									"handler": "proxy",
									"upstreams": [{
											"dial": ["localhost:27017"]
										}
									]
								}
							]
						}, {
							"match": [{
									"tls": {
										"sni": ["webserver.127.0.0.1.nip.io"]
									}
								}
							],
							"handle": [{
									"handler": "tls"
								},{
									"handler": "proxy",
									"upstreams": [{
											"dial": ["127.0.0.1:8080"]
										}
									]
								}
							]
						}
					]
				}
			}
		}

	}
}

To test I first connected to mongo. Worked fine. (Had to use windows client, see screenshots below)

Then I tested curl over https, which failed:

❯ curl https://webserver.127.0.0.1.nip.io/
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

A non https curl yields:

❯curl http://webserver.127.0.0.1.nip.io:8080/
Hostname: 816bb0554f2c
IP: 127.0.0.1
IP: 172.17.0.3
RemoteAddr: 172.17.0.1:44410
GET / HTTP/1.1
Host: webserver.127.0.0.1.nip.io:8080
User-Agent: curl/7.68.0
Accept: */*

My stop gap solution is proxying the non terminated connections to another port with this config:

{
	"logging": {
		"logs": {
			"default": {
				"level": "DEBUG"
			}
		}
	},
	"apps": {
		"tls": {
			"certificates": {
				"automate": ["mongo.127.0.0.1.nip.io"]
			},
			"automation": {
				"policies": [{
						"issuer": {
							"module": "internal"
						},
						"key_type": "rsa4096"
					}
				]
			}
		},
		"http": {
			"servers": {
				"http-srv0": {
					"listen": ["127.0.0.1:444", "127.0.0.1:80"],
					"routes": [{
							"match": [{
									"host": ["webserver.127.0.0.1.nip.io"]
								}
							],
							"handle": [{
									"handler": "subroute",
									"routes": [{
											"handle": [{
													"handler": "reverse_proxy",
													"upstreams": [{
															"dial": "127.0.0.1:8080"
														}
													]
												}
											]
										}
									]
								}
							],
							"terminal": true
						}
					]
				}
			}
		},
		"layer4": {
			"servers": {
				"layer4-srv0": {
					"listen": ["0.0.0.0:443"],
					"routes": [{
							"match": [{
									"tls": {
										"sni": ["mongo.127.0.0.1.nip.io"]
									}
								}
							],
							"handle": [{
									"handler": "tls"
								}, {
									"handler": "proxy",
									"upstreams": [{
											"dial": ["localhost:27017"]
										}
									]
								}
							]
						}, {
							"match": [{
									"tls": {}
								}
							],
							"handle": [{
									"handler": "proxy",
									"upstreams": [{
											"dial": ["127.0.0.1:444"]
										}
									]
								}
							]
						}
					]
				}
			}
		}

	}
}

Then curl yields:

❯ curl https://webserver.127.0.0.1.nip.io/
Hostname: 816bb0554f2c
IP: 127.0.0.1
IP: 172.17.0.3
RemoteAddr: 172.17.0.1:44474
GET / HTTP/1.1
Host: webserver.127.0.0.1.nip.io
User-Agent: curl/7.68.0
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 127.0.0.1
X-Forwarded-Proto: https

Another interesting tidbit, if you curl a mongodb port it says that you are trying to access it with http when it should be a native driver:

❯ curl http://mongo.127.0.0.1.nip.io:27017
It looks like you are trying to access MongoDB over HTTP on the native driver port.

However, when we run the same with the "working" config above, it just hangs:

❯ curl https://mongo.127.0.0.1.nip.io
[HANGS indefinitely]

Is caddy-l4 perhaps waiting for a TLS hello indefinitely when it shouldn't? (I am not very familiar with tcp programming)

conenct
config
test

from caddy-l4.

regbo avatar regbo commented on September 28, 2024 1

Here's a simple demonstration without docker of this "issue" (perhaps it's by design). The below script installs caddy-l4 and this tls termination tunnel: https://github.com/cmpxchg16/go-sslterminator . Both rely on caddy to generate local certs. The two tunnels will listen on :8443 (caddy) and :8444 (go-sslterminator), and should terminate ssl and then tunnel traffic to port 8080, which is running this websever: http://github.com/jpillora/serve

CRT_PATH=./storage/certificates/local/webserver.127.0.0.1.nip.io/webserver.127.0.0.1.nip.io.crt
KEY_PATH=./storage/certificates/local/webserver.127.0.0.1.nip.io/webserver.127.0.0.1.nip.io.key

#make dummy file
echo "test script run at $(date)" > now.txt

#kill anything in place
pkill -9 caddy
pkill -9 serve
rm -f ./caddy
rm -rf build
mkdir build
rm -rf storage
mkdir storage

#install caddy
git clone [email protected]:mholt/caddy-l4.git build/caddy/caddy-l4
xcaddy build --with github.com/mholt/caddy-l4=${PWD}/build/caddy/caddy-l4 --output ./caddy
chmod +x ./caddy

#install tunnel alternative
git clone [email protected]:cmpxchg16/go-sslterminator.git build/go-sslterminator

#install simple file server
go get -v github.com/jpillora/serve
serve -p 8080 . &

#run caddy
./caddy run --config config.json &

#wait for certs
while [ ! -f $CRT_PATH ]; do sleep 1; done
while [ ! -f $KEY_PATH ]; do sleep 1; done

#run go-sslterminator
go run ./build/go-sslterminator/go-sslterminator.go -b :8080 -c $CRT_PATH -k $KEY_PATH -l :8444

The config file used by caddy is simple:

{
	"storage": {
		"module": "file_system",
		"root": "./storage"
	},
	"logging": {
		"logs": {
			"default": {
				"level": "DEBUG"
			}
		}
	},
	"apps": {
		"tls": {
			"certificates": {
				"automate": ["webserver.127.0.0.1.nip.io"]
			},
			"automation": {
				"policies": [{
						"issuer": {
							"module": "internal"
						},
						"key_type": "rsa4096"
					}
				]
			}
		},
		"layer4": {
			"servers": {
				"layer4-srv0": {
					"listen": ["0.0.0.0:8443"],
					"routes": [{
							"match": [{
									"tls": {
										"sni": ["webserver.127.0.0.1.nip.io"]
									}
								}
							],
							"handle": [{
									"handler": "tls"
								}, {
									"handler": "proxy",
									"upstreams": [{
											"dial": ["localhost:8080"]
										}
									]
								}
							]
						}
					]
				}
			}
		}

	}
}

Now, I would expect curling both 8443 and 8444 to have the same response (TLS terminated, and then serve the http content from :8080). That's been my experience to date in http1.1 land with haproxy.

Here's the output from caddy:

❯ curl -v https://webserver.127.0.0.1.nip.io:8443/now.txt
*   Trying 127.0.0.1:8443...
* TCP_NODELAY set
* Connected to webserver.127.0.0.1.nip.io (127.0.0.1) port 8443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: [NONE]
*  start date: Oct  6 21:24:49 2020 GMT
*  expire date: Oct  7 09:24:49 2020 GMT
*  subjectAltName: host "webserver.127.0.0.1.nip.io" matched cert's "webserver.127.0.0.1.nip.io"
*  issuer: CN=Caddy Local Authority - ECC Intermediate
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fffdbf8baa0)
> GET /now.txt HTTP/2
> Host: webserver.127.0.0.1.nip.io:8443
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* http2 error: Remote peer returned unexpected data while we expected SETTINGS frame.  Perhaps, peer does not support HTTP/2 properly.
* Connection #0 to host webserver.127.0.0.1.nip.io left intact
curl: (16) Error in the HTTP2 framing layer

Here's the output from go-sslterminator:

❯ curl -v https://webserver.127.0.0.1.nip.io:8444/now.txt
*   Trying 127.0.0.1:8444...
* TCP_NODELAY set
* Connected to webserver.127.0.0.1.nip.io (127.0.0.1) port 8444 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: [NONE]
*  start date: Oct  6 21:24:49 2020 GMT
*  expire date: Oct  7 09:24:49 2020 GMT
*  subjectAltName: host "webserver.127.0.0.1.nip.io" matched cert's "webserver.127.0.0.1.nip.io"
*  issuer: CN=Caddy Local Authority - ECC Intermediate
*  SSL certificate verify ok.
> GET /now.txt HTTP/1.1
> Host: webserver.127.0.0.1.nip.io:8444
> User-Agent: curl/7.68.0
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Content-Length: 48
< Content-Type: text/plain; charset=utf-8
< Last-Modified: Tue, 06 Oct 2020 21:34:23 GMT
< Date: Tue, 06 Oct 2020 21:34:28 GMT
<
test script run at Tue Oct  6 17:34:23 EDT 2020
* Connection #0 to host webserver.127.0.0.1.nip.io left intact

Looking at the diff I can see that the difference is probably the "server accepted to use h2" part. I am not very familiar with http2, but I am assuming it's because caddy is using http2 and the file server above is not, like you suggested earlier.

I looked through the docs to see if there is a way to disable http2 globally in caddy, but can't find it. Although I'm not sure if that would apply given that this is an add-on.

I then saw your comment and can confirm it works correctly when disabling http2 in my curl command:

❯ curl -v --http1.1 https://webserver.127.0.0.1.nip.io:8443/now.txt
*   Trying 127.0.0.1:8443...
* TCP_NODELAY set
* Connected to webserver.127.0.0.1.nip.io (127.0.0.1) port 8443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: [NONE]
*  start date: Oct  6 21:49:23 2020 GMT
*  expire date: Oct  7 09:49:23 2020 GMT
*  subjectAltName: host "webserver.127.0.0.1.nip.io" matched cert's "webserver.127.0.0.1.nip.io"
*  issuer: CN=Caddy Local Authority - ECC Intermediate
*  SSL certificate verify ok.
> GET /now.txt HTTP/1.1
> Host: webserver.127.0.0.1.nip.io:8443
> User-Agent: curl/7.68.0
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Content-Length: 48
< Content-Type: text/plain; charset=utf-8
< Last-Modified: Tue, 06 Oct 2020 21:52:42 GMT
< Date: Tue, 06 Oct 2020 21:52:42 GMT
<
test script run at Tue Oct  6 17:49:09 EDT 2020
* Connection #0 to host webserver.127.0.0.1.nip.io left intact

Perhaps this is by design, but it'd be great to have one of the following options (in order of my personal preference):

  1. Detect if backend can't support http2 and disable if so (I'm going to assume this is a big undertaking, and a no-go)
  2. Have a special "http" "https" handler that will terminate ssl before moving on the "proxy" handler
  3. Disable http2 on a per upstream basis
  4. Disable http2 globally

For now I think I can just force clients to use http1.1, or create a local tunnel to an http server in my caddy file as a work around. For example, I think this configuration performs TLS termination in the Http app, which can handle http2, making this a non-issue even though the back end server doesn't.

{
	"storage": {
		"module": "file_system",
		"root": "./storage"
	},
	"logging": {
		"logs": {
			"default": {
				"level": "DEBUG"
			}
		}
	},
	"apps": {
		"tls": {
			"certificates": {
				"automate": ["webserver.127.0.0.1.nip.io"]
			},
			"automation": {
				"policies": [{
						"issuer": {
							"module": "internal"
						},
						"key_type": "rsa4096"
					}
				]
			}
		},
		"http": {
			"servers": {
				"server0": {
					"listen": ["localhost:8445"],
					"routes": [{
							"match": [{
									"host": ["webserver.127.0.0.1.nip.io"]
								}
							],
							"handle": [{
									"handler": "reverse_proxy",
									"transport": {
										"protocol": "http"
									},
									"upstreams": [{
											"dial": "localhost:8080"
										}
									]
								}
							]
						}
					]
				}
			}
		},
		"layer4": {
			"servers": {
				"layer4-srv0": {
					"listen": ["0.0.0.0:8443"],
					"routes": [{
							"match": [{
									"tls": {
										"sni": ["webserver.127.0.0.1.nip.io"]
									}
								}
							],
							"handle": [{
									"handler": "proxy",
									"upstreams": [{
											"dial": ["localhost:8445"]
										}
									]
								}
							]
						}
					]
				}
			}
		}

	}
}

Here's the curl for the above config:

❯ curl -v https://webserver.127.0.0.1.nip.io:8443/now.txt
*   Trying 127.0.0.1:8443...
* TCP_NODELAY set
* Connected to webserver.127.0.0.1.nip.io (127.0.0.1) port 8443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: [NONE]
*  start date: Oct  6 21:49:23 2020 GMT
*  expire date: Oct  7 09:49:23 2020 GMT
*  subjectAltName: host "webserver.127.0.0.1.nip.io" matched cert's "webserver.127.0.0.1.nip.io"
*  issuer: CN=Caddy Local Authority - ECC Intermediate
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fffb7681aa0)
> GET /now.txt HTTP/2
> Host: webserver.127.0.0.1.nip.io:8443
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 200
< accept-ranges: bytes
< content-type: text/plain; charset=utf-8
< date: Tue, 06 Oct 2020 22:13:49 GMT
< last-modified: Tue, 06 Oct 2020 22:13:49 GMT
< server: Caddy
< content-length: 48
<
test script run at Tue Oct  6 17:49:09 EDT 2020
* Connection #0 to host webserver.127.0.0.1.nip.io left intact

Hope this insight helps, or if it's all by design, feel free to close this!

from caddy-l4.

regbo avatar regbo commented on September 28, 2024

Figured out a stopgap solution. I can tunnel the TLS connection to an http app running on another port with the proxy handler. It'd be great if we could skip this step, perhaps with an "https" handler.

from caddy-l4.

mholt avatar mholt commented on September 28, 2024

Hmm good q, lemme look into that tomorrow!

Can you post which curl request yields the error? I'll try to reproduce it.

from caddy-l4.

regbo avatar regbo commented on September 28, 2024

Also, FWIW here's what I get in chrome on https://webserver.127.0.0.1

First config:
test1

Second config:
test2

from caddy-l4.

mholt avatar mholt commented on September 28, 2024

Thanks for the detailed instructions!

I've been having trouble reproducing this locally. (For one, I don't/can't use Docker, so I have my own upstreams that I proxy to.)

I haven't been able to get this error:

curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

What version of curl and openssl do you have?

In fact, your config -- as far as I can tell -- works fine for me. When I curl -v "https://localhost:5443" (again, I set up my own local backend) I got an error about HTTP/2 framing, then I remembered that my backend doesn't use HTTP/2 (it is a plaintext HTTP server, which doesn't support H2C), so I used the --http1.1 flag with curl and then everything worked fine.

What is the output if you use curl -v, and what are the server-side logs?

from caddy-l4.

mholt avatar mholt commented on September 28, 2024

Great writeup, thank you!

Looking at the diff I can see that the difference is probably the "server accepted to use h2" part. I am not very familiar with http2, but I am assuming it's because caddy is using http2 and the file server above is not, like you suggested earlier.

Indeed; Caddy's default TLS configuration sets the following values in the NextProtos field as ALPN:

var defaultALPN = []string{"h2", "http/1.1"}

Server ALPN basically tells the client connecting to a TLS server what (application-layer) protocols the server can speak after TLS is negotiated. As you can see we enable h2 by default because that works with most services. However, older or less capable software won't support HTTP/2.

Since Project Conncept is a layer 4 proxy, it terminates the transport / TLS, but it does not interpret application packets like HTTP. For that, you'd of course use Caddy's HTTP proxy which is already built-in: it can read an HTTP/2 request from the client, decode it, then re-encode it as HTTP/1.1 for the upstream backend that may not support HTTP/2. And this negotiation happens automatically.

But with a layer 4 proxy, it just terminates TLS and then will shuttle raw bytes across the wire, so if it negotiated HTTP/2 with the client, the client will send HTTP/2 packets which get proxied, uninterpreted, to the backend.

I think the solution in your case might be to remove "h2" from the ALPN values. This is done in the l4 module the same way it is in the HTTP server, that is, with a TLS connection policy: https://caddyserver.com/docs/json/apps/http/servers/tls_connection_policies/#alpn -- except, for l4, you'd do it in the connection_policies field:

{
	"handler": "tls",
	"connection_policies": [
		{"alpn": ["http/1.1"]}
	}
}

Give that a shot and see if that works better.

(Note that the structure of each connection policy and the features that are available to you are the exact same as those in the HTTP server, I just can't link to docs for this app because it's not public yet. So basically, this means you can customize the TLS server in other ways too, like filtering policies by ServerName, or changing other things about the TLS connection.)

from caddy-l4.

mholt avatar mholt commented on September 28, 2024

Closing due to inactivity. Assuming resolved, I guess.

from caddy-l4.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.