remove file check to support proxied SSL connection

[HoneyryderChuck]

remove file check to support proxied SSL connection

[rhenium]

Currently, SSLSocket passes the file descriptor to OpenSSL and doesn't use Ruby's IO interface.

openssl/ext/openssl/ossl_ssl.c

Lines 1688 to 1689 in 45c731c

	if (!SSL_set_fd(ssl, TO_SOCKET(rb_io_descriptor(io))))
	ossl_raise(eSSLError, "SSL_set_fd");

OpenSSL uses an abstraction layer called BIO to interact with the underlying socket. Here is the default implementation (a "BIO method" - SSL_set_fd() will set up the BIO_s_socket()): https://github.com/openssl/openssl/blob/master/crypto/bio/bss_sock.c

It's more work than just removing a check, but it should be possible to define a custom BIO method that wraps an IO-ish object (as long as it supports non-blocking IO methods).

[HoneyryderChuck]

I was anticipating it to be much harder than just removing the check, no worries.

To better illustrate what I'm looking for, here's a practical example using docker: I've just pushed the "infra setup" changes to this MR branch. Once you download the repo and switch to the "https-proxy" branch:

• run docker-compose -f docker-compose -f docker-compose-ruby-{MAJOR}-{MINOR}.yml run --entrypoint bash httpx to open a shell
• run these and these commands

And you should be in a position to replicate the issue. You can run:

> curl -x https://proxyuser:password@httpsproxy:9080 \
          --proxy-cacert test/support/ci/certs/ca.crt \
          https://nghttp2/get -v

to verify what/how curl does it. To replicate what at least I'd expect to work in ruby, you can run this script:

require "socket"
require "openssl"
require "base64"


AUTH = Base64.strict_encode64("proxyuser:password")

CONNECT_REQ = <<-CONNECT.gsub("\n", "\r\n")
CONNECT nghttp2:443 HTTP/1.1
Host: nghttp2
Proxy-Authorization: Basic #{AUTH}

CONNECT

REQ = <<-REQ.gsub("\n", "\r\n")
GET /get HTTP/1.1
User-Agent: smth
Accept: */*
Connection: close
Host: nghttp2

REQ

tcp = TCPSocket.new("httpsproxy", 9080)
ctx = OpenSSL::SSL::SSLContext.new
ssl = OpenSSL::SSL::SSLSocket.new(tcp, ctx)
ssl.hostname = "httpsproxy"
ssl.sync_close = true
ssl.connect
ssl.post_connection_check("httpsproxy")

ssl.write(CONNECT_REQ)

response = ssl.readpartial(2048)

puts "response from proxy connect"
puts response.inspect

raise unless response.start_with?("HTTP/1.1 200")

ssl.write(REQ)

puts ssl.readpartial(2048)

which currently breaks on the last line with ...openssl/buffering.rb:157:in sysread': end of file reached (EOFError)`

[HoneyryderChuck]

@rhenium just opened an MR to explore a possible approach to solving this. LMK what you think, and whether you think there are other alternatives worth pursuing.

[rhenium]

I was exploring the approach to implement a BIO_METHOD that simply translates those Ruby IO methods. It's not reviewed carefully, but here is a WIP PR: #736

Thanks for the pointer to usage of the memory BIO in Python's ssl library. I didn't know about it, and that approach is interesting. While it is slightly inefficient due to one additional data copy, if I understand it right, it would let us interact with the IO-like object outside of SSL_{read,write,..}() and reduce the amount of rb_protect() state variables churn. It's quite a mess in my WIP PR.

[HoneyryderChuck]

hey @rhenium , after a bit of back-and-forth with my approach, and hitting some limitations of my openssl C API expertise, and also researching other implementations, I'm leaning towards your approach of BIO_METHOD as being the most viable.

To begin with, it works! I've tested it with the script pasted above against a server behind-a-squid-behind-stunnel, and even if still a rough patch, it worked the first time. So that's something I haven't been able to produce.

I've also been looking at how curl works on top of openssl, and it seems that it also ships with its own BIO_METHOD. The main advantage seems to be that SSL_connect usage seems to "just work", whereas you'd have to "drop down" a layer of abstraction if maintaining bio-mems around (ssl connect was the hurdle I was never to pass from in my attempt). This sees to be how cpython works with openssl, where different APIs are used (SSL_write_ex instead of SSL_write, and SSL_do_handshake instead of SSL_connect, just to name a couple).

I also find that approach to be a better template to implement on top of openssl QUIC connections; not sure how much you've considered introducing them at this point though.

About rb_protect usage, I'm not sure if you really need it, considering you're using the "exception: false" variants of the non-blocking APIs? Moreover (and a bit tangential), having researched the cpython implementation (a bit superficially, I admit), I was wondering why can't the gem release the GVL more often, for example when calling the SSL_* family of APIs (python examples here and here), as I believe roughly the same constraints apply?

Thx for the help sofar, and LMK how I can help move this forward! 🙏