[macOS] SysCallError: (32, 'EPIPE') when the server requires certs but the client does not present one

[ShaneHarvey]

[macOS] SysCallError: (32, 'EPIPE') when the server requires certs but the client does not present one

[alex]

To ask a maybe obvious question: Do your tests do TLS over a pipe?

…

On Wed, Feb 22, 2023 at 4:17 PM Shane Harvey ***@***.***> wrote: In pymongo we have various tests to ensure that TLS is configured correctly. One such test is when the server is configured to require a client cert but the client does not present one. In this test we expect the client to see an error with "certificate required", "SSL handshake failed", "Connection reset by peer", or one of the equivalent errnos (like ECONNRESET) but we're actually seeing pyopenssl raise (32, 'EPIPE') on macOS (in https://jira.mongodb.org/browse/PYTHON-3607). My understanding is that EPIPE indicates a bug in openssl/pyopenssl, what do you think? Here's more info about the environment: Collecting cryptography>=2 Using cached cryptography-39.0.1-cp36-abi3-macosx_10_12_x86_64.whl (2.9 MB) .... Collecting dnspython<3.0.0,>=1.16.0 Using cached dnspython-2.3.0-py3-none-any.whl (283 kB) Collecting pymongo-auth-aws<2.0.0 Using cached pymongo_auth_aws-1.1.0-py2.py3-none-any.whl (11 kB) Collecting pyopenssl>=17.2.0 Using cached pyOpenSSL-23.0.0-py3-none-any.whl (57 kB) Requirement already satisfied: requests<3.0.0 in ./venv-encryption/lib/python3.9/site-packages (from pymongo==4.4.0.dev1) (2.28.2) Collecting service_identity>=18.1.0 Using cached service_identity-21.1.0-py2.py3-none-any.whl (12 kB) Requirement already satisfied: certifi in ./venv-encryption/lib/python3.9/site-packages (from pymongo==4.4.0.dev1) (2022.12.7) Requirement already satisfied: boto3 in ./venv-encryption/lib/python3.9/site-packages (from pymongo-auth-aws<2.0.0->pymongo==4.4.0.dev1) (1.26.73) Requirement already satisfied: botocore in ./venv-encryption/lib/python3.9/site-packages (from pymongo-auth-aws<2.0.0->pymongo==4.4.0.dev1) (1.29.73) Requirement already satisfied: cryptography<40,>=38.0.0 in ./venv-encryption/lib/python3.9/site-packages (from pyopenssl>=17.2.0->pymongo==4.4.0.dev1) (39.0.1) Requirement already satisfied: idna<4,>=2.5 in ./venv-encryption/lib/python3.9/site-packages (from requests<3.0.0->pymongo==4.4.0.dev1) (3.4) Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./venv-encryption/lib/python3.9/site-packages (from requests<3.0.0->pymongo==4.4.0.dev1) (1.26.14) Requirement already satisfied: charset-normalizer<4,>=2 in ./venv-encryption/lib/python3.9/site-packages (from requests<3.0.0->pymongo==4.4.0.dev1) (3.0.1) Collecting pyasn1-modules Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB) Collecting attrs>=19.1.0 Using cached attrs-22.2.0-py3-none-any.whl (60 kB) Requirement already satisfied: six in ./venv-encryption/lib/python3.9/site-packages (from service_identity>=18.1.0->pymongo==4.4.0.dev1) (1.16.0) Collecting pyasn1 Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB) Requirement already satisfied: cffi>=1.12 in ./venv-encryption/lib/python3.9/site-packages (from cryptography<40,>=38.0.0->pyopenssl>=17.2.0->pymongo==4.4.0.dev1) (1.15.1) Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in ./venv-encryption/lib/python3.9/site-packages (from boto3->pymongo-auth-aws<2.0.0->pymongo==4.4.0.dev1) (1.0.1) Requirement already satisfied: s3transfer<0.7.0,>=0.6.0 in ./venv-encryption/lib/python3.9/site-packages (from boto3->pymongo-auth-aws<2.0.0->pymongo==4.4.0.dev1) (0.6.0) Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in ./venv-encryption/lib/python3.9/site-packages (from botocore->pymongo-auth-aws<2.0.0->pymongo==4.4.0.dev1) (2.8.2) Requirement already satisfied: pycparser in ./venv-encryption/lib/python3.9/site-packages (from cffi>=1.12->cryptography<40,>=38.0.0->pyopenssl>=17.2.0->pymongo==4.4.0.dev1) (2.21) Building wheels for collected packages: pymongo Building wheel for pymongo (setup.py): started Building wheel for pymongo (setup.py): finished with status 'done' Created wheel for pymongo: filename=pymongo-4.4.0.dev1-cp39-cp39-macosx_11_0_x86_64.whl size=385239 sha256=6907c24d0b03797d5cbe1245fb6ac1bc786e92b4e0a40c82c029112bc9ef858f Stored in directory: /System/Volumes/Data/data/mci/4527dad69325df5546afe7b616b18d7d/drivers-tools/.evergreen/orchestration/db/pip-ephem-wheel-cache-ldh4fy0r/wheels/39/f3/d9/27a37bbf2df83660f5cbf8bae4a0c81495ac5a49e4975b8ea9 Successfully built pymongo Installing collected packages: pyasn1, pyasn1-modules, dnspython, attrs, pymongo, service_identity, pyopenssl, pymongo-auth-aws Successfully installed attrs-22.2.0 dnspython-2.3.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pymongo-4.4.0.dev1 pymongo-auth-aws-1.1.0 pyopenssl-23.0.0 service_identity-21.1.0 Python version: 3.9.10 (main, Jan 15 2022, 11:48:00) [Clang 13.0.0 (clang-1300.0.29.3)] macOS 11 — Reply to this email directly, view it on GitHub <#1189>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAGBHFRSJYOFGNKX2QVTLWYZ65RANCNFSM6AAAAAAVE2XUK4> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

-- All that is necessary for evil to succeed is for good people to do nothing.

[ShaneHarvey]

No pipes, just standard tcp sockets. I actually just remembered that we're using pykmip in these tests and these errors might be caused by it's funky use of shutdown() which I'm trying to fix in: OpenKMIP/PyKMIP#682

It would be possible to confirm that theory by testing the "server requires certs but the client does not present one" case using only pyopenssl.

[ShaneHarvey]

Update, I just tested two scenarios:

1. A MongoDB server (running locally on macOS) configured to require client certs.
2. A PyKMIP server configured to require client certs with and without the fix for OpenKMIP/PyKMIP#682.

In 1) I correctly see this error:

>>> Traceback (most recent call last):
  File "/Users/shane/git/mongo-python-driver/pymongo/pool.py", line 1061, in _configured_socket
    sock = ssl_context.wrap_socket(sock, server_hostname=host)
  File "/Users/shane/git/mongo-python-driver/pymongo/pyopenssl_context.py", line 369, in wrap_socket
    ssl_conn.do_handshake()
  File "/Users/shane/git/mongo-python-driver/pymongo/pyopenssl_context.py", line 125, in do_handshake
    return self._call(super(_sslConn, self).do_handshake, *args, **kwargs)
  File "/Users/shane/git/mongo-python-driver/pymongo/pyopenssl_context.py", line 108, in _call
    return call(*args, **kwargs)
  File "/Users/shane/work/pycharm/pymongo-py310/lib/python3.10/site-packages/OpenSSL/SSL.py", line 2075, in do_handshake
    self._raise_ssl_error(self._ssl, result)
  File "/Users/shane/work/pycharm/pymongo-py310/lib/python3.10/site-packages/OpenSSL/SSL.py", line 1715, in _raise_ssl_error
    _openssl_assert(
  File "/Users/shane/work/pycharm/pymongo-py310/lib/python3.10/site-packages/OpenSSL/_util.py", line 71, in openssl_assert
    exception_from_error_queue(error)
  File "/Users/shane/work/pycharm/pymongo-py310/lib/python3.10/site-packages/OpenSSL/_util.py", line 57, in exception_from_error_queue
    raise exception_type(errors)
OpenSSL.SSL.Error: [('STORE routines', '', 'unregistered scheme'), ('system library', '', ''), ('STORE routines', '', 'unregistered scheme'), ('system library', '', ''), ('SSL routines', '', 'certificate verify failed')]

In 2) I see EPIPE:

  File "/Users/shane/git/mongo-python-driver/pymongo/encryption.py", line 726, in create_data_key
    return self._encryption.create_data_key(
  File "/Users/shane/work/pycharm/pymongo-py310/lib/python3.10/site-packages/pymongocrypt/explicit_encrypter.py", line 174, in create_data_key
    key = run_state_machine(ctx, self.callback)
  File "/Users/shane/work/pycharm/pymongo-py310/lib/python3.10/site-packages/pymongocrypt/state_machine.py", line 150, in run_state_machine
    callback.kms_request(kms_ctx)
  File "/Users/shane/git/mongo-python-driver/pymongo/encryption.py", line 143, in kms_request
    conn.sendall(message)
  File "/Users/shane/git/mongo-python-driver/pymongo/pyopenssl_context.py", line 151, in sendall
    sent = self._call(super(_sslConn, self).send, view[total_sent:], flags)
  File "/Users/shane/git/mongo-python-driver/pymongo/pyopenssl_context.py", line 108, in _call
    return call(*args, **kwargs)
  File "/Users/shane/work/pycharm/pymongo-py310/lib/python3.10/site-packages/OpenSSL/SSL.py", line 1899, in send
    self._raise_ssl_error(self._ssl, result)
  File "/Users/shane/work/pycharm/pymongo-py310/lib/python3.10/site-packages/OpenSSL/SSL.py", line 1699, in _raise_ssl_error
    raise SysCallError(errno, errorcode.get(errno))
OpenSSL.SSL.SysCallError: (32, 'EPIPE')

Interestingly, from this trace I can see that the client thinks the TLS handshake completed successfully (do_handshake() completes without error) but then the connection raises EPIPE on the first send() of application data.

The kmip server is here https://github.com/mongodb-labs/drivers-evergreen-tools/blob/62f34e8/.evergreen/csfle/kms_http_server.py#L231:

$ bash
$ git clone [email protected]:mongodb-labs/drivers-evergreen-tools.git
$ cd drivers-evergreen-tools/.evergreen/csfle
$ . activate_venv.sh
$ python -u kms_http_server.py --ca_file ../x509gen/ca.pem --cert_file ../x509gen/server.pem --port 8002 --require_client_cert
Mock KMS Web Server Listening on port 8002

[ShaneHarvey]

There's some work I could do here to try and create a minimal repro but I won't have time for a while. In the meantime, any theories would be appreciated!