FAIL: test_set_verify_callback_exception on FreeBSD (pyopenssl 0.14)

[pyca]

FAIL: test_set_verify_callback_exception on FreeBSD (pyopenssl 0.14)

[kouk]

What about OpenSSL.test.test_ssl.ContextTests.test_set_default_verify_paths? On my FreeBSD system that throws an error because my OpenSSL installation does not have any default trusted certificates. This might not be a problem for you of course, but you can check with $ openssl s_client -connect verisign.com:443. If it says "cannot get local issuer certificate" then probably that's the problem.

[kouk]

Having said that, I should add that the test also fails for me on FreeBSD regardless of whether verification is properly set up.

[koobs]

@kouk This was the only test failure I saw (other than unrelated #37)

Verify return code: 20 (unable to get local issuer certificate)

FWIW, security/ca_root_nss is installed with ETCSYMLINK option

Can the test be made robust, to either skip or warn based on this? What is actually being tested here?

[kouk]

For me this specific problem was solved with f012467. The sleep value could be even lower for me but I left it at 50ms. By the way I get another 8 failures (9 if you count #37) on this old FreeBSD laptop of mine, but I haven't had time to check them out.

[kouk]

@koobs do you have security/openssl installed or did you build against /usr/lib/libssl.so ? What version? 7/8 fails I was getting were due to /usr/bin being before /usr/local/bin in PATH and a different openssl version was being used by the test run. But I'm getting a fail from OpenSSL.test.test_crypto.X509Tests.test_digest because the calculated digests are different.

[koobs]

Hmm, so I found an issue (minor) which led me to something else.

Prior to 0.14, the py-openssl port depended on OpenSSL. Now that pyopenssl uses cryptography for its OpenSSL bits, I neglected to move the openssl depends from py-openssl to py-cryptography.

So I checked links for the .so's created there and found:

./build/lib.freebsd-9.2-STABLE-amd64-2.7/cryptography/_cffi__x9ee276ddxb4fc9075.so:
        libcrypto.so.6 => /lib/libcrypto.so.6 (0x80125f000)
        libssl.so.6 => /usr/lib/libssl.so.6 (0x801607000)
        libc.so.7 => /lib/libc.so.7 (0x80081d000)

I do have security/openssl installed, and the dependency was 'supposed' to have linked against that in /usr/local.

Here's what the py-openssl port does (that is no longer relevant since 0.14) to set the correct include/library paths:

pre-configure:
        @${ECHO_CMD} "[build_ext]" >> ${WRKSRC}/setup.cfg
        @${ECHO_CMD} "include-dirs = ${OPENSSLINC}" >> ${WRKSRC}/setup.cfg
        @${ECHO_CMD} "library-dirs = ${OPENSSLLIB}" >> ${WRKSRC}/setup.cfg

I imagine I now need to do the same for cryptography, but it does lend weight to your theory. Why and how do the tests use different bits when pyopenssl uses cryptography for all its functions?

[kouk]

The tests that were failing were just running the openssl command to get "good" output to compare against the test output. There were some formatting differences in the output of /usr/bin/openssl so the tests were failing.

As for cryptography, I built it by hand in a virtualenv but first did:

export CFLAGS="-I/usr/local/include -L/usr/local/lib -DCRYPTO_MDEBUG" 

This is how I managed to get cffi which is used by cryptography to pass the correct flags to the compiler. I don't know of any better way atm.

[koobs]

Adding:

CFLAGS+=        -I${OPENSSLINC}
LDFLAGS+=       -L${OPENSSLLIB}

To cryptography's build, results in:

ldd ./work/cryptography-0.2.1/build/lib.freebsd-9.2-STABLE-amd64-2.7/cryptography

/_cffi__x9ee276ddxb4fc9075.so
./work/cryptography-0.2.1/build/lib.freebsd-9.2-STABLE-amd64-2.7/cryptography/_cffi__x9ee276ddxb4fc9075.so:
        libcrypto.so.8 => /usr/local/lib/libcrypto.so.8 (0x80125f000)
        libssl.so.8 => /usr/local/lib/libssl.so.8 (0x80165a000)
        libc.so.7 => /lib/libc.so.7 (0x80081d000)
        libthr.so.3 => /lib/libthr.so.3 (0x8018c0000)

Tests:

 62912 passed, 12 skipped in 145.50 seconds

[koobs]

With py-cryptography compiled & linked against OpenSSL from ports, the following tests fail in pyopenssl:

Ran 352 tests in 1.774s -FAILED (failures=10, errors=1)

FAIL: test_dump_certificate_request (OpenSSL.test.test_crypto.FunctionTests)
FAIL: test_export_without_args (OpenSSL.test.test_crypto.PKCS12Tests)
FAIL: test_export_without_mac (OpenSSL.test.test_crypto.PKCS12Tests)
FAIL: test_friendly_name (OpenSSL.test.test_crypto.PKCS12Tests)
FAIL: test_various_empty_passphrases (OpenSSL.test.test_crypto.PKCS12Tests)
FAIL: test_digest (OpenSSL.test.test_crypto.X509Tests)
FAIL: test_wantWriteError (OpenSSL.test.test_ssl.ConnectionTests)
FAIL: test_set_verify_callback_exception (OpenSSL.test.test_ssl.ContextTests)

/usr/bin/openssl version

OpenSSL 0.9.8y 5 Feb 2013

/usr/local/bin/openssl version

OpenSSL 1.0.1f 6 Jan 2014

[kouk]

ok, we have the same versions. Also I believe if you make sure that /usr/local/bin is before /usr/bin in $PATH all tests will pass except test_digest. I opened #41 for test_digest in particular.

[koobs]

Not sure if you expected this:

PATH="/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin" /usr/bin/make regression-test

ERROR: test_set_default_verify_paths (OpenSSL.test.test_ssl.ContextTests)
Error: [('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')]

FAIL: test_digest (OpenSSL.test.test_crypto.X509Tests)
AssertionError: 'C1:B5:90:A4:41:11:C8:30:BF:D4:AA:78:13:46:66:59' != 'A8:EB:07:F8:53:25:0A:F2:56:05:C5:A5:C4:C4:C7:15'

FAIL: test_wantWriteError (OpenSSL.test.test_ssl.ConnectionTests)
AssertionError: <class 'OpenSSL.SSL.WantReadError'> raised instead of WantWriteError

FAIL: test_set_verify_callback_exception (OpenSSL.test.test_ssl.ContextTests)
AssertionError: Exception not raised (None returned)

Ran 352 tests in 1.835s - FAILED (failures=3, errors=1)

Note: regression-test is just the port make target that runs python setup.py test

How might the tests be made to run without requiring the PATH be modified?

[koobs]

FWIW, if you're using FreeBSD, I've committed the latest update to the security/py-cryptography port, which has already propagated to the portsnap mirrors if you want it:

http://svnweb.freebsd.org/ports?view=revision&revision=345962

CFLAGS & LDFLAGS environment variables were used.

[kouk]

Hmm for test_set_default_verify_paths try export SSL_CERT_FILE=/usr/local/share/certs/ca-root-nss.crt.
Apart from test_digest the other problems are fixed I think in #39 and #43.
Also, thanks for committing the port. Although my work with pyopenssl isn't on FreeBSD (Windows unfortunately) it's nice to be able to get it for cheap on my workstation.

[kouk]

about the modified PATH requirement perhaps there is another to get "good" test output apart from running the openssl binary, but off the top of my head I can't think of one.

[koobs]

If the definition of good is that the underlying system openssl output matches that which comes out of the library you leverage for crypto, then it sounds like a test that only cryptography can do, since it has private knowledge of the environment its been built within.

Perhaps the rest of the @pyca crew could provide some input on making these tests more robust?

[0-wiz-0]

The test fails for me on NetBSD using pyOpenSSL-0.14 and cryptography-0.2.2.

FAIL: test_set_verify_callback_exception (OpenSSL.test.test_ssl.ContextTests)

Traceback (most recent call last):
File "/scratch/security/py-OpenSSL/work/pyOpenSSL-0.14/OpenSSL/test/test_ssl.py", line 1002, in test_set_verify_callback_exception
Exception, self._handshake_test, serverContext, clientContext)
File "/scratch/security/py-OpenSSL/work/pyOpenSSL-0.14/OpenSSL/test/util.py", line 270, in failUnlessRaises
% (exception.__name__, result))
AssertionError: Exception not raised (None returned)

This is in a sandbox with no certificates installed. If the test needs any, IMHO it should install them.

[alex]

This code has also certainly been rewritten in the last 3 years, so I'm going to close. Please comment if this is still an issue.