tlsfuzzer/tlsfuzzer

SSL and TLS protocol test suite and fuzzer

tlsfuzzer

tlsfuzzer is a test suite for SSLv2, SSLv3, TLS 1.0, TLS 1.1, TLS 1.2, and TLS 1.3 implementations. It's in early stages of development, so there are no API stability guarantees. While it uses fuzzing techniques for testing (randomisation of passed in inputs), the scripts are generally written in a way that verifies correct error handling: unlike typical fuzzers it doesn't check only that the system under test didn't crash, it checks that it returned correct error messages.

You can find ready-to-use scripts testing for many vulnerabilities ( ROBOT, DROWN, etc.) and general standards conformity (RFC 5246, RFC 7627, RFC 7905, etc.) in the scripts/ directory.

Dependencies

You'll need:

• Python 2.6 or later or Python 3.5 or later
• tlslite-ng 0.8.0-beta1 or later (note that tlslite will not work and they conflict with each other)
• ecdsa python module (dependency of tlslite-ng, should get installed automatically with it), use at least version 0.15 for optimal performance

Optionally, to make cryptographic calculations significantly faster, you may want to install the following libraries (see tlslite-ng and python-ecdsa README files for details):

• m2crypto
• gmpy

To get pip (if your python installation doesn't already have it) download get-pip.py and run (or see USAGE.md for alternative configuration that does not require installation of packages):

python get-pip.py

Then install tlslite-ng:

pip install --pre tlslite-ng

(Use --upgrade --pre if you did install it before)

Download the tlsfuzzer:

git clone https://github.com/tlsfuzzer/tlsfuzzer.git

Usage

After all dependencies are installed, make sure:

• you're in the directory of the project (after git clone just cd tlsfuzzer)
• the server you want to test is running on the same computer (localhost)
• the server is listening on port 4433
• and the server will answer with data to HTTP queries (answer with valid HTTP responses is optional)

Then you can run one of the tests in scripts directory, like so:

PYTHONPATH=. python scripts/test-invalid-compression-methods.py

If test has additional requirements, it will output them to console. No errors printed means that all expecations were met (so for tests with bad data the server rejected our messages).

All scripts also accept --help to print the help message (specification of all the options given script supports), -h to specify the hostname or IP address of the server-to-be-tested and -p to specify the port of the service to be tested.

See USAGE.md for more info and how to interpret errors and failures reported by scripts.

You can find mode detailed documentation for the project at tlsfuzzer.readthedocs.io.

Using tlsfuzzer to test for timing side-channel attacks (Lucky13, padding oracle attacks and timing-based Bleichenbacher oracle) is described in the TIMING.md document.

Server under test configuration

In general, the server under test requires just a RSA certificate, you can create it using the following OpenSSL command:

openssl req -x509 -newkey rsa -keyout localhost.key -out localhost.crt -subj \
/CN=localhost -nodes -batch

Note: tlsfuzzer verifies only TLS level behaviour, it does not perform any checks on the certificate (like hostname validation, CA signatures or key usage). It does however verify if the signatures made on TLS message by the server (like in Server Key Exchange or Certificiate Verify message) match the certificate sent by the server.

More detailed instructions, including how to build the different frameworks from source, are available in the Server setup wiki page.

Example server configurations:

OpenSSL

To test OpenSSL, it's sufficient to pass an extra -www option to a typical s_server command line:

openssl s_server -key localhost.key -cert localhost.crt -www

GnuTLS

To test GnuTLS server, you need to tell it to behave as an HTTP server and additionally, to not ask for client certificates:

gnutls-serv --http -p 4433 --x509keyfile localhost.key --x509certfile \
localhost.crt --disable-client-cert

NSS

To test the Mozilla NSS library server, you first need to create a database with server certificate:

mkdir nssdb
certutil -N -d sql:nssdb --empty-password
openssl pkcs12 -export -passout pass: -out localhost.p12 -inkey localhost.key \
-in localhost.crt -name localhost
pk12util -i localhost.p12 -d sql:nssdb -W ''

Finally, start the server with support for TLSv1.0 and later protocols, DHE ciphers and with the above certificate:

selfserv -d sql:./nssdb -p 4433 -V tls1.0: -H 1 -n localhost

Advanced configuration

More advanced and complex configurations as well as description how to compile the above servers from source is available on the wiki page Server setup.

Contributing

See the CONTRIBUTING.md document for description how to set up your development environment, sanity check the changes and requirements the changes need to follow.

You may also want to read the VISION.md to learn more about the planned scope of the project.

Contributors are expected to follow the project's CODE OF CONDUCT when interacting with other members of the community.

tlsfuzzer's Issues

More usable error messages - repr on nodes

Decision tree nodes should support calling repr() on them.
Constructors of the nodes should support assigning human-readable names to them (name= in construction)

Support for DSA certificates/key exchange

add support for DHE-DSA key exchange

depends on tlsfuzzer/tlslite-ng#99

Add test script for alternative curves for ECDHE_RSA kex

Server should automatically select curve advertised by client or downgrade to non ECC ciphersuite.

Check if server supports, for example secp256r1 and secp384r1, and will select only the one advertised by client. At the same time, if only a weak curve is advertised (e.g. secp160k1), that it will
downgrade to a DHE_RSA or straight RSA ciphersuite.

Run as many scripts as possible against tlslite-ng test server

To verify that the scripts are still running, add a python script that starts the test server and runs the scripts against it one by one (add to Continuous Integration)

Don't expect server to support renegotiation_info if not required by testcase

The scripts often require the server to reply with a renegotiation_info extension in the server hello - even though something completely different shall be tested (e.g. scripts/test-invalid-content-type.py).
On a server that does not support re-negotiation at all, this extension won't be present - making all testcases fail.

Script to check renegotiation with client certificates

write a script to test behaviour of servers with regards to client certificates in renegotiated handshakes

Duplicate TLS extension testing

RFC5246 mentions that There MUST NOT be more than one extension of the same type.. There is currently no test of tlsfuzzer checking the behavior of the server in that scenario.

I checked test-extensions.py and to my understanding it is not possible to send duplicate extensions at this point.

Test with odd length of cipher suites in Client Hello

Need to write a test that checks if server correctly handles malformed Client Hello messages, ones that send the cipher_suites list with one value that is truncated.

"cannot import name CertificateStatus" when running "make test"

I am having an issue with the error "Cannot import name CertificateStatus" when I run "make test" after "make install." I tried uninstalling and reinstalling tlslite-ng as you said, but it did not fix the error. Also, when I run sudo commands with pip I get a message that says The directory ../home/kincade/.cache/pip/html or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want to use sudo's -H flag." I'm not quite sure what this means or what I should do. Any suggestions? Thank you.

handling of "empty" supported_groups extension

For servers that do support FFDHE extension to supported_groups (RFC 7919), the case of Client Hello including ECDHE ciphers, but no ECDHE curves is a bit tricky, see which parts of RFC's apply, and what is the expected behaviour.

(Test in TLS 1.2)

Collect timing statistics of server responses

Many attacks on TLS (e.g. Lucky 13) depend on timing of responses. The tool should be able to collect and correlate the timing between responses to valid and invalid messages.

Most likely, the easiest and most precise way would be by using tcpdump packet capture.

Depends on #47

Detect supported features in server, only run applicable tests

First scan the server what features (extensions, ciphers) it supports, run only applicable tests against it.

Test in CI if a script added to `scripts/` is also added to CI

tests/scripts_retention.py makes sure that existing tests continue to work when run against tlslite-ng, make sure that any new tests added to scripts/ are also added to the scripts_retention.py so that they too can benefit from that check

that will require extending Travis CI script to do that

Fuzz mac in AEAD ciphers (GCM/CHACHA20-POLY1305)

fuzz_mac() is not working with AEAD ciphers, what happens is that the server sends application data and everything works fine. To test it, swap "ciphers = [CipherSuite.TLS_DHE_RSA_WITH_AES_128_GCM_SHA256]" in test-fuzzed-mac.py script.
Any hints how to make this work?

STARTTLS support

many servers require special commands to be exchanged first, before TLS Client Hello can be sent

Add support for negotiating TLS with:

• SMTP
• IMAP and POP3
• LDAP
• XMPP
• NNTP
• IRC
• FTP
• PostgreSQL
• HTTP(?)
• MySQL
• etc.

Support statistical analysis of certain fields

Server's random value should be, as name suggests, random (tlsdate not withstanding). Other values like AES-GCM nonces should be counters.

Allow the runner to collect such values, allow for subsequent analysis of them.

• ServerHello random
• Server key share
• nonce in AEAD
  • see also test-aes-gcm-nonces.py
• salt in RSA-PSS signatures
• session ID in TLS 1.2 and earlier
• TLS 1.3 HRR cookie

some statistical methods to check for predictability of random numbers: http://lcamtuf.coredump.cx/oldtcp/tcpseq.html

Tests for supported curves

check if server will accept a good set of curves (P-256, P-384, P-521, X25519, X448), but reject all others, including undefined (watch out for FFDHE range).

The set of acceptable curves should be configurable (see #123) as a script parameter.

Testing fragmentation on TCP level

Allow the Runner to fragment the sent records on TCP level - that is, send them in small, just few bytes long, TCP packets

more readable assert failures

e.g.

Traceback (most recent call last):
  File "scripts/test-x25519.py", line 359, in main
    runner.run()
  File "/home/travis/build/tomato42/tlsfuzzer/tlsfuzzer/runner.py", line 168, in run
    node.process(self.state, msg)
  File "/home/travis/build/tomato42/tlsfuzzer/tlsfuzzer/expect.py", line 156, in process
    assert self.cipher == srv_hello.cipher_suite
AssertionError

should not use the AssertionError, should print the expected value(s) and the actually received one

Add CI on Windows

Appveyor allows for doing CI on Windows machines, add it.

depends on tlsfuzzer/tlslite-ng#70

ImportError: cannot import name SSL2HandshakeType

Hello

I got this issue when running test-sslv2-force-export-cipher.py.

ImportError: cannot import name SSL2HandshakeType

I have the recommended tlslite-ng version

$ pip2 list | grep tlslite-ng
tlslite-ng (0.6.0a4)

I run the python script with:

$ PYTHONPATH=. python2 scripts/test-sslv2-force-export-cipher.py -h example.com -p 443

Traceback (most recent call last):
  File "scripts/test-sslv2-force-export-cipher.py", line 9, in <module>
    from tlsfuzzer.runner import Runner
  File "/home/Code/drown_attack/tlsfuzzer/tlsfuzzer/runner.py", line 11, in <module>
    from tlslite.constants import ContentType, HandshakeType, AlertLevel, \
ImportError: cannot import name SSL2HandshakeType

script to test session resume with a different SNI host

resuming session to a different virtual host should be rejected

Feature parity with OpenSSL 1.0.1 on crypto and TLS extensions

This is a tracker bug for the general goal to achieve feature parity with regards to supported crypto algorithms with OpenSSL 1.0.1.

In particular:

• ECDSA certificates #197
• RSA certificates
• DSA certificate #54
• ECDHE-ECDSA kex #197
• ECDH-ECDSA kex #272
• ECDHE-RSA kex
• ECDH-RSA kex #272
• AECDH kex
• DHE-DSS kex #54
• DHE-RSA kex
• ADH kex
• RSA kex
• PSK kex tlsfuzzer/tlslite-ng#22
• KRB5 kex
• EXP kex (low prio, just checking if they are not supported should be sufficient)
• AES-CBC cipher
• AES-GCM cipher
• 3DES cipher
• RC4 cipher
• Camellia-CBC cipher tlsfuzzer/tlslite-ng#55
• DES-CBC cipher (low prio)
• IDEA-CBC cipher
• SEED-CBC cipher
• renegotiation
• client certificates
• server_name (SNI) extension #88
• SessionID resumption
• Session ticket resumption #52
• Certificate status (OCSP stapling) #53
• Next Proto Negotiation #51
• Heartbeat tlsfuzzer/tlslite-ng#58 #271
• SSLv2
• SSLv3
• TLS1.0
• TLS1.1
• TLS1.2
• DTLS (though probably with the goal of post 1.0.0 release)

Add CI on Mac OSX

Travis supports OSX builds now, do a test on them too.

depends on tlsfuzzer/tlslite-ng#73

AttributeError: 'Conversation' object has no attribute 'is_command'

Run the scripts/test-tls-server.py test, raise this error:
Also sys module that uses in this file, not imported.

(env)$ PYTHONPATH=. python scripts/test-tls-server.py 
Error encountered while processing node <tlsfuzzer.generators.Conversation object at 0x7f54e1451ed0> with last message being: None
Traceback (most recent call last):
  File "scripts/test-tls-server.py", line 27, in main
    res = runner.run()
  File "..../github/tlsfuzzer/tlsfuzzer/runner.py", line 79, in run
    if node.is_command():
AttributeError: 'Conversation' object has no attribute 'is_command'

Test end
successful: 0
failed: 2

Support for TLS1.3 - basic 1-RTT (no HRR) handshake

Support for basics of TLS1.3 protocol.

Depends on tlsfuzzer/tlslite-ng#174

Implement enough of TLS 1.3 support in tlsfuzzer to be able to perform a standard 1-RTT handshake, without use of HelloRetryRequest.

Allow configurable port

Currently all tests utilize port 4433. When integrating that into a CI, it requires a system with port 4433 free, which may not be under the control of the runner.

However, the important advantage of allowing a configurable port, is that tests requiring a different server (e.g., one verifying client certificates, one accepting them without verification, one which doesn't ask for a certificate), can be run independently and in parallel.

script to test CVE-2016-9244 (Ticketbleed)

Ticketbleed (CVE-2016-9244) detector
Sending a (N - 31 byte) sessionId and alerting on received ticket being

• serverHello.sessionId larger than clientHello.sessionId
• equals( serverHello.sessionId, clientHello.sessionId, length(clientHello.sessionId) )

Probability of a false alert should be very low with a good value of N and a decent random pattern.

Ref: https://blog.filippo.io/finding-ticketbleed/

test-chacha20.py: max size plaintext: incorrect assumptions on server behavior

The test 'max size plaintext' of test-chacha20.py assumes that the server will reply with a single application data message to its request. Snippet:

    node = node.add_child(ApplicationDataGenerator(data))
    node = node.add_child(ExpectApplicationData())
    node = node.add_child(AlertGenerator(AlertLevel.warning,
                                         AlertDescription.close_notify))
    node = node.add_child(ExpectAlert(AlertLevel.warning,
                                      AlertDescription.close_notify))

That expectation is arbitrary, and doesn't work in general. gnutls-serv for example will reply with a message forwarding the original request, and thus its reply will span across two Application data messages.

A better behavior would be to check the incoming queue for a close notify, and if it exists it should ignore anything else.

test-extended-master-secret-extension.py: doesn't accept -e

It seems that both test-extended-master-secret-extension.py and test-record-layer-fragmentation.py run all tests even if '-e' is specified. Thus at the moment to avoid known failures it has to be disabled.

Script to verify that server doesn't mask ciphersuite bits

Some old servers (SSLv2 era) don't check the first byte of ciphersuite ID advertised by client in SSLv3 and later Client Hello messages.

Write a test script that goes over all possible ciphersuite IDs and verifies that server either negotiates the single one advertised by client or aborts connection with an Alert message.

ability to name nodes in the tree

when we have multiple generator or expects of the same time in a decision tree (in case of ExpectAlert or renegotiation test cases), it's hard to tell them apart - add ability to name the nodes and make the error messages returned by them include the name in error report.

script to test off-by-one TLS version negotiation

Check that if version 1.X is the highest version supported by server, sending 1.X+1 or 1.X+100 will always negotiate version 1.X, but if version 1.X-1 is advertised by client, server selects 1.X-1 (as long as it is strictly higher than SSLv3 in default configuration)

just part of backwards compatibility testing for TLS 1.3, but important for TLS 1.2

Support for RFC5077 session tickets

add support for session resumption using session tickets (RFC 5077)

most likely will require extending tlslite-ng first (tlsfuzzer/tlslite-ng#60)

Add mutation testing to CI

Since the test coverage we're using is only a line coverage (not even statement coverage let alone condition coverage), a 100% coverage isn't indicative of all code paths being tested by unit tests.

Add mutation testing to the unit testing workflow so that we know how good the unit tests are.

https://github.com/sixty-north/cosmic-ray looks like a good candidate

Support sending duplicated extensions in Client Hello

the RFC does not allow the client to send a Client Hello with multiple extensions of the same type. Check if the server rejects such messages.

ImportError from tlslite(-ng)

Hi,

I've tried running a few tests, resulting in the following error message:

daniel: ~/code/tlsfuzzer > PYTHONPATH=. python2.7 scripts/test-record-layer-fragmentation.py
Traceback (most recent call last):
  File "scripts/test-record-layer-fragmentation.py", line 8, in <module>
    from tlsfuzzer.runner import Runner
  File "/home/daniel/code/tlsfuzzer/tlsfuzzer/runner.py", line 7, in <module>
    from tlslite.messages import Message, Certificate
ImportError: No module named tlslite.messages

After trying sudo pip install tlslite, I receive the following error message:

daniel: ~/code/tlsfuzzer > PYTHONPATH=. python2.7 scripts/test-record-layer-fragmentation.py
Traceback (most recent call last):
  File "scripts/test-record-layer-fragmentation.py", line 8, in <module>
    from tlsfuzzer.runner import Runner
  File "/home/daniel/code/tlsfuzzer/tlsfuzzer/runner.py", line 7, in <module>
    from tlslite.messages import Message, Certificate
ImportError: cannot import name Message

OS: Debian testing (x64_86)

Python2: Python 2.7.10+

pip list:

argparse (1.2.1)
chardet (2.3.0)
colorama (0.3.3)
Cython (0.23.2)
distlib (0.2.1)
gmpy (1.17)
gmpy2 (2.0.7)
gyp (0.1)
html5lib (0.999)
pip (1.5.6)
pycrypto (2.6.1)
pycurl (7.19.5.1)
Pyste (0.9.10)
python-apt (1.0.1)
python-debian (0.1.27)
requests (2.8.1)
scapy (2.2.0)
scapy-ssl-tls (1.2.1)
setuptools (18.4)
six (1.10.0)
tlslite (0.4.9)
tlslite-ng (0.5.0)
urllib3 (1.12)
wfuzz (0.0.0)
wsgiref (0.1.2)

I'm not familiar with python, could you give me a hint?

Thanks!

Collecting network traffic

Allow tlsfuzzer to collect network traffic (using tcpdump or similar) to aid in later debugging and to perform timing analysis.

Issues found while integrating in a test suite

As it is now including this test suite in an upstream project has some issues, which I'll try to summarize.

• The output of the test suite is too verbose and it is hiding the actual failures. Running it provides several pages of output and in the end it prints something like:

Ran 65 scripts
good: 45
bad: 20

However, that summary isn't helpful. As an implementer I want to see what failed (and why). With that summary one has to go through many pages of output to see what failed. A better format of output is the one of automake's "make check".

test1 [PASS]
test2 [FAIL]

and test2.log will contain output to be checked.

• Going through the output I see several failures as Alert description 49 != 50. These seem like pedantic checks, but do not provide any information at all, i.e., why is this alert is expected not not the received? A flag -pendantic should be used to enable or disable such strictness.

• Assertions are hard to trace back. The Alert description error described above traced back to a file but no line information was provided.

• The input file contains all the tests an implementation must run. That works well, but is not future proof. Any tests added in the future will not be run by the test suite. That is quite important, as it indicates that this input file format is going to change in the future to address this, and thus prevents me from including that in a test suite.

Running system under test from tlsfuzzer

Allow the tlsfuzzer to run the SUT by itself, needed to run the SUT with valgrind or to collect core files.

While the tests/scripts_retention.py essentially can do that, it's quite stupid about it – can't select key files (or generate them itself on demand), restart the server when it crashes, etc.

depends on #58

test-unsupported-cuve-fallback.py: uses secp192r1

This is not a universally unsupported curve. In fact gnutls supports this curve when run on debian, thus the debian part of the test suite would fail. Switching to secp160r1 or even secp128r1 would address this for gnutls at least.

script to resume session but with different cipher

try to resume TLS 1.2 session with a Client Hello that omits cipher selected in previous session

Script to reproduce CVE-2015-7824

Reproducer for CVE-2015-7824 (https://botan.randombit.net/security.html)

Script to test client certificates when they are optional

check how server behaves if the certificates are optional, check if sending empty Certificate is mandated

Support for Status Request

Add support for status_request extension (OCSP stapling)

the handling of it in TLS 1.3 is very different compared to TLS 1.2 – the extension is attached to single certificates and is advertised by server in CertificateRequest message.

Test with RSA keys that have modulus size not multiple of 8

Check if server can handle RSA moduli that are not mulitple of 8, e.g. 2049 or 2084. It may be problematic especially for RSA-PSS signatures.

test-certificate-malformed.py: expecting decode_error alert instead of bad_certificate alert

Several of the tests in test-certificate-malformed.py fail if anything else than decode_error is returned.

That however is a bit arbitrary. The decode_error refers mostly to messages according to its description, and there is a specific alert for corrupt certificates, that is the bad_certificate alert.

   bad_certificate
      A certificate was corrupt, contained signatures that did not
      verify correctly, etc.

Tests for Application-Layer Protocol Negotiation with session tickets

Test ALPN with session tickets, extend scripts/test-alpn-negotiation.py to check if ALPN can be negotiated correctly when session tickets are in use.
Use existing examples in the script that use session_id based resumption as template.

blocked by #52

Tests for Next Protocol Negotiation

Add support for testing NPN

• also check invalid names
• verify that server rejects client sent EncryptedExtensions if client didn't advertise NPN

client_random always fixed to 00...00

In oder to have real client random, messages.py chould be changed:

diff --git a/tlsfuzzer/messages.py b/tlsfuzzer/messages.py
index 231b9d9..c5bb825 100644
--- a/tlsfuzzer/messages.py
+++ b/tlsfuzzer/messages.py
@@ -24,6 +24,7 @@ from tlslite.bufferedsocket import BufferedSocket
 from .handshake_helpers import calc_pending_states
 from .tree import TreeNode
 import socket
+import random
 
 
 class Command(TreeNode):
@@ -367,7 +368,7 @@ class ClientHelloGenerator(HandshakeProtocolMessageGenerator):
         if self.session_id is None:
             self.session_id = state.session_id
         if not state.client_random:
-            state.client_random = bytearray(32)
+            state.client_random = bytearray(random.getrandbits(8) for i in range(32))
 
         extensions = None
         if self.extensions is not None: