crossbario/autobahn-python

WebSocket and WAMP in Python for Twisted and asyncio

autobahn-python's Issues

cookie support

support getting/setting cookies in WS clients/servers

http://tools.ietf.org/html/rfc6265

make invalid close frame handling consistent

receiving a close frame with payload length 1 will always drop the TCP, while payload length 126 will respond with 1002 when failByDrop = False

the response to a protocol violation during the closing handshake, here when peer iniated close and we receive a close frame with invalid payload length should be

1. either 1002 (failByDrop = False) or
2. drop tcp (failByDrop = True)

make sure this is how AB behaves.

same with

• invalid utf-8 as close reason
• invalid (reserved) close code

Thus:

invalid payload length => 1002 or drop
reserved close code => 1002 or drop
invalid close reason => 1002 or 1007 (?) or drop

For the last one, decide which of 1002 or 1007 to do:

Argument for 1007: "one sends the most specific reason, the other sends the least"
Argument for 1002: "somehow more consistent with the others"

Go with 1007!

But allow both 1002 and 1007 for the test cases.

variant of 6.4.2

add new case variant of 6.4.2 where the middle frame is large (64k)

open - chopped handshake + WS message

send handshake chopped up, followed by message frame

openingHandshakeTimeout

a client should timeout upon not receiving a handshake reply
server should timeout after tcp connect, but no handshake request

probably fail both client and server simply on "handshake not finished within X secs"

close - close code only : allowed / reserved

when payload length == 2: test the different allowed codes and different reserved codes

reserved = 0-999, 1004, 1005, 1006, 1011-2999, >= 5000

close - ignore frames after close

test sending close frame, then pings or messages after that: peer must ignore everything after the close frame

close - invalid close reason (not utf-8)

when payload length > 2: the rest must (?) be valid UTF-8, test sending invalid UTF-8 / binary data

sendClose API

sendClose => 3 variants are needed:

1. user level
2. WS-internal level
3. fuzzing/raw level

"1." restricts close code to 1000 and 3000-4999 like the browser client api

"2." restricts close code to [CLOSE_STATUS_CODE_NORMAL,
CLOSE_STATUS_CODE_GOING_AWAY,
CLOSE_STATUS_CODE_PROTOCOL_ERROR,
CLOSE_STATUS_CODE_UNSUPPORTED_DATA,
CLOSE_STATUS_CODE_INVALID_PAYLOAD,
CLOSE_STATUS_CODE_POLICY_VIOLATION,
CLOSE_STATUS_CODE_MESSAGE_TOO_BIG,
CLOSE_STATUS_CODE_MANDATORY_EXTENSION]

"3." does not restrict close code at all

close - close in middle of fragmented msg

test close handshake in the middle of a fragmented message (one that isn't finished yet)

exclude cases per agent

JSON roundtrips

JSON roundtrip tests ... need to see if that is possible in that generality

utf-8 fail fast

UTF-8 fail fast => differentiate 4 behaviors:

  1. only when complete message was received
  2. only when complete frame was received
  3. only when complete octets for encoded integer was received
  4. as soon as invalid octet is received

Currently, there are tests to distinguish between 1., 2. and 4.

So for completeness, there need to be tests to differentiate between 3. and 4.

HTTP authentication

support HTTP authentication:

• basic
• digest
• cram-md5, cram-sha1, cram-sha256

limits: send 2**63 frame

sending a 2^63 frame using streaming producer .. case must use streaming API.

check if and when the peer implementation bails out.

limit parallel connection attempts

The spec says:

"There MUST be no more than one connection in a CONNECTING state. If multiple connections to the same IP address are attempted simultaneously, the client MUST serialize them"

close - timeout on peer-reply close

make the peer initiate close ("extended test protocol"), ignore the request (never reply to close), test timeout of peer on close reply

test suite exception generating reports with timeouts

On cases where the test suite reaches it's timeout on client tests (ex: test 1.1.6) it writes a KLE type message to the wire log. The report generator can't handle this and throws an exception.

File "/Library/Python/2.7/site-packages/autobahn-0.4.3-py2.7.egg/autobahn/fuzzing.py", line 1002, in createAgentCaseReport
        raise Exception("logic error (unrecognized wire log row type %s - row %s)" % (t[0], str(t)))
    exceptions.Exception: logic error (unrecognized wire log row type K - row KLE)

I believe the offending line is here:

https://github.com/oberstet/Autobahn/blob/master/lib/python/autobahn/fuzzing.py#L203

self.wirelog.append(("KLE"))

should be

self.wirelog.append(("KLE", )) ?

close - multiple close frames

test sending 2 or more close frames: peer must only respond to first with close reply and ignore (fail?) the others

hybi-17 support

http://trac.webkit.org/changeset/97249

Reports Database

Persistently store all run cases / reports into SQLite.

Currently, everything is stored only in run-time Python objects, and when the fuzzer ends, that state is lossed. Users have to run everything they want to test in one session, and there is no history of reports.

close - invalid payload length

payload length = 1 or > 125 must be denied

TLS client cert authentication

support / write demo for authenticating clients on WSS connections via client certs

close - does peer echo close code/reason?

when the fuzzer initiates close with code/reason, what is the peer replying as code/reason?

non-char codepoints

exhaustively test UTF-8 non-chars:

EF BF [BE-BF] or F[0-7] [89AB]F BF [BE-BF]

(any codepoint whose low 16 bits are 0xFFFE or 0xFFFF ??)

see: https://bugzilla.mozilla.org/show_bug.cgi?id=686312

server-to-client masking

mask server frames, check that client fails

direct command mode

In "case mode", a test case will completely be driven by the case implementation class running in the fuzzing client or server.

In "direct mode", the testee will command the fuzzing client or server to "fuzz back" by specifying what to do .. including things that break the WS spec.

forward onCloseFrame to Case

forward the onCloseFrame we introduced in websocket.py via fuzzing.py to the case - at least to have the option to override in the case

split 6.4.2

split out 6.4.3 to a new test section: "Implementation Limits" => see how behaves

don't send / ignore receive after close

The spec says:

"""
After sending a control frame indicating the connection should be
closed, a peer does not send any further data; after receiving a
control frame indicating the connection should be closed, a peer
discards any further data received.
"""

Make sure AB behaves like this under all conditions.

close - pings, close, pings

test: sending 5 pings + close + 5 pings in one go (fast): see what the peer pongs back

client-to-server masking

send unmasked frames from client, check that server fails

proper url encoding for path

https://github.com/oberstet/Autobahn/blob/master/lib/python/autobahn/websocket.py#L1853

=> urllib.quote or urllib.quote_plus

masking performance optimization

Further optimize masking:

http://stackoverflow.com/questions/2119761/simple-python-challenge-fastest-bitwise-xor-on-data-buffers

Probably check out NumPy for masking

https://github.com/kanaka/websockify/blob/6e263063c2b63642a0e17cb91d6b9f6c9365c0c0/websocket.py#L293

overlong payload length

peer must fail on overlong payload length encodings:

send frames having non-minimal payload length encoding, check that peer fails

reserved close code behavior

assumption: using reserved close code values is a protocol violation.

reserved close codes = 0-999, 1004, 1005, 1006, 1011-2999, >= 5000

the response to a protocol violation during the closing handshake, here when peer iniated close and we receive a close frame with a reserved close code is

1. either 1002 (failByDrop = False) or
2. drop tcp (failByDrop = True)

make sure this is how AB behaves

close - protocol violation after close hs start

make peer initiate close ("extended test protocol"), but respond with something violating the protocol (i.e. a fragmented close reply, a close frame with invalid code/reason).

see how the peer acts when confronted with this "double error situation"

Allow test suite usage without installing the package

I'm using Auobahn's test suite to test Webbit's WebSocket implementation. In order to easily update the test suite when it changes we're thinking about including it as a git submodule.

It would be nice if we could run it without having to install the package, simply by setting PYTHONPATH. This is currently not possible since lib/python/autobahn/fuzzing.py uses pkg_info to get the autobahn_version.

Would you be willing to add a fallback solution for this? For example falling back to "UNKNOWN" or something if the package is not installed?

Cheers,
Aslak

frame/streaming API - FIN in beginMessageFrame

allow FIN = true on beginMessageFrame instead of endMessage

currently, with the frame-based/streaming API, the final frame is only marked in endMessage which means the last frame will have payload = 0

client reconnect frequency throttling

limit client reconnect frequency:

truncated binary exponential backoff

automatic pings

one-for-one
one-for-all

randomized
deterministic

close behavior descr in reports

in the detail report table for close behavior, add a 3rd column with description of the rows taken from the comments in the code

timestamps in wireload

add ms timestamps for rows in wirelog

close - timeout on server-drop TCP

when the fuzzer == server, send close, receive client response, but then test client timeout on server-drop TCP after close

hybi-17 support

hybi-17 was landed last week to WebKit: http://trac.webkit.org/changeset/97249

This is a backwards incompatible change. Does Autobahn support it?

If not, what are the best ways to go about adding support for it? (Where do I begin?)

WS settings page

generate settings page for report (with WS protocol option settings)

extended close codes

Extended Close Codes: Reason + Reconnect (ms)

• internal service error
• service restart
• service overload

close - pings/msgs after close

make the peer initiate close, but then ignore that and continue by sending messages/pings.

does the peer reply to those?

does it time out on his close iniation and fail the connection?

extended testee protocol

Probably, we need to have a way to instruct the tested peer to iniate a closing handshake to test certain things (like timeouts).

Currently, a tested implementation only needs to echo back any message (text or binary) it has received.

We slightly extend that: when a text message is received with first char = '#', then the rest is a command, like i.e. "#close".

A tested implementation that only implements the echo would echo back the command, and the test case sending the command could recognize that the peer does not implement the extended testing commands.

standard testee protocol: peer must echo anything (like today)

extended testee protocol: as above, unless msg = text, and msg[0] == "#"

the rest msg[1:] should be the command: comma separated key:value pairs

where key, value = us-ascii7 without "," or ":"

when a testee does not understand the extended protocol, it'll just echo .. so the fuzzer/case can then recognize that the peer does not speak the "extended test protocol".

open: msg[0] == "#"

=> test protocol command ... either from peer-to-fuzzer or fuzzer-to-peer

is there any conceivable scenario where the tested peer wants to "command" the fuzzer?

note that we want to have "direct mode" also .. so above would only be : does the tested peer want to "command" the fuzzer when running in "case mode"?

Testsuite Web UI

The fuzzing_server already runs an embedded web server (Twisted Site).

This could be extended to allow configuration/access to "Reports Database" (see issue #46), and also to interactively start fuzzing_clients.