mozilla-services/autopush-loadtester

Autopush Load-Tester

Autopush Load-Tester

The Autopush Load-Tester is an integrated API and load-tester for the Mozilla Services autopush project. It's intented to verify proper functioning of autopush deployments under various load conditions.

Please note, that this application is not installable via PIP or any other python installation application or process.

Supported Platforms

This application should run on most Linux distro(s). Though we provide some notes for OSX users (see below), please note that we only support usage of this tool on Linux.

Getting Started

This application uses PyPy 5.3.1 which can be downloaded here: http://pypy.org/download.html

You will also need virtualenv installed on your system to setup a virtualenv for this application. Assuming you have virtualenv and have downloaded pypy, you could then setup the loadtester for use with the following commands:

Linux:

$ tar xjvf pypy2-v5.3.1-linux64.tar.bz2
$ virtualenv -p pypy2-v5.3.1-linux64/bin/pypy apenv

OSX:

$ tar xjvf pypy2-v5.3.1-osx64.tar.bz2
$ virtualenv -p pypy2-v5.3.1-osx64/bin/pypy apenv

Activate Virtualenv:

$ source apenv/bin/activate
$ pip install --upgrade pip

The last two commands activate the virtualenv so that running python or pip on the shell will run the virtualenv pypy, and upgrade the installed pip to the latest version.

You can run this program as a program to run test scenarios you create, or if adding scenarios/code to this application continue to Developing.

Program Use

Run the basic scenario against the dev server:

$ aplt_scenario aplt.scenarios:basic wss://autopush.dev.mozaws.net/

Run 5 instances of the basic scenario, starting one every second, against the dev server:

$ aplt_testplan "aplt.scenarios:basic,5,1,0" wss://autopush.dev.mozaws.net/

Either of these scripts can be run with -h for full help documentation.

See SCENARIOS for guidance on writing a scenario function for use with this application.

Developing

Checkout the code from this repository and run the package setup after the virtualenv is active:

$ pip install -r requirements.txt -e .

See Contributing for contribution guidelines.

Notes on Installation

'openssl/aes.h' file not found

If you get the following error:

$ fatal error: 'openssl/aes.h' file not found

Linux: You'll need to install OpenSSL:

$ sudo apt-get install libssl-dev

OSX: Apple has deprecated OpenSSL in favor of its own TLS and crypto libraries. If you get this error on OSX (El Capitan), install OpenSSL with brew, then link brew libraries and install cryptography.
NOTE: /usr/local/opt/openssl is symlinked to brew Cellar:

$ brew install openssl
$ ARCHFLAGS="-arch x86_64" LDFLAGS="-L/usr/local/opt/openssl/lib" \
  CFLAGS="-I/usr/local/opt/openssl/include" pip install cryptography

missing required distribution pyasn1

If you get the following error:

$ error: Could not find required distribution pyasn1

re-run:

$ python setup.py develop

autopush-loadtester's Issues

fix the notification_forever_unsubscribed scenario

This scenario is currently no different than notification_forever. It's supposed to unregister the client before beginning the forever loop, but due to a bug it doesn't actually do so (it's missing a yield statement):

https://github.com/mozilla-services/ap-loadtester/blob/6a769/aplt/scenarios.py#L194

I don't understand the intention of the scenario -- if the client's unregistered it won't be accepting the notifications later in the loop. The scenario should be rethought or removed

Add ability to yield generators

To allow tests to be broken up into parts, a test should be able to yield a generator which will then be executed before its resumed.

Fix references to old autopush-dev URL in tests, README

Create "spawn all" scenarios to simplify API verification, loadtesting

We need to have a way to spawn all the scenarios at once (with desired params) for these situations:
[1]. API verification

• aka: run all scenarios once, then stop
• def _api_test()

[2]. loadtest

• aka: run all scenarios continuously to generate load
• def _loadtest()

Per benbangert, we can create 2 additional 'wrapper' scenarios that spawn the others with the desired input params:
https://github.com/mozilla-services/ap-loadtester/blob/master/SCENARIOS.md#spawn

Add notification_forever_with_vapid scenario

VAPID support was added to ap-loadtester, but we still need a scenario to test this.

from the README:
send_notification
Send a notification to the push service. If an empty data payload is desired, None can be used for data. If VAPID headers are desired, include a valid claims dict.

Package for Lambda runs

It'd be useful to be able to easily deploy ap-loadtester to run as an AWS Lambda function for regularly end-to-end verification.

Persistent reconnects of failed instances

There should be an option for instances that throw an exception (unintended disconnect, etc) to reconnect in the event of a disconnect. The Load runner harness should be the one that checks to see if instances failed and respawn new ones per the original schedule for that portion of the test plan.

adapt to stricter header validations

autopush 1.22.0 includes stricter validation of the encryption headers, which aplt currently fails.

aplt probably doesn't need a fully compliant push notif payload, but something at least slightly more compliant

Add metric reporting function

txstatsd + yielding a counter and a timer should be an option for the scenarios

Send a more valid Origin header

The latest autopush (autobahn) denies the hardcoded 'localhost:9000'

Update README with scenario descriptions

As-is, a user would have to poke around in the aplt/scenarios.py file to know which scenarios are available for execution. Probably better to document their usage in the README.

Run once
def basic(*args):

Run forever
def connect_and_idle_forever():
def reconnect_forever(reconnect_delay=30, run_once=0):
def register_forever(reg_delay=30, run_once=0):
def notification_forever(notif_delay=30, run_once=0):
def notification_forever_stored(qty_stored=32, ttl=300, notif_delay=30,
def notification_forever_direct_store(cycle_delay=10, run_once=0):
def notification_forever_unsubscribed(notif_delay=30, run_once=0):
def notification_forever_bad_tokens(notif_delay=30, run_once=0,
def notification_forever_bad_endpoints(notif_delay=30, run_once=0):

Multi-Scenario
def api_test():
def loadtest():

OpenSSL.crypto.Error ('no start line') w/ muti scenarios and ENDPOINT_SSL_CERT

multi scenarios (scenarios that call spawn() on other scenarios) trigger a mishandling of re-reading the ENDPOINT_SSL_CERT/KEY env variables:

  File "/home/pjenvey/src/python/pypy2-v5.6.0-linux64/lib_pypy/_functools.py", line 45, in __call__
    return self._func(*(self._args + fargs), **fkeywords)
  File "/home/pjenvey/src/moz/ap-loadtester-env/site-packages/OpenSSL/_util.py", line 48, in exception_from_error_queue
    raise exception_type(errors)
OpenSSL.crypto.Error: [('PEM routines', 'PEM_read_bio', 'no start line')]

env var data's read through a StringIO object: it's not being reset for the re-reading.

Add scenario for verifying notification topics

connect_and_idle scenario drops bulk connections (intermittent)

On long test runs (>30 mins), ap-loadtester will drop a large number of connections (not usually all). This usually happens at least once on long-running tests, but can happen multiple times.

I've never been able to get a connection test to exceed 11M connections.
https://github.com/mozilla-services/ap-loadtester/blob/master/aplt/scenarios.py#L44

Allow passing a list of expected channels to expect notification

For tests when its uncertain whether the notifications are in order, there should be a way to pass a list of all expected channel ID's, and any of them are ok.

Passing args to a scenario using aplt_testplan doesn't work

Use dashes in uaid/chids per UUID normalization

autopush now normalizes chids/uaids to include dahes, only channels made with them should be submitted.

Shutdown ap-loadtester after time n

This would facilitate moving control to ECS task scheduler
NOTE:
Timeout should already be defined in an env var.

fix Docker + pypy-v5.3.1

We encountered a dependency issue w/ the new pypy-v5.3.1 inside of Docker.

Switch its base docker image to pypy:2 instead of plain debian should solve this and simplify the Dockerfile

Update README to clarify supported OS

The README currently offers some notes for supporting OSX, but since we won't be able to support any platform folks may want to run this on, we should explicitly state what we do support (aka: Linux).

Peg dependencies to avoid breakage

Not enough dependencies are isolated, as such the latest autobahn broke things.

Detailed user documentation

Someone should be able to learn how to write a detailed scenario and run it with the commands available.

aplt_testplan throws intermittent twisted:Unhandled Error

running aplt_testplan w/ api_loadtest, basic scenarios on OSX

INFO:twisted:Starting factory <twisted.web.client._HTTP11ClientFactory instance at 0x00000001076058c0>
INFO:twisted:Handling websocket data:  {u'status': 200, u'channelID': u'022e5015-0994-49f5-8143-6e88876cce46', u'messageType': u'register', u'pushEndpoint': u'https://updates.push.services.mozilla.com/wpush/v1/gAAAAABXkQwgidYSCDQGTGrieuy8RIpTSS0xtmCvDGN97zMoiizRX88x_cv74yBdGEQCbMVPhUM3-WJ5BnPCvBTGrqUj8ClpxQmYuqRaNn-9500JvSoeQnW-Gmw7T1OSmxlRVkSoAVIr'}
INFO:twisted:Running command:  TimerStart(name='update.latency')
INFO:twisted:Running command:  SendNotification(endpoint_url=u'https://updates.push.services.mozilla.com/wpush/v1/gAAAAABXkQwgidYSCDQGTGrieuy8RIpTSS0xtmCvDGN97zMoiizRX88x_cv74yBdGEQCbMVPhUM3-WJ5BnPCvBTGrqUj8ClpxQmYuqRaNn-9500JvSoeQnW-Gmw7T1OSmxlRVkSoAVIr', data=None, ttl=60)
INFO:twisted:Starting factory <twisted.web.client._HTTP11ClientFactory instance at 0x00000001062f84a0>
INFO:twisted:Handling websocket data:  {u'status': 200, u'channelID': u'e5b1225b-2c5f-41de-818e-155651b6d507', u'messageType': u'register', u'pushEndpoint': u'https://updates.push.services.mozilla.com/wpush/v1/gAAAAABXkQwgseDUMiD45fyfTKx8yZKXcKnoBB-4gvc4n903Lam0P1s5vsUZH0kjoylvDMscW4tmC2clV1VzyYrUAGpLr7pbVQx0hTzdVBK4erZWk7g9ZI9upoNeJd1MqfJwcfIA9pkM'}
INFO:twisted:Running command:  TimerStart(name='update.latency')
INFO:twisted:Running command:  SendNotification(endpoint_url=u'https://updates.push.services.mozilla.com/wpush/v1/gAAAAABXkQwgseDUMiD45fyfTKx8yZKXcKnoBB-4gvc4n903Lam0P1s5vsUZH0kjoylvDMscW4tmC2clV1VzyYrUAGpLr7pbVQx0hTzdVBK4erZWk7g9ZI9upoNeJd1MqfJwcfIA9pkM', data='\xe1$\x17\x1b$\x84[\x06\xcc\x85\xc7C\x96%7\xdf\xc4\x15\xce\xb1dr::f\xf4\x0e\xe0"\x83\x90\xa7\xeds]\xd1\x06Hh1\r\x0e\xa8\xcf\xeb\xda\xe2\x94D\xb2\x0b2v\xdf{U\x02\xf9\xadC\x198%I!\x9e\xa6\xa9V\x87\x9b\xc8\xf7(\xb6\xa0\xcf\x7fF_#\xe1\xe8\xc2\xf5\x83\xa0\xdfj\x11\x1f\xab\x86\xf3\x0b\x96\\d\xfe\x8aYu\xc0\xb9:\xd6\x030 \xf4\x9b\xf0\xde\x19O\xc4\xbf\xb8\xf5\x95.\x85z\x9a\x02nO\x0c\x8cX\xfb\x9du\x00\xc9\xea\xb2\xb6\x1dV\xc9\xd3\xf1\xe0s\xec\xd9\x18\xf9,\x84V\x01\x16N\xaecB\x1e/\xa1\xd0!6\xc8\xe1\xc9\xa2\xbc\x19R\xb8\x9fH\xc5\x19b%\xe8\xf5[\x19\x16)H{\x10\xe0\xa5\xa9\x1a\xa6\xe1\x8d\x93~0@8\x9b\x11$F\r\x0e\xe2\xf8\x0e6\x0e\x1f\xe8<\x97O\x10\xf9T\xab\xdb\x945\xceD\xb09]JR\xee\xab\x13\xa4\x01\x17r\xfc#\xc7\xd4\xa5\x17\x95PIR\xf1>\x01\x9fCcH\xc8\x80\x9b\xdd5\xc0\xed\x9d\x07\x87R\xee\xa4{\xeel\xdc\x8a\xbf\x02\xf0\xe0\x15\xe7\x89\x00-\xc5\xc7\xd6\xa5\xd7\x07\x9d\x04\x8f\xaa\xf8+lRj\x96\xe0\xa8\xb2:#d\xf6\x81\xaam\xcf\x91\x01\xd7\x93\xa4KhGc\xb8|\xad\xb9\x11\xd4\xce\xb5_\x05Q\xe8\xcfb#\xb2*\xd3\x069\xba\x13\xf1\xe8\x91u\x14\x1c\xb8B~2\xc1-\xa5[:#\x86\xcea\x85\x0b\xa2\xd9@/\xd4\xcf>1\xc5$\xb6Y}\xe9*\xf0"\xd2\xc060\xff\x9e)\xe7]\xe6\xaa-\xb4[\xbe@I\x99\x86\xf23\x08&\x8c\x8dF\xe9a--E\xc5_\xf9\xad\xf0B\xbe\xf5>\x8d\xd15\xd1\xc8\xe4>\xaa\xa5:[u\xac|\xb2\xce\x9eXw\xden\x9b_\xce\xc4\xbc\xc6\xa7o\xe6m\xd2)\xfb\x92+4\xc0\x9b\x0f\x02\x08\x1bHM\xac7\x80\xca\x9e\x8b\x8d\xd3Br\xeeG\xa1d\xb2F3\xf9S\x91\nK|\xa1\xc6\x8f\xf1\xb7\x8a\x01\xc6\r/\x14k@\xbc\x16G\xa8\xf6Q\xab\x8bh]\x8c\x8d\xab\x08\xef\xf0J<\x98p\x94\x1em\xbcT\x82\x85\x9cj_\x98\xfc\x87\x82\x10\xf4\xa1T\xd3\xedb~\x81\x7f\x86\xb04~\'\x1eT{\xa4 E\n\xb7~<\x18\xd9r\x00dkV\xf8K*\x8dwMfrbQV\x05\x05\xf0n\x9e\x9cK\xac\xf1s \xa4B\x1d\x9aV\xf6\xd1fc\xb2\xecL\xd8\x12[\xb9$\xf6r\xf3\x06\x04\x91\x04_y)\xcb\x10\xc4\xf7\x08x\xa0\xd2",\xc6~\x17C\x07\xc7`\x0c\xb3\x14\x1cg\x96F\x08bekGw\x97\xc1\xed\xb6\x87\xe3\xd5\xac\x16^\x80\x9fNA\xf6\x9f=\x7f\xba\xa7\xc5\x860\xe0\x05\xac\xcb\xe2\xaf\x1c%\xd1\x1en\x95\xc8\x14g\xe45\xf8\xfaU\x06\x84\x7f\x948\x7f|\xd9@\xbc\xa2\x16E/\xcc\xc8\xb8\xdf\x97\xd4\x02H\x01\xba\xac*\xf7+\x94\xef\xe0\xa6]\x01\xd3\xfa\x85\xd5<\xc5\x8fY4\x14)\x14\x10c\xea\xd0\x9d\xe8\xcb\xd7\xb8\x1ap\xd5D*\xd8g\x03\xbd\x7f\xf4\xd7\xa6\xd6\xd6\xf1\xba\x1ac\x08w\x1b;Q\x96\xbf\xf6\xc17\x17\xd4\xf8\xc5\x87\xa1XRD\xc8\xc9\x1ed\x844\x91\xc8\xec|\xa1c\r\tyP\xe8\x08\x04&\x17\x07\x8ax\x06\xbb\x99\xf6\xa0\xa9\n\x05\x19\x9ag\xc3\x88\x14\xde\x16\x1e\x05H\xaa!W\xc7\xb0|\xb5\x05\xdc\xcd\x94t\xa6\x1a\x9dO\xf53\x15QK&\x9b\xfck\x0e^E\xb0\x9d$\x87\xd5\x08\xbcv\xb3\xbb\x10\x83\xfc1\xf07e\xe9\x85!\xfe\xb08}`|\r\x18\x8d\r\x18\xc5\x9ce\x83\xae\x9a\xdbE\xdel\x86`\xdd\xc2\xbb!\x7f5\x02lP\'\xfa\xd9~\x97\x9f$?S\xf2\xbd-\xa0\x93\x82\xd7\x17\xc6/\xb7\xc4\xa9\x1b\xbe\xf7\xec\xbe}\x85\x93\xc5\xa4\xee\x85\xca\xba^\xecYH\x8d\x9d\x9c\xe5\xab\xd2V\xf6jN-\x7f=\xf9p.\xd7\xdfdqY\x95\xe5\x9f\x81\x02\xcb\xc57\xd5\xbe1\xf8IS\x1f\x93\xca8\xe2\x06\x13uf{\xee\x7f\x03\xa8\t#\x8e\xa8\x0f\x0b\x02P\x06m\xde\xf4M\xf8\xe7\x90\xe3\xb1\xf5\xe2\xb3~*\x9f,\x0e\t\x9eg\x91\xfb\x8c\x1bl8R\xb8f E(mh\xc1\x03\x7f\x8a\x10\xa0\xd5\xc3\r\x01c\x18`R\x1cP9\x8a\xe7gd4~\xdf)=\x9d\x02\x93\xf9\xf4\xf6L\xda\x99>\x82\xa6\xe8&p\x83\xa1\x17uuY"\x91\x12\x9b\'\xf8\x853X\x9b\xafG\xcc\x03|\xac\xe2\xc7\xff/\x16\x81\xbc;\x03\xae;\x9d\x1b\xf8ws\xf4I\x84]\x00og\x10\tuCi\xd0\'g\x9f\x0e\x9a#D6S\x9e\xff\x92\x9a\x0c\xb1\xc6\x93 CZ\xbc?\x9eN\x00\xb6\xb0\xc3\x06$\xeajvM\x08\xc6MG\xdc\x99\x02\x82\xdcRs\xca\x0fE,f\xeb\xca>\x81{\x11\xd1C\x1d\xa9\xf4\x91\xc7\xd7\x1b\xd6\xa6\x80\x9aP\x85d\xd7]\xe6>\x13\x92\x9c\x8bi\x04\xf9\x98\xdb\x01\xe1\x08#\x08\xffQ%\xe4\x8cp(z\xd6\xf8d\xf6\xcf \xd2\xaa\x05xa\x83\xbeR\xc2\xa7\xe8k>\xf7\xbes\x8e\xedlx\xfb~\x93\xd6\xfaY\xb3\x9a"\xf2\xf0o\xc2k\xf2\x9b\xd5\x86\x1f\xe9\xaac.\x18\xa8\xcfB\xdf/GD\xf9Y8\xc2\x03\t\xb9\x00\xdb\xdb\x0b\x88\x1d4\x83 \xfe\x9ad\xa5\xa2\xce\x83\x10=\xe1\x00\xb8\x1f\xc8\xea\xfb:\x7f\xdaO2M\xf7\x1d\xbe\x16rx\xd2%j!%\xe0IM$;[\x1d)}\xe0V\xcd\\\xde\x1f\x7f\xb8\x9aS)\x1d\x03\x05W\xe3`\x08^\x82\xd1;\xd7\xd6\x19\xef\xe9\xb1\xdb\x87\x9d\xcfj\xf9\x13\xda ,\x1e\xd9/f\xb4\xf9\xf7x\x1b\xe6hk\xb1\xbb#\x8b\xbfk\x02@av\x9a\xac\x90\x94\xd2\xaa\x8d\x93\xaf\x12\x9c\x8ci\x9a&Z\x1f\x06\x92\xfe\x17\xe9M\x10\x96a>\x02\xcf\x9d\x8f}I\x00\x1a\xc3\xbe\xf47S\xaa\xc4\x12\xadEl\x8bg\xbb\xa8\xacP\xa4\xd3}\x99\xe7?\x8cIKb\xf8\xec\xe7G|\xe2Y?\xfe{\xa18\x86\x06\xd9\xe2r\xbb\xc7q(\x83\x8a\xd8=+\xe2\xbc\xf0 \xe0)\xa27\xe1\xc6\xe4\xb0\xc8\xef}=\xcb\x1fd\x93\xb4$\x7f\x91\x15\xc4\xa9\xedS->e\'\x0e\\76BHL_A\xd0y\xf0\xe5Y\xf5\xad\\\xd7\xfd\xd5\xe1\xfa\xa0\xd7\x1b\xba\xf0\x1a_C\xb7F\x8670\x98\x85\xde\xbaI\x9c\x05\x04\xccF<+kS\x85u^A\xbe\xa8\x1bF\x11\xa9\xd8C\xcds\xf9\x07\tc\x88,\x82\x02"\xbdf\xe5r\xa1\xdf\xba\x07\x9f\x80\x845\xac\x04\x89x:\xe7\xd0\x04\xbd\xb2\xedq\xa7&\xf7\x8f\x88\x92\xbfF\xf2\xda\xc9\x0b\xe8*\xb2\xb7\xd9\x009J\xd5\xa3\xdc\x97g\xc8\xe8Ct\xdfp\x84\xfe\xf1\x1d\xff\xf1}W\xae \x06\x99\x9d\xa1\xac\xf7%m9\xac\xdf\xb2\'x\xbc\xa9\x96\x02\xc4\xf9&\xa2\xe3\xfe0<\\*\x99\x03\xd3\x07~\xa1{<\xb35\xc4\xf3k\xcd\xa7\xf2\x89\x9b9\xecH\xd7\x80\x8f:\x94\x96;(#\x0c\x03\x8d\xc5Pe>\t\x0e\x16\xc3\x8e\xe1\x9cq>\x08\xb4\xda,\xe6s\x0f%#\x15\x8c\tX\x0cKl\x10\xb4s\x03\xd3\x1bN6K!w\xd8\xca^\xc7>\xa1vD\x14\x0b\x893K\x17\xcdbi\xa2\tP\xfe\x11\xed%\x99\xff\x04#\x80B\xd0\xcd\xa2\xbd\x973\xbf%\xa2\x04Y\xd4\xa9d\xacV\xd6-\xa0j\xd2\xfb\xb5\xdf\xd9\xe9\xa4\xaf\x91u\xa3\xe6\x13M\xebm\xdc\xcbYC\x932^\x89\x18c_\xa2\xceLRO\x02\x0c\x0f\xd2\x9b\xc8.\x9a\xd8kf7\xf0\xe0\xa4#c\x83UtE\xb0\xfeG1]\xc0\xe5(\x8a\xd1\xcf\xe5\xe2\x84Dt\xf1\x89\xf89\x17m\xc9\x97(K\x82l\xbd\xee9\xba\xfc\xa3_\\vB\x01\xd7\xa4\xb3_\xc9Pa\x19\x94\x90\xe4\xd6-\x1f\x0ci\x8d\xa0n\x10\x9f\xb1$\xe6\xae?\xca\x7f\xaf\x08q\xf6D\xbb\xc5\x82\x90Z\xc9>\x9c\x81\x85s,\x97\x80w[\xc4\xaa\x7fh\x8c\x1e\xcf\xa5\x9e\xa8\xc1v\xaf\xaeI7\xa9\x05c\xa6j\xb3G\x16\xc7\xa0)\x1c]\xac\x0b\xcd\x9f B+\xcda\x19F\xe4\xc0vR@c\xc7\xf8\x07\xe4\x0f:|&\xc4\xd8\x03\xd9\x97\xd5\x13\x17\x8f\xb68\xff\xefD\'K\x03\x0f\xa1\n\xc6\xcf\x923Z\xc8M(a\x01_\x1a\xc4\xf8\x1f\xd0\xf0k\xef\xf8\x18\xf7`}\x92\x95R\xea0n\x8d\xf9G\x15\x8dQ\xa1-\xe6+\\Zd5\x8b\xd2\xac\xde\xb6\xc8$"\xa6\x15\xd1\x9f\x05\x1e^\x19qx\xaf%\xfb\xd6\x11\xc7\x84\xe0\'K\n\xe8:u\xc6\x91t?G2\x1d\xfe@\xdf\xe2\xa5O\xa2p\xe3\xa7\xc2\x9a\xc1\x0e\x04b\xb7#\xb9\xff\xae7\x8d\x90)\xe4k\xa2\x9e"\xdb\xfa\x15K\xa2\xc6\xf4\x8f\xd4&\x95fj\x9bM\xc7KY\x86x@\xeb9\xaf\xe3\xb8M\xdaAb\xa9\xac\xdd\xb4\xd0\xd3.\r\xcbBT\xa7\xa3\xb9!\x96(+\xacNQ\xbc\xa0$\x03\xe8&\xe438\xfa\xba\xac\xe2\xc5\xbc?\xe0\x16#E\xdc\xfa\x1e\x7fJ,n\xa0\\\xf6\xb8\r\xb0Z~.\xfc\xdd\x87\x02\xc5d1\xd8\x92\x83\xf3\x0c\xb9\tx\xc1V(,\x91UT\x87\xccl\xb4\x02\x94>w\xee\xb3P\xabE\xbf\x0c\\\xa6\x1d\xdb\xaa8\x1fB\xce\x92\x1e\x19d\x1d\xa1\x1b\xb8\xc2g\x00E\x7fD=p\xdf\xe9F^\xcb\x8f\xec\x04\xbd\x99,`\xb0\xb3\x8b&\x9f\x9eu\xa5;\xd9h\x91\xd0A\x013C>b\xc2\xe34\x11\x08k\xe90\t\xd2|2\xcb\t\x88\xd7\xabd\xeb\x98\x95\xe1A\x11U\xd18@]6\xf9\xbc\xb4\xac\xdd\xc5"]\xd74\xb7\xcc\xbc\xe3\xa3\xad\x1a\x86,_&$v\xd7C\xf7\xa0=b\xb2\xc9\xd3f3\xdc\xc2Pc\x10z\xd8\xd7\xf2l\xc3\xb0\xa3\xe4\xd1\xe1m0\xbe\xbd\x1a;\x88\x0eHzo{Y\xf0\xa5!\x88\xa2T\x8d\x93\x11\xd3\xd2\x96\x8f\x19Z\x8c\xd9\xe02Y\xd7\xa3\xac\xe2\xa7Y\x9bv\xb4\xd8H\xf0i~\xea8\xddx\xe6\xab7\xe2\xc2\xe7\xd3\xac\x85\xb0\x13I\x08\xdf\xd2\xa76\xcc~-\xd8\x02(\xe1\xfc\x91+(\xfe\xfd4\xd6\xc2\xb0\xf0\x06\xd7\x1bd\xf1g\x84\xa9\x81\xf7\x1c9\xb0\xcf\xbc\xf3\xf1\xfa~n\xb5\x88D>\xbc\xff\xc4\xd3+\xa9;"\xd3\x9dGV6"$\x9c3\xac\xb4\xf5]4zylU\xb5\xa5>\xacv%Dt\xcap\xe7\xf0\x12\xc0V+\xfa\xed\x99T[,\xea\x99Q\x96\xca\']o\xc1\xcc\xd9&r2\xe4tQ0\x8aTN\xba\x1a\x19\xc9C\xd9>;E\xf4-\xbb\xe1\xcd\x9f\xdb\x7f\xaf\x077\x18\x95\x8b\x8c\x04\xa5\x03p/\x12\x97\x0c\xb5\x05\x1c\xd65\xce\xf7\x953\xe7\x00\x91x\x1b\x00\x90\xc0\xa6\t\xaf6\x15\x8ft\xc8\xa2_\xd9\x04\x98T\xf1\x7f\xd4\xbd\xb7\x80]\x0fI\xac\x82\xc5S\xb9\x02\x8c\x10]f\r\xfd9\xe4\xa2#X\xea\x04\xfei\xce\x86j\x15NE\xe4z\xd6\xf9\xdc\'\xf3%\x15\xe5V\x19\xe4$\xef*\xb4\xab\xc0=6Mt\x0fH\xee\xa2>\rT,\xe1`Q\xbe\xf7V\xf8\xe0\rXO\x00\x0c1<\xca\x06\x8d\xf4y\xa9-!\x02\xff\x89\x14A*N\xaaS\xd2\x88\x88v\xf6]t\xcf\tp;\xdd\t\x81%\xdd\xf4\xc8Y\xd6\xa9{\xce\'\xae\xc3\xb1\xbc\xae\x935N\x99\xd7\xb0H\xf9\x94\xdf\xcf\xf1g\xd9\x90-\x80\x8c\x84\xd7}]$\xb7 G-\xc2x\x83\xca^\x92U\xf4n\xf1\x96\x1f}1\xa2\x02\xb1U\xf1\xfd\xf0t\x91\x86 \x1a&[\x15\x88z^@\x13\x12m\x94\xa2v\xd9\xa1\x08k\xe5\xbe\x81o\x1a\xbdH\xa9\n\xf6R,3%Ez\x97\x97\t\x84p\xa1\xf97\xd7\xe2A\x8b\x84gE\xb3\xb9\x97\xce\x11y\xdd\xab\xa4\xe6\xf7\x88\xd3\xb0hg\x07\x10\xea]3I\x80#O\xf0\xbfW\xf75\xe5\xe9\x1c\xc6\x89Y\x01\xbcI\xbfB\x98\x0f\x1e\xb7\xc5', ttl=60)
INFO:twisted:Starting factory <twisted.web.client._HTTP11ClientFactory instance at 0x00000001062fb0c0>
INFO:twisted:Running command:  Counter(name='notification.throughput.bytes', count=2633)
INFO:twisted:Running command:  Counter(name='notification.sent', count=1)
INFO:twisted:Running command:  Wait(time=2)
INFO:twisted:Running command:  Counter(name='notification.throughput.bytes', count=2912)
INFO:twisted:Running command:  Counter(name='notification.sent', count=1)
INFO:twisted:Running command:  ExpectNotification(channel_id=u'e5b1225b-2c5f-41de-818e-155651b6d507', time=5)
INFO:twisted:Running command:  Counter(name='notification.sent', count=1)
INFO:twisted:Running command:  ExpectNotification(channel_id=u'022e5015-0994-49f5-8143-6e88876cce46', time=5)
INFO:twisted:Stopping factory <twisted.web.client._HTTP11ClientFactory instance at 0x00000001076058c0>
INFO:twisted:Running command:  Disconnect()
INFO:twisted:Handling websocket data:  {'messageType': 'disconnect', 'was_clean': True, 'code': 1000, 'reason': None}
INFO:twisted:Stopping factory <autobahn.twisted.websocket.WebSocketClientFactory object at 0x000000010695e0c8>
INFO:twisted:Running command:  Disconnect()
INFO:twisted:Handling websocket data:  {'messageType': 'disconnect', 'was_clean': True, 'code': 1000, 'reason': None}
INFO:twisted:Stopping factory <autobahn.twisted.websocket.WebSocketClientFactory object at 0x0000000106532838>
INFO:twisted:Running command:  Counter(name='notification.sent', count=1)
INFO:twisted:Running command:  TimerEnd(name='update.latency')
Unhandled Error
Traceback (most recent call last):
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 117, in _run_safely
    self._run_command(func())
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 142, in _run_command
    command_func(command)
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 256, in timer_end
    self._send_command_result(duration)
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 97, in _send_command_result
    self._run_safely(lambda: self._scenario[-1].send(result))
--- <exception caught here> ---
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 117, in _run_safely
    self._run_command(func())
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 97, in <lambda>
    self._run_safely(lambda: self._scenario[-1].send(result))
  File "/Users/rpappalardo/git/ap-loadtester/aplt/scenarios.py", line 206, in notification_forever_unsubscribed
    yield ack(channel_id=notif["channelID"], version=notif["version"])
exceptions.TypeError: 'NoneType' object is not subscriptable

CRITICAL:twisted:Unhandled Error
Traceback (most recent call last):
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 117, in _run_safely
    self._run_command(func())
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 142, in _run_command
    command_func(command)
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 256, in timer_end
    self._send_command_result(duration)
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 97, in _send_command_result
    self._run_safely(lambda: self._scenario[-1].send(result))
--- <exception caught here> ---
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 117, in _run_safely
    self._run_command(func())
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 97, in <lambda>
    self._run_safely(lambda: self._scenario[-1].send(result))
  File "/Users/rpappalardo/git/ap-loadtester/aplt/scenarios.py", line 206, in notification_forever_unsubscribed
    yield ack(channel_id=notif["channelID"], version=notif["version"])
exceptions.TypeError: 'NoneType' object is not subscriptable

INFO:twisted:Running command:  Counter(name='notification.received', count=1)
INFO:twisted:Running command:  TimerEnd(name='update.latency')
INFO:twisted:Got notif:  None
Unhandled Error
Traceback (most recent call last):
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 117, in _run_safely
    self._run_command(func())
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 142, in _run_command
    command_func(command)
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 256, in timer_end
    self._send_command_result(duration)
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 97, in _send_command_result
    self._run_safely(lambda: self._scenario[-1].send(result))
--- <exception caught here> ---
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 117, in _run_safely
    self._run_command(func())
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 97, in <lambda>
    self._run_safely(lambda: self._scenario[-1].send(result))
  File "/Users/rpappalardo/git/ap-loadtester/aplt/scenarios.py", line 38, in basic
    yield ack(channel_id=notif["channelID"], version=notif["version"])
exceptions.TypeError: 'NoneType' object is not subscriptable

CRITICAL:twisted:Unhandled Error
Traceback (most recent call last):
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 117, in _run_safely
    self._run_command(func())
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 142, in _run_command
    command_func(command)
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 256, in timer_end
    self._send_command_result(duration)
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 97, in _send_command_result
    self._run_safely(lambda: self._scenario[-1].send(result))
--- <exception caught here> ---
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 117, in _run_safely
    self._run_command(func())
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 97, in <lambda>
    self._run_safely(lambda: self._scenario[-1].send(result))
  File "/Users/rpappalardo/git/ap-loadtester/aplt/scenarios.py", line 38, in basic
    yield ack(channel_id=notif["channelID"], version=notif["version"])
exceptions.TypeError: 'NoneType' object is not subscriptable

INFO:twisted:Handling websocket data:  {'messageType': 'disconnect', 'was_clean': False, 'code': 1006, 'reason': u'connection was closed uncleanly (peer dropped the TCP connection without previous WebSocket closing handshake)'}
Unhandled Error
Traceback (most recent call last):
  File "/Users/rpappalardo/git/ap-loadtester/apenv/site-packages/autobahn-0.13.0-py2.7.egg/autobahn/twisted/websocket.py", line 172, in _onClose
    self.onClose(wasClean, code, reason)
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 39, in onClose
    reason=reason))
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 307, in handle
    self._send_exception()
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 113, in _send_exception
    self._run_safely(throw)
--- <exception caught here> ---
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 117, in _run_safely
    self._run_command(func())
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 112, in throw
    self._scenario[-1].throw(*sys.exc_info())
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 305, in handle
    self._raise_unexpected_event(data)
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 282, in _raise_unexpected_event
    self._last_command, data))
exceptions.Exception: Unexpected event. Last Command: timer_end; Data: {'messageType': 'disconnect', 'was_clean': False, 'code': 1006, 'reason': u'connection was closed uncleanly (peer dropped the TCP connection without previous WebSocket closing handshake)'}

CRITICAL:twisted:Unhandled Error
Traceback (most recent call last):
  File "/Users/rpappalardo/git/ap-loadtester/apenv/site-packages/autobahn-0.13.0-py2.7.egg/autobahn/twisted/websocket.py", line 172, in _onClose
    self.onClose(wasClean, code, reason)
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 39, in onClose
    reason=reason))
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 307, in handle
    self._send_exception()
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 113, in _send_exception
    self._run_safely(throw)
--- <exception caught here> ---
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 117, in _run_safely
    self._run_command(func())
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 112, in throw
    self._scenario[-1].throw(*sys.exc_info())
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 305, in handle
    self._raise_unexpected_event(data)
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 282, in _raise_unexpected_event
    self._last_command, data))
exceptions.Exception: Unexpected event. Last Command: timer_end; Data: {'messageType': 'disconnect', 'was_clean': False, 'code': 1006, 'reason': u'connection was closed uncleanly (peer dropped the TCP connection without previous WebSocket closing handshake)'}

INFO:twisted:Stopping factory <autobahn.twisted.websocket.WebSocketClientFactory object at 0x0000000107b13c90>
INFO:twisted:Handling websocket data:  {'messageType': 'disconnect', 'was_clean': False, 'code': 1006, 'reason': u'connection was closed uncleanly (peer dropped the TCP connection without previous WebSocket closing handshake)'}
Unhandled Error
Traceback (most recent call last):
  File "/Users/rpappalardo/git/ap-loadtester/apenv/site-packages/autobahn-0.13.0-py2.7.egg/autobahn/twisted/websocket.py", line 172, in _onClose
    self.onClose(wasClean, code, reason)
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 39, in onClose
    reason=reason))
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 307, in handle
    self._send_exception()
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 113, in _send_exception
    self._run_safely(throw)
--- <exception caught here> ---
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 117, in _run_safely
    self._run_command(func())
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 112, in throw
    self._scenario[-1].throw(*sys.exc_info())
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 305, in handle
    self._raise_unexpected_event(data)
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 282, in _raise_unexpected_event
    self._last_command, data))
exceptions.Exception: Unexpected event. Last Command: timer_end; Data: {'messageType': 'disconnect', 'was_clean': False, 'code': 1006, 'reason': u'connection was closed uncleanly (peer dropped the TCP connection without previous WebSocket closing handshake)'}

CRITICAL:twisted:Unhandled Error
Traceback (most recent call last):
  File "/Users/rpappalardo/git/ap-loadtester/apenv/site-packages/autobahn-0.13.0-py2.7.egg/autobahn/twisted/websocket.py", line 172, in _onClose
    self.onClose(wasClean, code, reason)
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 39, in onClose
    reason=reason))
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 307, in handle
    self._send_exception()
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 113, in _send_exception
    self._run_safely(throw)
--- <exception caught here> ---
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 117, in _run_safely
    self._run_command(func())
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 112, in throw
    self._scenario[-1].throw(*sys.exc_info())
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 305, in handle
    self._raise_unexpected_event(data)
  File "/Users/rpappalardo/git/ap-loadtester/aplt/client.py", line 282, in _raise_unexpected_event
    self._last_command, data))
exceptions.Exception: Unexpected event. Last Command: timer_end; Data: {'messageType': 'disconnect', 'was_clean': False, 'code': 1006, 'reason': u'connection was closed uncleanly (peer dropped the TCP connection without previous WebSocket closing handshake)'}

INFO:twisted:Stopping factory <autobahn.twisted.websocket.WebSocketClientFactory object at 0x0000000107bf1b08>
INFO:twisted:Stopping factory <twisted.web.client._HTTP11ClientFactory instance at 0x00000001062fb0c0>
INFO:twisted:Stopping factory <twisted.web.client._HTTP11ClientFactory instance at 0x00000001062f84a0>
INFO:twisted:Main loop terminated.

Sending notification with bad token causes TypeError: 'NoneType' object is not subscriptable

Replacing a valid token with a random one (of same length) causes Unhandled error

File "/Users/rpappalardo/git/ap-loadtester/aplt/scenarios.py", line 205, in notification_forever_invalid_tokens
yield ack(channel_id=notif["channelID"], version=notif["version"])
exceptions.TypeError: 'NoneType' object is not subscriptable

CRITICAL:twisted:Unhandled Error

NOTE: 
using ap-loadtester scenario notification_forever_invalid_tokens

aplt_scenario throws Twisted error

aplt_scenario wss://autopush.stage.mozaws.net/ aplt.scenarios:basic

see:
https://pastebin.mozilla.org/8864659

note:
aplt_testplan is OK.

Options for mass concurrency

Add the ability to run various counts of multiple scenarios at once for full load-testing.

Add JWT token option for notifications

An option should be available on notification sends to include a JWT token.

Prefer TwistedStatsDClient.create vs its constructor (aplt)

It discourages use of its own constructor:

https://github.com/sidnei/txstatsd/blob/a221e89/txstatsd/protocol.py#L118

When a non ip address hostname's specified, only its create method correctly sets up its host resolver. That's required to run before it actually begins flushing out the metrics.

Running locally, I wasn't getting any metrics sent for a hostname of 'localhost', whereas 127.0.0.1 avoided the issue.

Allow metrics to go to Datadog

Per #24, it'd be useful if running regularly on AWS Lambda, that the metrics could be emitted as CloudWatch data.

Port remaining scenarios over from haskell tester

Add checks to each scenario to ensure push server returns proper HTTP status codes

As-is, ap-loadtester can be used as a blunt-instrument to 'hammer' autopush in a variety of scenarios at scale. If the server falls over or throws a traceback, that's the closest thing to a "fail" that we can currently measure. The closest thing to a test "pass" is the absence of a traceback.

Ideally, (for API testing), we'd walk through the various scenarios and ensure that autopush returns us the HTTP status code(s) we expect with each step. Currently the only way this information could be accessed would be tracing a given test all the way back through Kibana.

Wait doesn't work

The wait command does not work properly, and halts after the second call.

Create websocket client with basic interaction

Create a websocket client in Python with a pypy setup for efficiency that can connect to the autopush websocket system and send webpush style notifications through.

This client should have a fairly easy way to extend/alter the types of interactions it has in a similar manner to how the Haskell version did.

loadtest multi-scenario throw intermittent Fatal RPython error: AssertionError

Seems more likely to happen when there is a larger number of instances spawned.
This happened for me on OSX (haven't yet tried it on Linux)

Example:

$ aplt_testplan wss://autopush.stage.mozaws.net/ "aplt.scenarios:loadtest,20,1,0"

LOG

.net/wpush/v1/gAAAAABXkBG2J5xr9Zi_92cRyigGJQXT7OhKS_TD4f2Wc-LRoqhGBBgzzI5zU_0o0Z9pTERn_oiGU0FZrfoU-33Gt3DMXXRtknHeUQ8x06TqoRZcUzQ_nKD9XIGVLJRo707lX9lfC8D4'}
INFO:twisted:Running command:  Wait(time=2)
INFO:twisted:Running command:  Counter(name='notification.throughput.bytes', count=3111)
INFO:twisted:Running command:  Counter(name='notification.sent', count=1)
INFO:twisted:Running command:  Wait(time=2)
INFO:twisted:Running command:  Counter(name='notification.throughput.bytes', count=2922)
INFO:twisted:Running command:  Counter(name='notification.sent', count=1)
INFO:twisted:Running command:  Wait(time=2)
RPython traceback:
  File "rpython_jit_metainterp_7.c", line 49173, in MIFrame_run_one_step
  File "rpython_jit_metainterp_9.c", line 61354, in handler_residual_call_ir_i_1
  File "rpython_jit_metainterp_11.c", line 66643, in MIFrame_do_residual_call
  File "rpython_jit_metainterp_13.c", line 10854, in MetaInterp_vrefs_after_residual_call
  File "rpython_jit_metainterp_14.c", line 47038, in MetaInterp_stop_tracking_virtualref
Fatal RPython error: AssertionError
Abort trap: 6
(apenv) ~/git/ap-loadtester $

Pass TTL to the send notification call

The TTL is not passed to the notification send.

Update README with notes on installing cryptography and using pypy2-v5.3.1

Even though we don't officially support OSX, it does work (for now) and it's handy to have installation notes for Mac.

Change note on compile flags to read as follows:

ARCHFLAGS="-arch x86_64" LDFLAGS="-L/usr/local/opt/openssl/lib" \
   CFLAGS="-I/usr/local/opt/openssl/include" pip install cryptography

see:
pyca/cryptography#2692

fix LogBeginner usage to avoid its log event buffering

aplt suffers from the same memory leak recently discovered and fixed in autopush:

mozilla-services/autopush#786

Install error on txaio version number

Seems there's a conflict between txaio version number required by autobahn and that required by aplt. When installing ap-loadtester on Ubuntu (exactly as on README - PyPy 4.0.1), I get:

Installed /home/ubuntu/git/ap-loadtester/apenv/site-packages/txaio-2.2.1-py2.7.egg
error: txaio 2.2.1 is installed but txaio>=2.2.2 is required by set(['autobahn'])

However, when I: $ pip install txaio=2.2.2, I get:
DistributionNotFound: The 'txaio==2.2.1' distribution was not found and is required by aplt

Add test result summary

ap-loadtester provides a means for creating / generating various kinds of load. You can then hit the server with a run-once execution or a forever execution, but there is currently no means to track whether a single scenario run achieved it's desired effect (PASS) or not (FAIL).

Once issue #73 has landed, it would be really useful to have a result summary output similar to molotov:

**** Molotov v0.6.1. Happy breaking! ****
[82217] Preparing 1 workers...OK
SUCCESSES: 34 | FAILURES: 0
*** Bye ***

This would allow ap-loadtester to also serve as a more effective API / integration tester (vs. using it with loads-broker for scalability testing).

Allow a scenario to spin up scenarios

It's hard to build a suite of scenarios to run at once, because each one has to be supplied individually to the testplan command.

A new scenario command run_scenario could let a scenario act as a scenario suite and spin up multiple other ones.

Add option to hardcode endpoint

aplt needs an option to hardcode the endpoint server hostname to ensure we're specifically testing the partner endpoint.

We currently pass it only the websocket server hostname. It uses the endpoint hostname provided from the register call's response, which likely isn't a partner's endpoint.

Verify arguments for test plan

Test plan arguments can be tuples with optional scenario args, the args should be verified and checked against the scenario function at execution.

Add test for spawning function doing multiple yields back-to-back

We're currently able to spawn a scenario from a spawning function thusly:
def _test_spawn():
yield spawn("aplt.scenarios:basic, 1, 1, 0")

However, when we try to spawn multiple scenarios back-to-back, we get a traceroute:
https://pastebin.mozilla.org/8869561

def _test_spawn_multiple():
yield spawn("aplt.scenarios:basic, 1, 1, 0")
yield spawn("aplt.scenarios:register_forever, 1, 1, 0")

We should add a test for scenarios yielding multiple (and fix this!)

Automated docker builds on travis success

When travis verifies that the load-tester builds correctly, a docker image should be built and ready for use.

This will most likely require a docker trusted build based off the repo, though travis has some docker build/upload options now as well.

aplt_scenario logging causes maximum recursion errors

This probably only happens when aplt_scenario is misconfigured and begins logging errors -- logging produces a lot of nonsense and finally a recursion error:

2016-08-30 23:37:10+0000 [-] Log opened.
2016-08-30 23:37:11+0000 [-] Running command: Connect()
2016-08-30 23:37:11+0000 [-] INFO:twisted:Running command: Connect()
2016-08-30 23:37:11+0000 [-] ERROR:twisted:INFO:twisted:Running command: Connect()
2016-08-30 23:37:11+0000 [-] ERROR:twisted:ERROR:twisted:INFO:twisted:Running command: Connect()

...

2016-08-30 23:37:13+0000 [-] Unable to format event {'log_failure': <twisted.python.failure.Failure exceptions.RuntimeError: maximum recursion depth exceeded>, 'observer': LegacyLogObserverWrapper(<bound method PythonLoggingObserver.emit of <twisted.python.log.PythonLoggingObserver object at 0x00007f74ed4db440>>), 'log_logger': <Logger 'twisted.logger._observer'>, 'log_level': <LogLevel=critical>, 'log_namespace': 'twisted.logger._observer', 'log_source': None, 'log_format': 'Temporarily disabling observer {observer} due to exception: {log_failure}', 'log_time': 1472600234.370311, 'message': (), 'time': 1472600234.370311, 'system': '-', 'format': '%(log_legacy)s', 'log_legacy': <twisted.logger._stdlib.StringifiableFromEvent object at 0x00000000047bdd00>, 'failure': <twisted.python.failure.Failure exceptions.RuntimeError: maximum recursion depth exceeded>, 'isError': 1}: maximum recursion depth exceeded

Allow specifying TLS cert content via environment variables

Followup to #71:

aplt could also allow specifying endpoint TLS certs by loading the PEM file content from an environment variable value.

This lets loads-broker conveniently pass along certs to attack nodes

upgrade pypy Docker/travis pypy versions

travis's is especically outdated

create client handler for SSL

We'll need this for testing against the new stack w/ SSL

Add new crypto, operational scenarios

1. register_forever
2. notification_forever_stored
3. notification_forever_unsubscribed
4. notification_forever_bad_tokens
5. notification_forever_bad_endpoints

Downgrade pypy version in Dockerfile (only)

The new pypy version is causing an issue with loads-broker so we're going to (temporarily) downgrade it to pypy-5.0.0.

see:
loads/loads-broker#47