tarekziade/molotov

Load Testing Tool

molotov

Simple Python 3.8+ tool to write load tests.

Based on asyncio, Built with aiohttp 3.x

Full Documentation

molotov's Issues

make duration infinite by default

right now it's 10 seconds. let's make it 9999 seconds

move to github/loads org?

Since this is intended to work w/ Loadsv2, would it be better housed there?
I was thinking it would be good to create a doc for using ailoads w/ loads-broker, but it would be easier to discover if they were all together.

Loading environment variables from a config file.

We should be able to tell ailoads to read some json or ini file where it could read and set environment variables.

add -vv

-v : just the exceptions
-vv : the full request and responses

replace redis by simple IPC

something low level to keep track of each process results

Fix molotov output (inconsistent line breaks)

**** Molotov v0.6. Happy breaking! ****
Forking 5 processes
[8] Preparing 4 workers...OK
[10] Preparing 4 workers...[9] Preparing 4 workers...OK
[12] Preparing 4 workers...OK
OK
[11] Preparing 4 workers...OK

add metrics reporting to molotov

@tarek We frequently have intermittent failures with loadtests that seem to be due to AWS network hiccups. This happens in particular w/ long-running tests we've done w/ ap-loadtester.
@pjenvey suggested we might start collecting metrics from within molotov to log things like test start/stop times, etc. to determine when tests fail (or connections dropped) due to latency.

With loads, we've talked about setting up InfluxDB for this. Not sure the best way ultimately for us to gather (and monitor) this data.

re-name --workers to --agents??

@tarekziade, question for you....

In ailoads, we had the concept of a "user." From a testing perspective, this makes some sense because (via a loadtest), we're trying to pretend that we are 1 user doing some activity and multiplying that by some scale value: --users n. But it's not entirely intuitive what a "user" is and this could easily get confused with the --name parameter in loads-broker where we specify the username of the person running the tests.

In molotov, we've renamed that to "worker." This is better in one sense as it's more like a thread, but could easily be confused with the "process" parameter which is a thread of the async activity. A process is essentially a thread and a thread is sometimes also referred to as a "worker", so sometimes I find that a bit confusing.

The loadtesting tool: beeswithmachineguns refers to these "users"/"workers" as "concurrent connections":
https://github.com/igkins/beeswithmachineguns/blob/master/beeswithmachineguns/main.py#L100

We could rename "workers" to "connections", but in ap-loadtester, a connection is used to refer to an open websocket connection, whereas with molotov our "workers" are essentially agents and the connections may be established and terminated multiple times over the course of a test.
Should we call it an "agent" instead?

---

NOTE:

When doing a loads-broker test, these params would be added to the loads-broker.json so that your scale would be determined by:

molotov.json
(1) # processes
(2) # agents

loads-broker.json
(3) # attack nodes

TOTAL # AGENTS = # processes / node X # agents / node X # attack nodes

create synchronous version of HTTP calls

https://github.com/mozilla-services/screenshots-loadtests/blob/master/utils.py#L113

so people don't have to do all that

proper doc

• Create a proper doc from the README in a Sphinx doc
• Include Loads' Logo
• Build it with Travis-CI
• push it to RTD

add fixtures for session

to improve smwogger integration, we want to be able to create the smowgger.API() class when the aiohttp session is created, so it can be shared.

• setup_session() called once the session object is created. gets the session object
• teardown_session() called just before the session is closed

Add function to generate a dummy FxA account and FXA_BEARER_TOKEN

Write a tutorial

content:

• how to use Molostart
• how to write a test
• how to run in with several workers, then several processes
• how to add statsd metrics
• how to use Moloslave
• how to deploy it for loads-broker, using Docker

create a generic moloslave-based image

using python3.5-slim

threads

we have processes, each one with its own async loop, that can generate over 3000 req/s - investigate if we want to add a threads layer, so each process runs several threads to boost the number of parallel coroutines we can have

Bad generated code on `@global_setup`

I ran the molostart command line tool and started editing loadtest.py. Then when I run it I got this:

▶ molotov --max-runs 1 -cx loadtest.py
**** Molotov v1.0. Happy breaking! ****
Traceback (most recent call last):
  File "/Users/peterbe/virtualenvs/tecken-loadtests/bin/molotov", line 11, in <module>
    sys.exit(main())
  File "/Users/peterbe/virtualenvs/tecken-loadtests/lib/python3.5/site-packages/molotov/run.py", line 98, in main
    return run(args)
  File "/Users/peterbe/virtualenvs/tecken-loadtests/lib/python3.5/site-packages/molotov/run.py", line 108, in run
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 662, in exec_module
  File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
  File "loadtest.py", line 11, in <module>
    @global_setup
TypeError: global_setup() takes 0 positional arguments but 1 was given

The code generated looked like this:

...MY STUFF...

@global_setup
def test_starts(args):
    """ This functions is called before anything starts.

    Notice that it's not a coroutine.
    """

    ...MY STUFF...

Add more debugging info

Not sure how to diagnose issue here. Is there a way to provide more info about what's going on?

[th][th][th][th][th][th][th][th][th][th]Expecting value: line 1 column 1 (char 0)
Expecting value: line 1 column 1 (char 0)
Expecting value: line 1 column 1 (char 0)
Expecting value: line 1 column 1 (char 0)
Expecting value: line 1 column 1 (char 0)
Expecting value: line 1 column 1 (char 0)
...
Expecting value: line 1 column 1 (char 0)
Expecting value: line 1 column 1 (char 0)
[-th]
0 OK, 10 Failed

Running molotov in --verbose mode generates tracebacks on encoding, and drops two otherwise-passing requests

Steps to Reproduce:

1. Grab https://github.com/stephendonner/stubattribution-loadtests/blob/f4e581c75b719d2cbaf4ed10ceb68609ee169d3e/loadtest.py as-is
2. Using the same HMAC_KEY (I'll get it to you, @tarekziade), run:

molotov -cxv -d 1 loadtest.py

Expected Results:

Aside from the expected, extra output from passing in --verbose, it should look something like:

**** Molotov v0.2. Happy breaking! ****
[27699] Preparing 1 workers...OK
..
2 OK, 0 Failed

Actual Results:

See https://gist.github.com/stephendonner/f3a1ebe7e47a046f17d7c929a2876528

In addition to the traceback, it seems unusual that we get:

0 OK, 0 Failed

Fail if all scenario weights are == 0

Sometimes we'll set at least one scenario weight to zero which is equivalent to test.skip. However, we should never have a test module in which all scenario weights are 0 and should probably notify the user with a meaningful message when they do so to avoid something like:
TypeError: 'Exception' object is not iterable
mozilla-services/antenna-loadtests#10

make molotov pypy3 compatible

• find out if everything works as expected with urwid
• add a python versions badge 3.5, 3.6, pypy3.5
• release !

raise an error if <3.5

Let's refuse to run if it's Python <3.5 with an explicit message

prettify the curse display

global header/footer, more colors & metrics

display a curse based output per process

Each process will have its feedback

We want to split the screen and display the right log per process

add a sleep per run

like

https://github.com/mozilla-services/ap-loadtester/blob/master/aplt/scenarios.py#L80

bubble up exceptions in LiveResult

So it works in multiple process mode

Missing quickstart/Makefile file for molotov installed via pip

$ molostart
**** Molotov Quickstart ****

Answer to a few questions to get started...
> Target directory [.]: 
> Create Makefile [y]: 
Generating Molotov test...
…copying 'Makefile' in '.'
Traceback (most recent call last):
  File "/home/arthur/.virtualenvs/molotov/bin/molostart", line 11, in <module>
    sys.exit(main())
  File "/home/arthur/.virtualenvs/molotov/local/lib/python3.5/site-packages/molotov/quickstart/__init__.py", line 83, in main
    _copy_file('Makefile', target_dir)
  File "/home/arthur/.virtualenvs/molotov/local/lib/python3.5/site-packages/molotov/quickstart/__init__.py", line 60, in _copy_file
    shutil.copyfile(os.path.join(_HERE, name), target)
  File "/home/arthur/.virtualenvs/molotov/lib/python3.5/shutil.py", line 114, in copyfile
    with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: '/home/arthur/.virtualenvs/molotov/local/lib/python3.5/site-packages/molotov/quickstart/Makefile'

increase the pool size

see http://aiohttp.readthedocs.io/en/stable/client.html#limiting-connection-pool-size

add template files to quickstart

boilerplate Makefile and Dockerfile.

NOTE: Eventually we may also want to combine molotov.json and loads-broker.tpl into one config file as well.

add @global_teardown and @teardown

Called when a worker is done, and when everything is done. Useful for doing some cleanups.

add a boom2 CLI

per https://github.com/tarekziade/boom/issues/64

Let's add a boom2 command to generate load on a single endpoint, then deprecate boom1

proper SIGTERM/SIGINT handling

• catch SIGTERM/SIGINT in the main process, gracefully stop all subprocesses
• remove all KeyboardInterrupt exceptions
• when a subprocess dies, we want to finish the rest

verbose mode consumes the body to display it

I think that's because stream is already consumed. Will investigate.

switch to aiohttp.client

So we can be fully async

add the ability to configure a test suite

from the CLI a sorted list of tests we want to run instead of scenario weights

add a verbose output

with request/response details

Molotov reporting inconsistent test results

I executed a test run against a service that returned a HTTP 503 and the test results reported back were inconsistent.

**** Molotov v0.4. Happy breaking! ****
[6399] Preparing 1 workers...OK
SUCCESSES: 0 | FAILURES: 0
0 OK, 1 Failed

add 3.6 to tox/travis

and see if we have any trouble

single hit / scenario mode

we should add a way to run a single scenario once, for testing purpose.

I guess by adding a 'name' option to scenario(name='something')

and then calling it with --scenario-name and --hits=1

printing requests assumes payload is text

https://github.com/loads/molotov/blob/5ab8c822fcd3c515b4614d624a4d67c9c7d5d40d/molotov/session.py#L85

That assumes that if the request body is bytes, then it must be utf-8. Antenna load tests send gzip compressed payloads, so this isn't true.

Maybe we can check the HTTP headers to see what the payload is before figuring out how to convert and print it to stdout? Maybe check to see how much data there is before printing it? If there's more than 1000 bytes, probably makes sense to truncate it.

add DNS resolver

Add a DNS resolver so the endpoint is called with its IP and a Host header
Option to deactivate it if needed

add a @global_setup

a setup that's called once, before processes are forked.

import json
from molotov import scenario, setup, global_setup

_TOKEN = None

@global_setup()
def global_init(args):
    # do something that needs to be shared across all processes and workers
    global _TOKEN
    _TOKEN = _build_token()

@setup()
async def init_test(args):
    # worker-specific, session-specific init
    headers = {'Authorization': _TOKEN}
    return {'headers': headers}

@scenario(40)
async def scenario_one(session):
    # XXX

quickstart script that generates a load test layout

Would include things like:

• Makefile (should be fully parameterized pulling all values from molotov.json or loadtest.env)
• Dockerfile (should be fully parameterized pulling all values from molotov.json or loadtest.env)
• molotov.json (with sample values)
• loads.tpl (with sample values)

NOTE:
currently we use loadtest.env to specify all key,value params (for Makefile and Dockerfile).
We should replace this with either a molotov.json or molotov.yaml

get rid of globals

refactor fmwk.py - make some classes and get rid of _STOP

rename to molotov

globals

when writing a global setup, we might want to share global objects e.g.

https://github.com/loads/mozlotov#fxa

maybe we could have a placeholder for this to avoid using an ad-hoc global mechanism.
e.g. set_global(name, object), get_global(name)

Include local/client failures in test-run output?

I pushed molotov a little too hard from my single laptop instance with:

(stubattribution-loadtests) sdonner-17447:stubattribution-loadtests sdonner$ molotov -cxv -p 10 -d 60 -w 130 loadtest.py

and got the following:

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
GET https://stubattribution-default-cdn.stage.mozaws.net/builds/test-stub/en-US/win/d6e71d42294ee73c7aaa42268cd3816e50b6ac8bc8258b413fa0d5135d5cb560/test-stub.exe
Accept-Encoding: gzip, deflate
Accept: */*
Host: stubattribution-default-cdn.stage.mozaws.net
User-Agent: Python/3.5 aiohttp/1.2.0
Content-Length: 0>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
GET https://stubattribution-default-cdn.stage.mozaws.net/builds/test-stub/en-US/win/93cd84e7b22c9a676d06ff24d5853ea449ba7f16822a2cbe10de7cef65d2ebd6/test-stub.exe
Accept-Encoding: gzip, deflate
Accept: */*
Host: stubattribution-default-cdn.stage.mozaws.net
User-Agent: Python/3.5 aiohttp/1.2.0
Content-Length: 0>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
GET https://stubattribution-default-cdn.stage.mozaws.net/builds/test-stub/en-US/win/748e00adf48dae757cad79142ee9553ce5fef9f3d212d7f282ce16a4dc2583d5/test-stub.exe
Accept-Encoding: gzip, deflate
Accept: */*
Host: stubattribution-default-cdn.stage.mozaws.net
User-Agent: Python/3.5 aiohttp/1.2.0
Content-Length: 0ClientOSError(8, 'Cannot connect to host stubattribution-default-cdn.stage.mozaws.net:443 ssl:True [nodename nor servname provided, or not known]')  File "/Users/sdonner/.pyenv/versions/3.5.0/envs/stubattribution-loadtests/lib/python3.5/site-packages/molotov/fmwk.py", line 81, in step
    await func(session, *args_, **kw)
  File "loadtest.py", line 37, in scenario_one
    async with session.get(_SERVER, params=params) as resp:
  File "/Users/sdonner/.pyenv/versions/3.5.0/envs/stubattribution-loadtests/lib/python3.5/site-packages/aiohttp/client.py", line 540, in __aenter__
    self._resp = yield from self._coro
  File "/Users/sdonner/.pyenv/versions/3.5.0/envs/stubattribution-loadtests/lib/python3.5/site-packages/molotov/session.py", line 40, in _request
    resp = await super(LoggedClientSession, self)._request(*args, **kw)
  File "/Users/sdonner/.pyenv/versions/3.5.0/envs/stubattribution-loadtests/lib/python3.5/site-packages/aiohttp/client.py", line 176, in _request
    conn = yield from self._connector.connect(req)
  File "/Users/sdonner/.pyenv/versions/3.5.0/envs/stubattribution-loadtests/lib/python3.5/site-packages/aiohttp/connector.py", line 318, in connect
    .format(key, exc.strerror)) from exc

1277 OK, 0 Failed

It's a little strange that with client-connection issues (DNS resolver, or sockets failing to connect, issue a request, etc.) that the test-run output says "0 Failed"

unfork cystatsd in travis.yml

if that gets merged scivey/cystatsd#4

add a number of HTTP hits

When Molotov leaves, let's display the number of requests that where done against a service. Just a counter per host and per status

expose the Session options when initializing

The user should have a way to pass options to the Session constructor when it's created (auth headers etc.)

I guess one way to do it is to provide an setup decorator that can return a mapping with the session options

adding a default weight

set a default value of 1 when the weight is not set