mozilla-services/antenna-loadtests

loadtests for antenna server: https://github.com/mozilla/antenna

antenna-loadtests

generic load test based on ailoads: https://github.com/loads/ailoads

Requirements

• Python 3.5

How to run the loadtest?

For dev

make test -e URL_SERVER=https://antenna.dev.mozaws.net/submit

For stage

make test -e URL_SERVER=https://antenna.stage.mozaws.net/submit

How to build the docker image?

make docker-build

How to run the docker image?

make docker-run

How to clean the repository?

make clean

antenna-loadtests's Issues

Create a testcase to make sure throttling is on

Any crash dump > 20M should be rejected by throttling. Let's make sure this is in place. This is less about the server behaving properly as it is about the ops config being in place.

Parameterize verbose flag in Dockerfile

chore: cleanup loads-broker.tpl generation

add test case that saves multiple dumps

All the test cases so far create a single upload_file_minidump dump.

I've seen this combination in real crashes: upload_file_minidump, upload_file_minidump_flash2, upload_file_minidump_browser, upload_file_minidump_flash1

It's probably fine to make all the dumps the same size. So we could take the output of generate_sized_crashes and then split upload_file_minidump into 4 strings for the four dumps.

Create loadtests for POST loadtests - small file, large file

Rename loads.json to loads-broker.json

Add test scenario: 1.5 mb uncompressed

Fix loads.tpl file

There is a typo in the loads.tpl file.

Update CONNECTIONS to TEST_CONNECTIONS

codify scenario weightings for a test run

One of the things we want to do is re-weight the scenarios depending on the load test type and what we're trying to answer. It'd be great for reproducibility and documentation purposes if we could codify the weights used along with the other test parameters in a single file.

One way to do that would be to move the scenario weightings to env vars thus exposing them in a way that lets us codify them in a loads.json file.

Doing this should be straight-forward. It'd involve changing these lines:

@scenario(20)
def test_...():

to something like this:

def weighted_scenario(fun):
    fun_name = fun.__name__  # or whatever it is
    weighting = int(os.environ.get(fun_name, 20))

    return scenario(weighting)(fun)


@weighted_scenario
def test_something(...):
    ...

Are there other ways to do this?

Makefile - if loadtest.env doesn't exist, then create it

Rename loadtest.env to molotov.env

purpose: to clarify that this config is tied to molotov (we'll eventually replace with either molotov.json format or molotov.yaml)

NOTE: we may also want to rename loads.json to loads-broker.json

Move ConnectionHandler class to separate utility module

Parameterize Makefile and Dockerfile out to loadtest.env

NOTE: loadtest.env will eventually be replaced by molotov.json

Update ailoads for molotov

Send crashids to stdout

Update Makefile for new loadtesting URL

the new loadtesting URL is antenna-loadtests.stage.mozaws.net. This will need to be changed in the Makefile (and eventually in molotov.json)

Create method for downloading dump files from S3

Create basic GET __version__ test

Create skeleton test to connect to the Antenna server. We'll use this to iterate on real test cases.

CODE_OF_CONDUCT.md file missing

As of January 1 2019, Mozilla requires that all GitHub projects include this CODE_OF_CONDUCT.md file in the project root. The file has two parts:

1. Required Text - All text under the headings Community Participation Guidelines and How to Report, are required, and should not be altered.
2. Optional Text - The Project Specific Etiquette heading provides a space to speak more specifically about ways people can work effectively and inclusively together. Some examples of those can be found on the Firefox Debugger project, and Common Voice. (The optional part is commented out in the raw template file, and will not be visible until you modify and uncomment that part.)

If you have any questions about this file, or Code of Conduct policies and procedures, please reach out to [email protected].

(Message COC001)

TypeError: 'Exception' object is not iterable

When I try to run master tip against Antenna running on my local machine, I get this:

(M=bc78b) ~/mozilla/socorro-zero/antenna-loadtests> make test -e URL_SERVER=http://localhost:8000/submit
env LDFLAGS="-L/lib" \
    CFLAGS="-I/include" \
    /home/willkg/mozilla/socorro-zero/antenna-loadtests/venv/bin/pip3 install cryptography
Requirement already satisfied: cryptography in ./venv/lib/python3.5/site-packages
Requirement already satisfied: idna>=2.0 in ./venv/lib/python3.5/site-packages (from cryptography)
Requirement already satisfied: pyasn1>=0.1.8 in ./venv/lib/python3.5/site-packages (from cryptography)
Requirement already satisfied: setuptools>=11.3 in ./venv/lib/python3.5/site-packages (from cryptography)
Requirement already satisfied: cffi>=1.4.1 in ./venv/lib/python3.5/site-packages (from cryptography)
Requirement already satisfied: six>=1.4.1 in ./venv/lib/python3.5/site-packages (from cryptography)
Requirement already satisfied: pycparser in ./venv/lib/python3.5/site-packages (from cffi>=1.4.1->cryptography)
/home/willkg/mozilla/socorro-zero/antenna-loadtests/venv/bin/pip3 install -r requirements.txt
Collecting https://github.com/tarekziade/ailoads/archive/master.zip (from -r requirements.txt (line 4))
  Downloading https://github.com/tarekziade/ailoads/archive/master.zip
     / 40kB 1.1MB/s
Requirement already satisfied: six in ./venv/lib/python3.5/site-packages (from -r requirements.txt (line 1))
Requirement already satisfied: requests in ./venv/lib/python3.5/site-packages (from -r requirements.txt (line 2))
Requirement already satisfied: flake8 in ./venv/lib/python3.5/site-packages (from -r requirements.txt (line 3))
Collecting aiohttp (from molotov==0.4->-r requirements.txt (line 4))
  Downloading aiohttp-1.2.0-cp35-cp35m-manylinux1_x86_64.whl (155kB)
    100% |████████████████████████████████| 163kB 1.6MB/s 
Requirement already satisfied: statsd in ./venv/lib/python3.5/site-packages (from molotov==0.4->-r requirements.txt (line 4))
Collecting urwid (from molotov==0.4->-r requirements.txt (line 4))
  Downloading urwid-1.3.1.tar.gz (588kB)
    100% |████████████████████████████████| 593kB 1.9MB/s 
Collecting humanize (from molotov==0.4->-r requirements.txt (line 4))
  Downloading humanize-0.5.1.tar.gz
Collecting redis (from molotov==0.4->-r requirements.txt (line 4))
  Downloading redis-2.10.5-py2.py3-none-any.whl (60kB)
    100% |████████████████████████████████| 61kB 2.9MB/s 
Requirement already satisfied: pycodestyle<2.3.0,>=2.0.0 in ./venv/lib/python3.5/site-packages (from flake8->-r requirements.txt (line 3))
Requirement already satisfied: pyflakes!=1.2.0,!=1.2.1,!=1.2.2,<1.4.0,>=0.8.1 in ./venv/lib/python3.5/site-packages (from flake8->-r requirements.txt (line 3))
Requirement already satisfied: mccabe<0.6.0,>=0.5.0 in ./venv/lib/python3.5/site-packages (from flake8->-r requirements.txt (line 3))
Collecting chardet (from aiohttp->molotov==0.4->-r requirements.txt (line 4))
  Downloading chardet-2.3.0.tar.gz (164kB)
    100% |████████████████████████████████| 174kB 6.7MB/s 
Collecting multidict>=2.1.4 (from aiohttp->molotov==0.4->-r requirements.txt (line 4))
  Downloading multidict-2.1.4-cp35-cp35m-manylinux1_x86_64.whl (357kB)
    100% |████████████████████████████████| 358kB 2.6MB/s 
Collecting async-timeout>=1.1.0 (from aiohttp->molotov==0.4->-r requirements.txt (line 4))
  Downloading async_timeout-1.1.0-py3-none-any.whl
Collecting yarl>=0.8.1 (from aiohttp->molotov==0.4->-r requirements.txt (line 4))
  Downloading yarl-0.8.1-cp35-cp35m-manylinux1_x86_64.whl (137kB)
    100% |████████████████████████████████| 143kB 2.2MB/s 
Building wheels for collected packages: urwid, humanize, chardet
  Running setup.py bdist_wheel for urwid ... done
  Stored in directory: /home/willkg/.cache/pip/wheels/e7/bb/9d/d7f7fcc9b3aeae20cf4685667635002415629e6395ff2dda7d
  Running setup.py bdist_wheel for humanize ... done
  Stored in directory: /home/willkg/.cache/pip/wheels/d4/80/38/cfbfd95752f71f3812505b948b43383ddc99eedf835fc13b09
  Running setup.py bdist_wheel for chardet ... done
  Stored in directory: /home/willkg/.cache/pip/wheels/28/8c/bf/a69199bd4901d84e13362f95a9ea7bc9a691fed2d655a90bc4
Successfully built urwid humanize chardet
Installing collected packages: chardet, multidict, async-timeout, yarl, aiohttp, urwid, humanize, redis, molotov
  Running setup.py install for molotov ... done
Successfully installed aiohttp-1.2.0 async-timeout-1.1.0 chardet-2.3.0 humanize-0.5.1 molotov-0.4 multidict-2.1.4 redis-2.10.5 urwid-1.3.1 yarl-0.8.1
bash -c "URL_SERVER=http://localhost:8000/submit /home/willkg/mozilla/socorro-zero/antenna-loadtests/venv/bin/ailoads -v -d 30"
[th]Traceback (most recent call last):
  File "/home/willkg/mozilla/socorro-zero/antenna-loadtests/venv/bin/ailoads", line 11, in <module>
    load_entry_point('ailoads==0.1', 'console_scripts', 'ailoads')()
  File "/home/willkg/mozilla/socorro-zero/antenna-loadtests/venv/lib/python3.5/site-packages/ailoads/run.py", line 71, in main
    return run(args)
  File "/home/willkg/mozilla/socorro-zero/antenna-loadtests/venv/lib/python3.5/site-packages/ailoads/run.py", line 85, in run
    for ok, failed in res:
TypeError: 'Exception' object is not iterable
Makefile:56: recipe for target 'test' failed
make: *** [test] Error 1
2 (M=bc78b) ~/mozilla/socorro-zero/antenna-loadtests> 

Am I doing something wrong? If not, any idea what's going on.