mozilla-services/stubattribution-loadtests

stubattribution-loadtests

generic load test based on molotov: https://github.com/loads/molotov

Requirements

• Python 3.5+

How to run the loadtest?

For STAGE

molotov -c -d 1 loadtest.py

How to build the docker image?

sudo make docker-build

How to run the docker image?

make docker-run

How to clean the repository?

make clean

stubattribution-loadtests's Issues

Load-testing approach, first test/attempt

1. Use https://github.com/loads/molotov
2. Write a simple test against (Bouncer? The stub attribution server?) using the Python 3 asyncio library, like so: https://github.com/rpappalax/shavar-loadtests/blob/f550099cab139cc0a760dcec920c802f675e9951/loadtest.py
3. For #2, ensure:

• separate HMAC_KEY for this test (w/help from @oremj and @pmac)
• safely abstracted (i.e. no keys in test code)

4. Ensure the following:

• auth works
• we can get and follow the response (HTTP 302?)
• verify that we get a success/200 OK response from (stub attribution server?)

[traceback] ImportError: cannot import name 'get_scenarios'

When running:

molotov -cx -d 1 loadtest.py

using molotov 0.2, I now see (https://gist.github.com/stephendonner/10f62588df81c315ebc589b4d90dfba0)

Successfully installed molotov-0.2
You are using pip version 7.1.2, however version 9.0.1 is available.
(stubattribution-loadtests) sdonner-17447:stubattribution-loadtests sdonner$ molotov -cx -d 1 loadtest
loadtest.py          loadtest_output.txt  
(stubattribution-loadtests) sdonner-17447:stubattribution-loadtests sdonner$ molotov -cx -d 1 loadtest.py
Traceback (most recent call last):
  File "/Users/sdonner/.pyenv/versions/stubattribution-loadtests/bin/molotov", line 9, in <module>
    load_entry_point('molotov==0.2', 'console_scripts', 'molotov')()
  File "/Users/sdonner/.pyenv/versions/3.5.0/envs/stubattribution-loadtests/lib/python3.5/site-packages/pkg_resources/__init__.py", line 558, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/Users/sdonner/.pyenv/versions/3.5.0/envs/stubattribution-loadtests/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2682, in load_entry_point
    return ep.load()
  File "/Users/sdonner/.pyenv/versions/3.5.0/envs/stubattribution-loadtests/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2355, in load
    return self.resolve()
  File "/Users/sdonner/.pyenv/versions/3.5.0/envs/stubattribution-loadtests/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2361, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/Users/sdonner/.pyenv/versions/3.5.0/envs/stubattribution-loadtests/lib/python3.5/site-packages/molotov/run.py", line 8, in <module>
    from molotov.fmwk import runner, get_scenarios
ImportError: cannot import name 'get_scenarios'

CODE_OF_CONDUCT.md file missing

As of January 1 2019, Mozilla requires that all GitHub projects include this CODE_OF_CONDUCT.md file in the project root. The file has two parts:

1. Required Text - All text under the headings Community Participation Guidelines and How to Report, are required, and should not be altered.
2. Optional Text - The Project Specific Etiquette heading provides a space to speak more specifically about ways people can work effectively and inclusively together. Some examples of those can be found on the Firefox Debugger project, and Common Voice. (The optional part is commented out in the raw template file, and will not be visible until you modify and uncomment that part.)

If you have any questions about this file, or Code of Conduct policies and procedures, please reach out to [email protected].

(Message COC001)

HMAC doesn't validate with current loadtest.py script + HMAC_KEY against https://stubattribution-default.stage.mozaws.net/

Running https://github.com/stephendonner/stubattribution-loadtests/blob/97d033071cf08b73d5af84ed49aef7d530c9fa00/loadtest.py against https://stubattribution-default.stage.mozaws.net/ as follows:

molotov -cx -d 1 loadtest.py

yields:

**** Molotov v0.2. Happy breaking! ****
[30818] Preparing 1 workers...OK
..
2 OK, 0 Failed

However, we see the following HMAC validation exception thrown, in Sentry:

https://sentry.prod.mozaws.net/operations/stub_attribution-stage/issues/379693/

could not validate attribution_code: source%3D(not+set)%26medium%3Dbac72540-45d6-4698-a0b7-63a00adcbea3%26campaign%3D(not+set)%26content%3D(not+set)%26timestamp%3D1485414893: HMAC would not validate. given: 87789e4c3815345752a686d323c476f9350f0d16f919e3ee7b2858c19b4ae07a expected: ca365c22665a271e1bf50eb9350d9cac15c499fcf8853e73e6e841f80c35ad4b

Vary input values for utm_source, utm_medium, utm_campaign, product, and lang params

With the utm_source whitelist in https://github.com/mozilla-services/stubattribution/blob/2c9f8a9d93ab544599209bf847980beb6c187984/attributioncode/sourcewhitelist.go, and the utm_medium and utm_campaign values in https://docs.google.com/spreadsheets/d/1U-0JHpc3INJnBwTFkdrPqpk7yqvpEn16DJtp6d6TRiY/edit#gid=1275474795 (some of them - we don't need to go all out), we should:

1. probably read them in from a separate file(s)
2. randomize the combinations of utm_source, utm_medium, and utm_campaign (their values)
3. looks like we'll also have to vary on product= and lang= param values (https://github.com/stephendonner/stubattribution-loadtests/issues/1#issuecomment-275494930)

@tarekziade or @pmac either one of you be up for helping with this? While I'm going to take a look at it, my Python + coding skills in general are weak/rusty - thanks!

Add open source software license

This Mozilla repository has been identified as lacking a license. Consistent with Mozilla's Licensing Policy an open source license should be applied to the code in this repository.

Please add an appropriate LICENSE.md file to the root directory of the project. In general, Mozilla's licensing policies are as follows:

• Client-side products created by Mozilla employees or contributors should use the Mozilla Public License, Version 2.0 (MPL).

• Server-side products or utilities that support Mozilla products may use either the MPL or the Apache License 2.0 (Apache 2.0).

In special cases, another license might be appropriate. If the repository is a fork of another repository it must apply the license of the original. Similarly, another license might be appropriate to match that of a broader project (for example Rust crates that Firefox depends on are published under an Apache 2.0 / MIT dual license, as that is the dual license used by the Rust programming language and projects).

Please ensure that any license added to the LICENSE.md file matches other licensing information in the repository (for example, it should match any license indicated in a setup.py or package.json file).

Mozilla staff can access more information in our Software Licensing Runbook – search for “Licensing Runbook” in Confluence to find it.

If you have any questions you can contact Daniel Nazer who can be reached at dnazer on Mozilla email or Slack.

OPENLIC-2023-01

Figure out which datapoints (and how to measure them) we care about for load testing

Just keeping a note here to help organize my notes/thoughts/questions about which metrics/datapoints we care about (and how/can we measure them) during load testing:

• requests per second (RPS)
• time to first byte (TTFB)
• CPU(s) usage (running 'top' command vs...?)
• memory usage on the server(s)
• load average (ancient article: http://www.linuxjournal.com/article/9001)
• disk I/O activity