grafana/k6

A modern load testing tool, using Go and JavaScript - https://k6.io

k6's Issues

HTTP options

Based on experience with current load generator scripting we know users have a need to customize the HTTP client behavior, this card details what options we need.

HTTP Options (set globally)

• TCP max connections per host
• TCP max connections per VU
• User agent string

HTTP options (local, request functions)

• ...

Imported from https://trello.com/c/5ISLT75I/115-http-options

Acceptable failure rates for checks

Imported from https://trello.com/c/nUovzXLz/102-acceptable-failure-rates-for-checks

Simplified one-check syntax

check("name", () => /* ... */);

Imported from https://trello.com/c/jW4KQLOw/109-simplified-one-check-syntax

Thresholds

export let options = {
  thresholds: {
    http_req_duration: ["avg<=100"], // 100ms
  }
}

Imported from https://trello.com/c/rD9GoW9R/103-thresholds

Full k6 control from API

After playing around with remote controlling k6 from another process I've discovered some things that's missing or would ease the integration considerably.

• I'd like to be able to start k6 in some kind of "slave mode" where it only spins up the API. From there on I want to interact with k6 via the API only. I plan to start k6 with standard in, out and err bound to /dev/null and detached from any shell.
• I'd like to have a "self destruct" API endpoint (v1/quit?) to kill speedboat when it's time to stop the test and tear down output plugins and similar.
• To even get the test going, I need a v1/run endpoint that would work as we discussed earlier. This would be the way to send the javascript scenario + config in a POST-request to a process started in a way as described above.

These are thoughts based on last weeks testing/discovery coding and are not finalized suggestions. Hoping for a good discussion.

Imported from https://trello.com/c/FGFfIwqG/88-full-k6-control-from-api

Output plugin: kcs metrics streaming

The interface for streaming metrics to the cloud services seems to match the intended use of output plugins. It might not be a 100% match, which is why we need to discuss it further (here). Some points to clarify:

• Authentication. I'm thinking the plugin can look for credentials in a file or in environment variables, like tools interacting with AWS do.

• How do we transport the data most effectively? Client-to-cloud streaming (chunking in HTTP/1 parlance), periodic batching with POST requests or something I don't know about in HTTP/2?

Imported from https://trello.com/c/0Rzly0xL/90-output-plugin-kcs-metrics-streaming

Module: crypto

• md5()
• sha1()
• sha256()
• sha512()
• rand()

Imported from https://trello.com/c/82UnP6nB/96-module-crypto

Notes on testing

Started testing Speedboat now and are writing things down as I go. It's both questions and suggestions in a mix.

Since I'm not that involved in the project many of the things might be already known.

So far I've tested the Docker case and will continue with running Speedboat using go on the system and add thoughts about that as well.

Minors

• Tutorial says import { test } from 'speedboat' should be import { check } ...?
• Link to the tutorials folder from README("For more information, see the included tutorials.")

Docker

• Add full volume mapping path in example to clarify what is the local file on the host. Eg docker run --net=host -v /path/to/local/test_file.js:/load_test.js loadimpact/speedboat run /load_test.js
• "Speedboat doesn't actually exit immediately after the test finishes" is this true? It seems like the test quits and the container is destroyed right after the test is done.
• options like -d 0 / --duration 0 is converted to 0 seconds and the tests quits immediately.
• --net=host does not work the same way when running Docker on Mac you'll need to expose the ports manually using the -p flag.
  • Does the Web UI work? Tried setting -p 6565:6565, -d 60s and navigated to http://127.0.0.1:6565 and get a "Site cannot be reached" error in Chrome.
• Pulling the image from docker hub works good once signed in(not sure if a note about this is needed since the image will be public upon release?).
• Building you own docker image from source is also working good.
• Link to "The Docker manual" is broken

Running with Go

Side note: Installed Go on OSX using binary .pkg installer

• go get github.com/loadimpact/speedboat will only work once public repository?
• When cloning repository to $GOPATH/src/github.com/loadimpact/speedboat and trying to run make/make build I'll have to manually install some go libs:
  • go get github.com/PuerkitoBio/goquery
  • go get github.com/influxdata/influxdb
  • go get github.com/julienschmidt/httprouter
  • go get github.com/robertkrimen/otto
  • go get gopkg.in/guregu/null.v3
  • go get gopkg.in/tylerb/graceful.v1
  • go get gopkg.in/urfave/cli.v1
• Setting duration to 0 gives panic error panic: runtime error: makeslice: len out of range ] 0s / 0s
• Web UI works good.
• The API works good for a running test(tested scaling, pause, start).
• Same as in Docker: "Speedboat doesn't actually exit immediately after the test finishes" is this true?

Implement more of the jQuery API

Manipulation API

• attr() [readonly]
• html() [readonly]
• prop() [readonly]
• text() [readonly]
• val() [readonly]

Traversal API

• add()
• children()
• closest()
• contents()
• each()
• end()
• eq()
• filter()
• find()
• first()
• has()
• is()
• last()
• map()
• next()
• nextAll()
• nextUntil()
• not()
• parent()
• parents()
• parentsUntil()
• prev()
• prevAll()
• prevUntil()
• siblings()
• slice()

Misc API

• get()
• toArray()
• index()
• data() [readonly]

Imported from https://trello.com/c/1XvRjs9A/89-implement-more-of-the-jquery-api

*Unit Tests*

Imported from https://trello.com/c/NiS6gUHI/108-unit-tests

Refactor stat collection

Imported from https://trello.com/c/7Sfx1C1a/129-refactor-stat-collection

logrus hooks to avoid output corruption

Imported from https://trello.com/c/8oS2EFzH/113-logrus-hooks-to-avoid-output-corruption

Colors

Imported from https://trello.com/c/oGjjQIP3/114-colors

Handle context cancellation better

Right now, the simple runner will error all outstanding requests, while the js runner can ignore it (imagine if someone did a while(true) {}…)

Imported from https://trello.com/c/5Ehq1Qi0/122-handle-context-cancellation-better

Scripting examples

We need some good, instructive and functional script examples

• Authentication
• Cookie management
• Add-to-store / Checkout
• API hammering
• Streaming
• File upload

Imported from https://trello.com/c/Ft2pGnSe/93-scripting-examples

SSL support and auth for the API

Checklist

• Serve API over SSL
• Token authentication
• Client cert authentication

Imported from https://trello.com/c/cNVU86AL/110-ssl-support-and-auth-for-the-api

Script API reference

We need a (Slate-powered?) reference documentation for the scripting API

Imported from https://trello.com/c/ixpCG8LD/95-script-api-reference

Thresholds with tags

Right now, we don't process tags locally inside k6 - it'd mean duplicating data for every possible combination of tags, which would result in an unreasonable amount of RAM usage for longer tests, especially tests containing a lot of URLs.

Unfortunately, this means thresholds become limited to querying full metrics, which isn't a useful level of granularity for many use cases.

The current workaround is to declare custom metrics, copy samples off requests into these, and set thresholds on these metrics instead:

let res = http.get("...");
myMetric.add(res.timings.duration);
check(res, {
    "status is 200": (res) => res.status === 200,
})

While this produces correct results, I don't like this approach, because:

1. It's verbose. Gone is the elegant simplicity of:

   check(http.get("..."), {
       "status is 200": (res) => res.status === 200,
   });

2. It clutters the output with useless metrics. Bad for UX. We could make a way to mark metrics as "hidden" here (perhaps by prefixing the name with an _ or something of the sort), but more importantly...

3. It encourages users to resort to the seemingly simpler solution of:

   check(http.get("..."), {
       "status is 200": (res) => res.status === 200,
       "time is <100ms": (res) => res.timings.duration < 1000,
   })

   Coupled with a raised -a/--acceptance, this may produce seemingly correct results with a single k6 instance. However:

   • -a/--acceptance is a blunt instrument. It silences all failures up to a certain point, and is meant to cover for sporadic errors or dropped packets. Outliers in performance are by nature more common, and tuning acceptance up to cover for this will bury legitimate issues with the application being tested.
   • The problem is magnified in distributed tests; a single instance will not have access to the full dataset, and it's entirely possible for a single instance to get a series of slow requests, tainting the entire run with a check failure. The remedy is to up -a/--acceptance, thus perpetuating the above. Thresholds don't have this problem, as they are by nature distributed, and can work against a database.

   Don't encourage users to shoot themselves in the foot.

---

What I want to do is to instantiate submetrics or derived metrics, using an InfluxDB-like syntax to filter out only certain tags. Let's assume we're using a parameter to http.<method>() to add the "kind" tag to requests that should be special-cased - for instance, a login endpoint.

export let options = {
    thresholds: {
        "http_req_duration{kind:}": "avg <= 100",
        "http_req_duration{kind:login}": "avg <= 1000",
    }
}

With these two thresholds set, the engine will figure out to create two hidden metrics called http_req_duration{kind:} (where sample.Tags["kind"] == "") and http_req_duration{kind:login} (where sample.Tags["kind"] == "login"). No explicit custom metrics required.

Issues with k6 v1 API

Feedback from engineering candidate Diego:

I seem to be running into problems when trying to interact with the v1 API, which in turn seems to cause the Ember front-end to display a "An error occurred." message.

It looks like all requests on the v1 API seem to be failing - both trying to fetch http://127.0.0.1:6565/v1/xxx and k6 commands that rely on the endpoints (for example "$ k6 stats") raise a panic on the lingering k6 instance. My scenario would be:

1) launching a k6 run for a script, pausing it when completes via the
lingering flag:
$ docker run --rm --net=host -v /tmp/scripts/localhost.js:/myscript.js loadimpact/k6:latest run -u 1 -d 1s -l /myscript.js
(execution is fine - http requests are made, etc and pauses when the specified amount of time passed)

2) launching a request for the v1 api, via HTTP or command
2.1) launch another command on another shell (ex. stats)
-----
$ docker run --rm --net=host  loadimpact/k6:latest stats
time="2016-12-17T15:26:47Z" level=error msg=Error error="unexpected end of JSON input"
unexpected end of JSON input

2.2) panic on the other shell with the k6 run
2016/12/17 15:26:47 [Recovery] panic recovered:
GET /v1/metrics HTTP/1.1
Host: 127.0.0.1:6565
Accept-Encoding: gzip
User-Agent: Go-http-client/1.1


interface conversion: interface {} is nil, not *lib.Engine
/usr/local/go/src/runtime/panic.go:458 (0x431883)
    gopanic: reflectcall(nil, unsafe.Pointer(d.fn), deferArgs(d), uint32(d.siz), uint32(d.siz))
/usr/local/go/src/runtime/iface.go:201 (0x414c2f)
    panicdottype: panic(&TypeAssertionError{iface.string(), haveString, want.string(), ""})
/go/src/github.com/loadimpact/k6/api/common/context.go:37 (0x59151e)
    GetEngine: return ctx.Value(ctxKeyEngine).(*lib.Engine)
(...)

Imported from https://trello.com/c/1kc7skeF/99-issues-with-k6-v1-api

Exit with a non-zero exit code if checks fail

Imported from https://trello.com/c/ClDp7wUf/104-exit-with-a-non-zero-exit-code-if-checks-fail

Hosted k6 documentation

Along with launch of k6.io marketing site, we want to plug in a developer-friendly, inviting and useful documentation framework.

Let's start by discussing our options in terms of good and potential frameworks to leverage. What other open source projects or public documentation will be useful as reference to model our site from?

Checklist

• decide on documentation framework
• write/merge documentation
• merge/integrate with k6.io

Imported from https://trello.com/c/GKPwLLr1/120-hosted-k6-documentation

JS: http: Local HTTP clients, with their own TLS state, etc.

Imported from https://trello.com/c/8IcCqfjD/97-js-http-local-http-clients-with-their-own-tls-state-etc

Expose request timings to JS

Imported from https://trello.com/c/npfe5seb/124-expose-request-timings-to-js

Design logotype + styleguide/typography

Imported from https://trello.com/c/8hcUWLnx/94-design-logotype-styleguide-typography

Separate threshold VMs by metric type

Different metric types have different variables. Right now, we're just setting inactive variables to null, but that means you can do avg<0.5 for rate metric by accident and get a threshold that always silently fails if there's also a threshold on a trend metric defined in your test.

Imported from https://trello.com/c/EkSvX9rr/112-separate-threshold-vms-by-metric-type

Custom Metrics

Imported from https://trello.com/c/ne1YecF8/126-custom-metrics

Run a set number of iterations

Imported from https://trello.com/c/0Q3eClL9/116-run-a-set-number-of-iterations

Deduplicate HTTP client code

Right now, every runner has its own HTTP client code, including separate, identical requests and errors metrics + submission code for them.

Full k6 control from API

DNS lookup produces duration spike

Initial response duration spikes are collected at least when testing loadimpact.com.

Imported from https://trello.com/c/mSdpLjRd/117-dns-lookup-produces-duration-spike

https://zapier.com/engine/hydrate/875802/.eJwVjFsOgyAQAO-y334AyiMcolcg67qthEob2cQY491Lf2cmc0GuTbASp7xA1JNzY_B-gGfm95IqbgwRUARp3bhKgwFoZSqp8AkxeBuU6ehTpdsk5_ffP3pVDtxfDeIFhHtfgw12xpm8dgYna0eljGfWDu77B2ucKck:1cK13e:HlTPrSiruzbYTOBnBQH2i1BYxr8/

improved error message when missing import file

JS-file:

import { Counter, Rate } from "k6/metrics";
import http from "k6/http";
import https from "jadaja/http";
:
:
:

yields:

time="2016-12-01T22:15:55Z" level=error msg="Couldn't create a runner" error="Error: open /go/src/github.com/loadimpact/k6/js/lib/jadaja/http.js: no such file or directory" 
Error: open /go/src/github.com/loadimpact/k6/js/lib/jadaja/http.js: no such file or directory
Davids-MacBook:samples davidrosen$ 

when file jadaja/http.js is missing.

Imported from https://trello.com/c/MX84A4Jp/111-improved-error-message-when-missing-import-file

Get otto pull #205 merged!!

This needs to be fixed and merged before we can release: robertkrimen/otto#205

We CANNOT release with a hard dependency on an external binary + a Node runtime.

Imported from https://trello.com/c/eG7k6Is3/91-get-otto-pull-205-merged

Don't reuse connections between VU iterations

All these zeroes in http_req_connecting…

Better flow control for test scripts

Checklist

• Rename test() -> check()
• Make check() return whether all checks succeeded
• Make group() return false if a falsy value was returned
• Add taint() for marking a test as "tainted", which results in a non-zero exit code (failed checks = auto-taint)

Imported from https://trello.com/c/r24TQPLf/105-better-flow-control-for-test-scripts

Please, cant we remove the directory traversal prevention?

This one:
// To protect against directory traversal, prevent loading of files outside the root (pwd) dir

Why? Because above all it makes using the app harder

There are also other arguments against it:

• Are we aiming to make an app that is safe from local exploits? That is a tall order. Right now, there are likely other, equally serious local exploits that we would have to start working on, if our aim is to provide a "secure" app. I question whether this feature alone makes much of a difference to a determined attacker. I think it is much better to avoid all security concerns inside our app and focus on the core functionality of generating traffic and measuring stuff.

• There exists simple-to-use external functionality that provides the same level of security, or better (chroot).

Spec API for cloud integration

The WIP document specifying the protocol/API:
https://loadimpact.quip.com/08WVAPpFVbfG

Tasks

• Specify how we intend to use JWTs
• Specify API endpoints in terms of JSONAPI
• Create state diagram for test statuses
• Specify metrics format for streaming/upload and download

Imported from https://trello.com/c/QbNxMPku/98-spec-api-for-cloud-integration

Implement window.performance

https://www.w3.org/TR/hr-time/

Checklist

• global: performance
• performance.now()

Imported from https://trello.com/c/6wyslOop/121-implement-window-performance

Read external JSON/YAML config files

Imported from https://trello.com/c/ZtgiqNWv/128-read-external-json-yaml-config-files

Engine: Internalize Options; no need to duplicate info

Imported from https://trello.com/c/PY23A15y/127-engine-internalize-options-no-need-to-duplicate-info

Rework CLI UI/UX

Imported from https://trello.com/c/wRUf1Gbf/101-rework-cli-ui-ux

https://zapier.com/engine/hydrate/875802/.eJwVjEEOwiAQAP-y5x4syLLhEX6BLBQswaIpm5im6d_F68xkTiitC7eYfFnAzXdETdZOkEt6Lb7xlsABi3Bct9SkwwRxTbH6mg5wZA3d1EDvJsN6OT7__jGq-uX92cGdEHkfazBkAusQtFUZZ1RIWfFCCNf1A22HKgk:1cK0cU:wREbIdBbmX3oQe_hoxXtsh7YuRI/

Tag metrics with groups!

Needed to handle display in the web UI

Imported from https://trello.com/c/ABerXvwQ/123-tag-metrics-with-groups

Full k6 control from API

After playing around with remote controlling k6 from another process I've discovered some things that's missing or would ease the integration considerably.

• I'd like to be able to start k6 in some kind of "slave mode" where it only spins up the API. From there on I want to interact with k6 via the API only. I plan to start k6 with standard in, out and err bound to /dev/null and detached from any shell.
• I'd like to have a "self destruct" API endpoint (v1/quit?) to kill speedboat when it's time to stop the test and tear down output plugins and similar.
• To even get the test going, I need a v1/run endpoint that would work as we discussed earlier. This would be the way to send the javascript scenario + config in a POST-request to a process started in a way as described above.

These are thoughts based on last weeks testing/discovery coding and are not finalized suggestions. Hoping for a good discussion.

Imported from https://trello.com/c/G5sdTlC4/86-full-k6-control-from-api

Make group IDs UUIDs or something

We need something that can be deterministically issued across multiple processes - no race conditions.

Imported from https://trello.com/c/AQNHjdUI/107-make-group-ids-uuids-or-something

Implement Stages

We need a way to define test stages with VU ramps. Should be exposed as a timeline in the API, and a loop in the engine should continously work the current state towards the desired state.

Imported from https://trello.com/c/rD7ZdNyy/125-implement-stages

Look into source maps

Imported from https://trello.com/c/0TkNzbgC/119-look-into-source-maps

More built-in metrics

Metrics

• VU: runtime
• VU: sleep time
• group: runtime
• request: total size
• request: header size
• request: body size

Tags

• ...

Imported from https://trello.com/c/x0XHOp8D/106-more-built-in-metrics

Ability to disable local thresholds

Imported from https://trello.com/c/mz8fXjEK/118-ability-to-disable-local-thresholds

Provide percentiles for trend metrics

Imported from https://trello.com/c/4eNlmHL4/100-provide-percentiles-for-trend-metrics

Installers / packaging

Create installers and packages that make it simple to install k6 for a large set of users

Imported from https://trello.com/c/rrMtv2Op/92-installers-packaging

Web UI isn't serving anymore

I accidentally let a commit that removes the static file serving into develop, adding it back ASAP.