yandex/pandora

A load generator in Go language

pandora's Issues

netsample binary Aggregator

First line: json metadata. Format like:

format: binary
date: 20-03-2017 20:17:53.122
type: response_stat
columns:
  - name: ts
    type: float64
  - name: urt
    type: int64
config: 
   pools:
     - id: 
     ....

Than print fields as is in big endian. One sample per line.

Benchmark Pandora performance on real stand and compare it to Phantom

Let's suggest test cases below:
Max HTTP RPS
Max HTTPS RPS
Max concurrent TLS connections

Extremely strict rules for uri ammo files

Please, just ignore the spaces instead of stopping the test:
Pool failed. Canceling started tasks {"pool": "pool_0", "error": "provider faile d: failed to decode ammo at line: 1; data: \"[Accept-Encoding: gzip,deflate,sdch] \": header line should be like '[key: value]\n

Pass common dependencies to core plugins

Pass core.Deps struct to Aggregator, Provider and Gun as an arg to their Run (Bind) methods.

What put to core.Deps:
1). zap.Logger
2) some metrics registry. I think something custom based on go-metrics
3) registry, to register what will be logged as every second shooting status
4) afero.Fs? Not shure really.
Document, that some fields may be added in future versions.

core.DataSource plugin

We want go use different data sources for Providers, but not create multiple providers with same decoding logic.
Proposed, to create core.DataSource plugin, that will contain logic of getting encoded ammo bytes, and use it as nested plugin in Providers that takes bytes from somewhere and decodes them.

Here some draft:

type DataSource {
    // Open opens data source. 
    // Returned source can be file, socket, string or byte reader, or anything else, that implements
    // more that io.Reader, so it's ok to check if source is io.Seeker, and seek if it is.
    // Specific type checking (*os.File, for example) highly not recommended - use interfaces.
    Open() (source io.Closer, err error)
}

What DataSources can be implemented:

• file (filepath to open)
• socket (some string, that can be passed to net.Listen)
• url (http/https URL to download)
• inline (string inlined to config)

nil pointer panic on config parse

https://gitter.im/yandex/yandex-tank?at=5a37f70e540c78242ddcebce

Support phantom ammo files

See doc for details: http://yandextank.readthedocs.io/en/latest/tutorial.html#request-style
Blocked by #30

Get config from stdin

Add a posibility to receive config contents from stdin. Like less UNIX command.

Advanced configuration and extension

Use configuration by spf13/viper + mitchellh/mapstructure to:

• get rid of manual map[string]interface{} decode in extpoints fabric functions
• support multiple configuration formats ( json, yaml, ...) for free
• case insensitive names
• decode hooks

Use go-playground/validate for config-parse time validation.

Also consider to make plugin creation easy like:

package custom_gun
type Gun struct { config Config}
var _ pandora.Gun = (*Gun)(nil)
type Config { Target string `validate:"hostname"` }
func New(config * Config) *Gun {  return &Gun{config} }

init () { pandora.RegisterGun( New, "my_gun") }

Of course that requires some reflect based magic.

Config ammo provider for http gun

Read ammo from config file and cache them in memory

Error in console in the end of a test

Error waiting utils promises: 1 error(s) occurred:

* context canceled2017/06/23 21:11:00 Done

Config:

pools:
  - id: HTTP pool                    # Pool name
    gun:
      type: http                     # Gun type
      target: [my-host-here]:80  # Gun target
    ammo:
      type: uri                      # Ammo format
      file: ./ammo.uri               # Ammo File
    result:
      type: phout                    # Report format (phout is for Yandex.Tank)
      destination: ./http_phout.log  # Report file name
    rps:                             # RPS Schedule
      type: periodic                 # tick periodically
      batch: 1                       # in batches of two ticks
      max: 300                         # three batches total
      period: 0.1s                     # one batch each second
    startup:                         # Startup Schedule
      type: periodic                 # start Instances periodically
      batch: 1                       # one Instance at a time
      max: 5                         # five Instances total
      period: 0.5s                   # every 0.5 seconds

Ammo:

/

Pareto schedule

Add schedule with Pareto distributed inter-arrival times. May be support other distributions.

Ubuntu packaging

Make an ubuntu package and upload to PPA

Use github.com/pkg/errors instead of github.com/facebookgo/stackerr

pkg/errors more popular, fast, and logged to zap in more readable way

Extra ammo consumed at instance schedule finish

Instance takes ammo before schedule token take, that allows be ready, when take next schedule token, and operate close to schedule.
But in case of individual schedules, extra ammo will be taken.

Propose add IsFinished() bool schedule method, that allows check is schedule finished before taking ammo.

In such case, in case of shared schedule, some extra ammos can be consumed anyway at schedule finish, but that is minor problem - that will happen once after shooting finish.

Pre resolve target address

Now HTTP gun make DNS resolve per every connection to target. Resolve should be done once.

Scenario gun

Try some scripting languages support libraries and implement a scenario gun using one of them.

• https://github.com/robertkrimen/otto
• https://github.com/aarzilli/golua
• https://github.com/sbinet/go-python
• https://github.com/mattn/go-v8
• http://blog.labix.org/2013/07/16/twik-a-tiny-language-for-go

Stop instance creation and finish shooting, if ammo or RPS schedule has been finished

Blocked by #55

Use github.com/golang/dep for dependency managment

It's looking good and should become official tool soon.

Refactor limiters

• use Timer and Timer.Reset instead of After
• implement Periodic limiter using Timer instead of Ticker
• don't use channels for limiters composition

Basic idea: generate timestamps with a function and use them to drive the Timer.

Automatic tag generation for phttp gun

Generate tags automatically based on ammo contents. For example, take first n uri elements:

/my/very/deep/page?id=23&param=33 -> _my_very

Use external JSON implementation for ammo

https://github.com/pquerna/ffjson

Headers in config file for URI-style http ammo

Add Headers config parameter for URI-style http ammo.

Now we can specify headers in ammo file, but there are cases when we want to use same ammo file with different headers. It would be convenient not to regenerate whole file.

See also: #60 #52

Make sample reporting non blocking

From Aggregator doc:

// If Aggregator can't process reported sample without blocking, it should just throw it away.
// If any reported samples were thrown away, Run should return error describing how many samples
// were thrown away.

We should implement that behaviour in all existing Aggregators.

Composite schedule

Add composite schedule that sequentially combines other schedules

Use glide for package management

Glide is most powerful package manager.
It supports aliasing packages (e.g. for working with github forks) and many other things

Use httptrace to get Connect/Send/etc metrics in guns

See Phout format metrics format and description.

Benchmark guns, user pools and aggregator

Implement reproducible benchmarks that will allow to measure performance of our components.

Update Yandex.Tank plugin

Update yandex-tank plugin to include new features support. https://github.com/yandex/yandex-tank

Get rid of /vendor folder

Pandora repo designed to be used as library, as well as binary.
But /vendor folder cause compilation error, when Pandora used ad library.
I propose to get rid of /vendor folder, but remain glide.lock and glide.yaml files, and use them to have reproducible builds in CI.

Test limiters correctness

Implement test cases to check if limiters produce ticks according to specified schedule.

Support POST requests in HTTP and SPDY guns.

Support POST requests (and other types also) in HTTP and SPDY guns.

Crashes on start with big "times" parameter when compiled with go 1.9.1

I got some weird random crashes of pandora compiled with go 1.9.1 with config such as "rps": {"type": "const", "ops": 15000, "duration": "5m"}, "startup": {"type": "once", "times": 500}

Sometimes it's fatal error: sweep increased allocation count, and sometimes it's a SIGSEGV.
pandora_ZaeI79.log

It seems that everything works fine when I compile with go 1.8

HTTP2 gun

Now HTTP gun allows HTTP/2, but this behavior is too implicit.
Create explicit HTTP/2 gun, and forbid HTTP/2 for HTTP gun.

Acceptance testing

Need to automate compile and run of Pandora binary and check its behaviour. It would be great to check it before every PR in CI.
Propose:
Create acceptance_tests directory and create onsi/gomega/gexec driven tests for

• CLI behaviour
• Main plugins behaviour

I imagine tests like

• Before suite: compile binary via onsi/gomega/gexec
• Some test
  • launch httptest.Server and run binary against testdata/http.yaml config
  • Check pandora log output (for data races end errors, for example) and return code.
• Other tests

Parallelism option

Make parallelism option in config, with 1 value by default.
When parallelism is more that one, instances should call Gun.Shoot in separate goroutine, using maximum parallelism in parallel.

That option will be very helpful for http2 gun.

Support uri-style ammo files

See doc for details: http://yandextank.readthedocs.io/en/latest/tutorial.html#uri-style-uris-in-file
Blocked by #30

Collect and publish generator status

Collect and publish generator status metrics such as parallel requests count, ammo loop count, users count and others. Publish every n seconds, n should be configurable.
Every plugin (gun, user, etc) should be able to publish its metrics in a common way.

Get rid of not working "multi pool" feature for phout Aggregator

We can add this feature in #61 if needed

Fast leveled logging

Get rid of stdlib logger, and use something leveled and fast. I suggest uber-go/zap.
At degug level, logs should include:

• requests
• responses
• internal actions
  • user start/finish
  • composite limiter steps
  • etc.

Update readme

Link to examples is broken.
Also I got this error if startup section configured to linear type (as suggested in config):

cli/cli.go:101	Config decode failed	{"error": "1 error(s) decoding:\n\n* error decoding 'pools[0].startup': no plugins of type core.Schedule has been registered for name linear"

JSON and MessagePack reflect aggregators & providers

JSON and MessagePack are fast enough, and supported widely.
Such modules will be nice for quick implementations.
When performance become problem, it will be easy to replace reflect version with fast code generated.

Try fasthttp

Try https://github.com/valyala/fasthttp when we'll have benchmarks.

Add http2 gun

https://godoc.org/golang.org/x/net/http2 - it is still under development, but works fine

TLS features like

Would be nice to have configurable options for cipher_suits and SNI in pandora

Plugin type defaults

Every plugin implementation can have it's own defaults. But user still needed to set 'type' field for every plugin.
I propose to add default types for every plugin.

Clear way to use one aggregator in multiple pools

Usually, even when using multiple instance pools, wanted to use one Aggregator for them.
Now, phout aggregator solving this problem in not clear way: aggregator config passed to all needed places, but underhood only aggregator created, but many Run routines started. And seems, that there are some bugs in this non clear implementation.

I propose add explicit mechanism to use one aggregator in multiple pools.
How I think it should be looking in config:

pools:
  - ...
    aggregator: 
      type: master
      # key: some_key # "default" by default, so should not be typed when only master created.
      config:
        type: phout
        ....
  - ...
    aggregator:
      - type: slave
       # key: some_key # "default" by default, so should not be typed when only master created.

Master and slaves use one global registry, where found each other config key.
Only master runs real aggregator background routine, and it's context should not be canceled until master and all slaves context cancel. Context values passed from master context.

Slave aggregator instance should just block until context cancel, or master subroutine finished

Ammo enumeration

Implement a flag in config that will switch on/off ammo enumeration. Each ammo will have a number after # sign in tag:

_my_tag#33

Abstract HTTP ammo

Make HTTP ammo something like

type HTTP interface {
    GetRequest() (*http.Request, SmthElse)
}

That allows easily use different ammo sources with different http based guns.

Use rcrowley/go-metrics instead of ugly wrappers over expvar

Current metrics reporting is very ugly.
I propose to use rcrowley/go-metrics with json HTTP reporter.