polleverywhere/chaperon

HTTP Service Performance & Load Testing Framework

Chaperon

HTTP Service Performance Testing Framework

This is a framework / library & tool for doing load and performance tests on web services. It tracks many kinds of metrics automatically and allows tracking custom ones, if needed.

A load test is a combination of target web services & scenarios to run against them. It also defines session & HTTP / WebSocket connection settings (like authentication credentials, custom headers, etc.) for each of the services.

Chaperon natively supports running both HTTP & WebSocket actions against a web server. It defines a Chaperon.Actionable protocol for which implementations for additional types of actions can be defined. Have a look at the examples/firehose.ex example file to see an example of both HTTP and WebSocket commands in action.

For a more in-depth introduction check out the basic starter tutorial here.

Documentation & Links

• API Documentation for latest release
• Package on hex.pm

Distributed Load-Testing

Aside from running Chaperon scenarios from a single machine, you can also run them in a cluster. Since Chaperon is written in Elixir, it makes use of its built-in distribution mechanics (provided by the Erlang VM and OTP) to achieve this.

To run a Chaperon scenario in distributed mode, you need to deploy your Chaperon scenario and load test code to all machines in the cluster, start them up and connect to the master node.

To start any node simply load up the code in an iex shell:

$ iex --cookie my-secret-cluster-cookie --name "[email protected]" -S mix

For the master node, run this inside the iex shell:

iex> Chaperon.Master.start

Then enter the following code into any worker's iex shell to connect it to the master node:

iex> Chaperon.connect_to_master :"[email protected]"

Pick one of the nodes as your master node and connect to it from the worker nodes (see above).
Before starting up the child nodes make sure you've given them the same VM cookie and config to point to the master node.
The master node can be identical to the worker nodes, the only difference being that it kicks off the load test and distributes the workload across all worker nodes. When a worker node is done with running a scenario / session task, it sends the results back to the master, which then merges all results to give the final metrics for display / output.

Is this ready for production use?

Chaperon is being used at Poll Everywhere and was written for load testing our infrastructure and polling services. It has been used to simulate over 100k concurrent vote participant sessions on a 4 node cluster.
It currently is still on a pre 1.0 version. A 1.0 release is not planned yet as the main focus is to get rid of any potential bugs, refine the public API and internal implementation code until we're confident that everything works as expected.
That doesn't mean it shouldn't be used in its current stage, though.

If you'd like to try out Chaperon, please give it a try. Any feedback, bug reports or patches are welcome.

chaperon's Issues

A lot of deprecation, credo and general code quality warnings.

warning: redefining @doc attribute previously set at line 1204.

Please remove the duplicate docs. If instead you want to override a previously defined @doc, attach the @doc attribute to a function head (the function signature not followed by any do-block). For example:

    @doc """
    new docs
    """
    def await_signal(...)

  lib/chaperon/session.ex:1229: Chaperon.Session.await_signal/2

warning: function async_metrics/2 is unused
  lib/chaperon/session.ex:1480

warning: function async_results/2 is unused
  lib/chaperon/session.ex:1472

warning: function required_config/2 is unused
  lib/chaperon/session.ex:1008

warning: Supervisor.Spec.worker/2 is deprecated. Use the new child specifications outlined in the Supervisor module instead
  lib/chaperon/master/supervisor.ex:12: Chaperon.Master.Supervisor.start_link/0

warning: Map.size/1 is deprecated. Use Kernel.map_size/1 instead
  lib/chaperon/master.ex:332: Chaperon.Master.running_load_test?/1

warning: Chaperon.Export.InfluxDB.child_spec/0 is undefined (module Chaperon.Export.InfluxDB is not available or is yet to be defined)
  lib/chaperon/supervisor.ex:37: Chaperon.Supervisor.children/0

warning: Supervisor.Spec.supervisor/2 is deprecated. Use the new child specifications outlined in the Supervisor module instead
Invalid call found at 2 locations:
  lib/chaperon/supervisor.ex:15: Chaperon.Supervisor.children/0
  lib/chaperon/supervisor.ex:16: Chaperon.Supervisor.children/0

warning: Supervisor.Spec.worker/2 is deprecated. Use the new child specifications outlined in the Supervisor module instead
Invalid call found at 2 locations:
  lib/chaperon/supervisor.ex:17: Chaperon.Supervisor.children/0
  lib/chaperon/supervisor.ex:27: Chaperon.Supervisor.children/0

Etc. we may do a maintenance cycle, but it would be cool to learn about the maintenance status? Maybe a maintained fork?..

How to calculate req/sec?

Thanks for this great tool!

I'm using this library to test my Phoenix app.

But I'm quite new to load testing. I wonder how to calculate req/sec (using add_metric/3 I assume)?

If this is a feature that can be supported officially?

basic_auth dependency is deprecated

basic_auth has been deprecated in favor of Plug.BasicAuth: https://hex.pm/packages/basic_auth

Can't get example to work

I've tried following the tutorial, but can't get it working. Am I forgetting something?

Here is the error message + setup:

iex(1)> Chaperon.run_load_test(CVA.LoadTest)

09:06:47.609 [error] 22df804b-580b-4a18-b5e8-e3cf9e55f7f3 CVA.Scenario | Failed to connect via WS to wss://echo.websocket.org/ - Connection closed remotely 
 
09:06:47.612 [error] Task #PID<0.358.0> started from #PID<0.359.0> terminating
** (CaseClauseError) no case clause matching: %Chaperon.Session{assigned: %{counter: 0}, async_tasks: %{}, cancellation: nil, config: %{base_url: "wss://echo.websocket.org", echo: %{iterations: 10}, merge_scenario_sessions: true, metrics: nil}, cookies: [], errors: %{}, id: "22df804b-580b-4a18-b5e8-e3cf9e55f7f3", interval_task: nil, metrics: %{}, name: "CVA.Scenario", parent_id: nil, parent_pid: nil, results: %{}, scenario: %Chaperon.Scenario{module: CVA.Scenario}, timeout_at: nil}
    (chaperon 0.3.1) lib/chaperon/session.ex:648: Chaperon.Session.run_action/2
    (chaperon_vote_api 0.1.0) lib/scenario.ex:14: CVA.Scenario.run/1
    (chaperon 0.3.1) lib/chaperon/scenario.ex:198: Chaperon.Scenario.with_scenario/3
    (chaperon 0.3.1) lib/chaperon/scenario.ex:172: Chaperon.Scenario.run/2
    (chaperon 0.3.1) lib/chaperon/scenario.ex:122: Chaperon.Scenario.execute/2
    (elixir 1.10.3) lib/task/supervised.ex:90: Task.Supervised.invoke_mfa/2
    (elixir 1.10.3) lib/task/supervised.ex:35: Task.Supervised.reply/5
    (stdlib 3.12) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Function: &Chaperon.Scenario.execute/2
    Args: [CVA.Scenario, %{base_url: "wss://echo.websocket.org", echo: %{iterations: 10}, merge_scenario_sessions: true, metrics: nil}]
** (EXIT from #PID<0.353.0>) shell process exited with reason: an exception was raised:
    ** (CaseClauseError) no case clause matching: %Chaperon.Session{assigned: %{counter: 0}, async_tasks: %{}, cancellation: nil, config: %{base_url: "wss://echo.websocket.org", echo: %{iterations: 10}, merge_scenario_sessions: true, metrics: nil}, cookies: [], errors: %{}, id: "22df804b-580b-4a18-b5e8-e3cf9e55f7f3", interval_task: nil, metrics: %{}, name: "CVA.Scenario", parent_id: nil, parent_pid: nil, results: %{}, scenario: %Chaperon.Scenario{module: CVA.Scenario}, timeout_at: nil}
        (chaperon 0.3.1) lib/chaperon/session.ex:648: Chaperon.Session.run_action/2
        (chaperon_vote_api 0.1.0) lib/scenario.ex:14: CVA.Scenario.run/1
        (chaperon 0.3.1) lib/chaperon/scenario.ex:198: Chaperon.Scenario.with_scenario/3
        (chaperon 0.3.1) lib/chaperon/scenario.ex:172: Chaperon.Scenario.run/2
        (chaperon 0.3.1) lib/chaperon/scenario.ex:122: Chaperon.Scenario.execute/2
        (elixir 1.10.3) lib/task/supervised.ex:90: Task.Supervised.invoke_mfa/2
        (elixir 1.10.3) lib/task/supervised.ex:35: Task.Supervised.reply/5
        (stdlib 3.12) proc_lib.erl:249: :proc_lib.init_p_do_apply/3

defmodule CVA.LoadTest do
  use Chaperon.LoadTest
  
  def default_config,
      do: %{
        base_url: "wss://echo.websocket.org",
        merge_scenario_sessions: true
      }
  
  def scenarios,
      do: [
        {{1, CVA.Scenario},
          %{
            echo: %{
              iterations: 10
            }
          }}
      ]
end

defmodule CVA.Scenario do
  use Chaperon.Scenario
  
  def init(session) do
    session
    |> assign(counter: 0)
    |> ok
  end
  
  def run(session) do
    iterations = session |> config([:echo, :iterations])
    
    session
    |> ws_connect("/")
    |> repeat_traced(:echo, iterations)
    |> log_info("Echo finished after #{iterations} iterations")
    |> ws_close
  end
  
  def echo(session) do
    msg = "Echo ##{session.assigned.counter}"
    
    session
    |> ws_send(msg)
    |> ws_await_recv(msg)
    |> update_assign(counter: &(&1 + 1))
  end
end

Passing list of parameters for iteration

I believe more information about how the session feature works when it comes to passing parameters to a loadtest.

I get different behavior for different things. When using the back ground noise example the following works:

   def run(session) do
       Enum.map(["foo","bar","tar"], &(async(session, :search, [&1])))
       |> Enum.map(await/1)
   end

This is in the iex though.

however if try to perform the following with a value list that is generated at the scenario definition level using the following:

	Enum.map(session.config.search.value, &(async(session, :search_call, [&1])))
	|> Enum.map(&await(&1))

I get some error. I believe it is related to await expecting a session + map but gets a list of session instead. I also tried the following:

              Enum.map(session.config.search.value, &get(session, "query_range", params: [query: &1, start: start, end: time_end, step: step]))

This cause a similar error, I tried to put this in to run. The intention was to move it then perform an async call.

I have also had errors based on add metrics not being able to deal with list.

I could try running async task using this example instead of the Session.await/2:
Task.async, however I think this will be an overkill since all I want to do is perform get request that iterates through a list of parameters.

I know that there is Scenario definition, but I am not sure how many of this I can spawn. I can define a a parameter per scenario in the list that gets targeted by runloadtest. How many of these can I have ? & Is there an alternative method that you recommend passing multiple parameters when run a load test.

https in default_config?

If I currently define https in the default_config

def default_config,
      do: %{
        base_url: "https://bbc.com"
      }
...

An HTTPoision error is thrown:

2020-10-14 12:33:35.087 [error] d7af5252-312a-41c1-9c34-028316744ca9 Scenario.Http | Session.run_action GET /news failed: %HTTPoison.Error{id: nil, reason: {:options, {:sslv3, {:versions, [:"tlsv1.2", :"tlsv1.1", :tlsv1, :sslv3]}}}}

Should a default TLS version be given?

Release a new version?

Could you release a new version?

We're using this library and experienced a few occasions where mix tries to lock httpoison down to v1.*.* and it conflicts with chaperon's current requirement of ~> 0.11.1. I see that you bumped httpoison version already but the latest release was Sep 6, 2018.

Thanks!