Φορτίο (fortio) is Istio's load testing tool. Fortio runs at a specified query per second (qps) and records an histogram of execution time and calculates percentiles (e.g. p99 ie the response time such as 99% of the requests take less than that number (in seconds, SI unit)). It can run for a set duration, for a fixed number of calls, or until interrupted (at a constant target QPS, or max speed/load per connection/thread).
The name fortio comes from greek φορτίο which is load/burden.
Fortio is a reusable, embeddable go library as well as a command line tool and server process, the server includes a simple web UI and graphical representation of the results (both a single latency graph and a multiple results comparative min, max, avg and percentiles graphs).
- Install go (golang 1.8 or later)
go get -u istio.io/fortio
- you can now run
fortio
(from your gopath bin/ directory)
Or use docker, for instance:
docker run -p 8080:8080 -p 8079:8079 istio/fortio server & # For the server
docker run istio/fortio load http://www.google.com/ # For a test run
Or download the binary distribution, for instance:
curl -L https://github.com/istio/fortio/releases/download/0.4.0/fortio-linux_x64-0.4.0.tgz \
| sudo tar -C / -xvzpf -
You can visit the new web UI at http://localhost:8080/fortio/
Fortio can be an http or grpc load generator, gathering statistics using the load
subcommand, or start simple http and grpc ping servers, as well as a basic web UI, result graphing and https redirector, with the server
command or issue grpc ping messages using the grpcping
command. It can also fetch a single URL's content using the -curl
flag to the load command. You can run just the redirector with redirect
. Lastly if you saved JSON results (using the web UI or directly from the command line), you can browse and graph those results using the report
command.
$ fortio
Φορτίο 0.6.4 usage:
fortio command [flags] target
where command is one of: load (load testing), server (starts grpc ping and
http echo/ui/redirect servers), grpcping (grpc client), report (report only UI
server) or redirect (redirect only server). where target is a url (http load
tests) or host:port (grpc health test) and flags are:
-H value
Additional Header(s)
-a Automatically save JSON result with filename based on labels & timestamp
-allow-initial-errors
Allow and don't abort on initial warmup errors
-c int
Number of connections/goroutine/threads (default 4)
-compression
Enable http compression
-curl
Just fetch the content once
-data-dir string
Directory where JSON results are stored/read (default ".")
-echo-debug-path string
http echo server URI for debug, empty turns off that part (more secure)
(default "/debug")
-gomaxprocs int
Setting for runtime.GOMAXPROCS, <1 doesn't change the default
-grpc
Use GRPC (health check) for load testing
-grpc-port int
grpc port (default 8079)
-halfclose
When not keepalive, whether to half close the connection (only for fast
http)
-health
client mode: use health instead of ping
-healthservice string
which service string to pass to health check
-http-port int
http echo server port (default 8080)
-http1.0
Use http1.0 (instead of http 1.1)
-httpbufferkb int
Size of the buffer (max data size) for the optimized http client in kbytes
(default 128)
-httpccch
Check for Connection: Close Header
-httpreqtimeout duration
Http request timeout value (default 15s)
-json string
Json output to provided file or '-' for stdout (empty = no json output,
unless -a is used)
-keepalive
Keep connection alive (only for fast http 1.1) (default true)
-labels string
Additional config data/labels to add to the resulting JSON, defaults to
target URL and hostname
-logcaller
Logs filename and line number of callers to log (default true)
-loglevel value
loglevel, one of [Debug Verbose Info Warning Error Critical Fatal]
(default Info)
-logprefix string
Prefix to log lines before logged messages (default "> ")
-n int
Run for exactly this number of calls instead of duration. Default (0) is
to use duration (-t). Default is 1 when used as grpc ping count.
-p string
List of pXX to calculate (default "50,75,99,99.9")
-payload string
Payload string to send along
-profile string
write .cpu and .mem profiles to file
-qps float
Queries Per Seconds or 0 for no wait/max qps (default 8)
-quiet
Quiet mode: sets the loglevel to Error and reduces the output.
-r float
Resolution of the histogram lowest buckets in seconds (default 0.001)
-redirect-port int
Redirect all incoming traffic to https URL (need ingress to work properly)
-1 means off. (default 8081)
-static-dir string
Absolute path to the dir containing the static files dir
-stdclient
Use the slower net/http standard client (works for TLS)
-t duration
How long to run the test or 0 to run until ^C (default 5s)
-ui-path string
http server URI for UI, empty turns off that part (more secure)
(default "/fortio/")
- Start the internal servers:
$ fortio server &
Https redirector running on :8081
UI starting - visit:
http://localhost:8080/fortio/
Fortio 0.6.1 grpc ping server listening on port 8079
Fortio 0.6.1 echo server listening on port 8080
- Simple grpc ping:
$ fortio grpcping localhost
02:29:27 I pingsrv.go:116> Ping RTT 305334 (avg of 342970, 293515, 279517 ns) clock skew -2137
Clock skew histogram usec : count 1 avg -2.137 +/- 0 min -2.137 max -2.137 sum -2.137
# range, mid point, percentile, count
>= -4 < -2 , -3 , 100.00, 1
# target 50% -2.137
RTT histogram usec : count 3 avg 305.334 +/- 27.22 min 279.517 max 342.97 sum 916.002
# range, mid point, percentile, count
>= 250 < 300 , 275 , 66.67, 2
>= 300 < 350 , 325 , 100.00, 1
# target 50% 294.879
- Load (low default qps/threading) test:
$ fortio load http://www.google.com
Fortio 0.3.6 running at 8 queries per second, 8->8 procs, for 5s: http://www.google.com
19:10:33 I httprunner.go:84> Starting http test for http://www.google.com with 4 threads at 8.0 qps
Starting at 8 qps with 4 thread(s) [gomax 8] for 5s : 10 calls each (total 40)
19:10:39 I periodic.go:314> T002 ended after 5.056753279s : 10 calls. qps=1.9775534712220633
19:10:39 I periodic.go:314> T001 ended after 5.058085991s : 10 calls. qps=1.9770324224999916
19:10:39 I periodic.go:314> T000 ended after 5.058796046s : 10 calls. qps=1.9767549252963101
19:10:39 I periodic.go:314> T003 ended after 5.059557593s : 10 calls. qps=1.9764573910247019
Ended after 5.059691387s : 40 calls. qps=7.9056
Sleep times : count 36 avg 0.49175757 +/- 0.007217 min 0.463508712 max 0.502087879 sum 17.7032725
Aggregated Function Time : count 40 avg 0.060587641 +/- 0.006564 min 0.052549016 max 0.089893269 sum 2.42350566
# range, mid point, percentile, count
>= 0.052549 < 0.06 , 0.0562745 , 47.50, 19
>= 0.06 < 0.07 , 0.065 , 92.50, 18
>= 0.07 < 0.08 , 0.075 , 97.50, 2
>= 0.08 <= 0.0898933 , 0.0849466 , 100.00, 1
# target 50% 0.0605556
# target 75% 0.0661111
# target 99% 0.085936
# target 99.9% 0.0894975
Code 200 : 40
Response Header Sizes : count 40 avg 690.475 +/- 15.77 min 592 max 693 sum 27619
Response Body/Total Sizes : count 40 avg 12565.2 +/- 301.9 min 12319 max 13665 sum 502608
All done 40 calls (plus 4 warmup) 60.588 ms avg, 7.9 qps
- Curl like (single request) mode
$ fortio load -curl -H Foo:Bar http://localhost:8080/debug
14:26:26 I http.go:133> Setting regular extra header Foo: Bar
HTTP/1.1 200 OK
Content-Type: text/plain; charset=UTF-8
Date: Mon, 08 Jan 2018 22:26:26 GMT
Content-Length: 230
Φορτίο version 0.6.0 echo debug server up for 39s on ldemailly-macbookpro - request from [::1]:65055
GET /debug HTTP/1.1
headers:
Host: localhost:8080
User-Agent: istio/fortio-0.6.0
Foo: Bar
body:
- Report only UI
If you have json files saved from running the full UI, you can serve just the reports:
$ fortio report
Browse only UI starting - visit:
http://localhost:8080/
Https redirector running on :8081
Fortio server
- has the following feature - http listening on 8080 (all paths and ports are configurable through flags above):
- A simple echo server which will echo back posted data (for any path not mentioned below).
For instance
curl -d abcdef http://localhost:8080/
returnsabcdef
back. It supports the following optional query argument parameters:
Parameter | Usage, example |
---|---|
delay | duration to delay the response by. Can be a single value or a coma separated list of probabilities, e.g delay=150us:10,2ms:5,0.5s:1 for 10% of chance of a 150 us delay, 5% of a 2ms delay and 1% of a 1/2 second delay |
status | http status to return instead of 200. Can be a single value or a coma separated list of probabilities, e.g status=404:10,503:5,429:1 for 10% of chance of a 404 status, 5% of a 503 status and 1% of a 429 status |
/debug
will echo back the request in plain text for human debugging./fortio/
A UI to- Run/Trigger tests and graph the results.
- A UI to browse saved results and single graph or multi graph them (comparative graph of min,avg, median, p75, p99, p99.9 and max).
- Proxy/fetch other URLs
/fortio/data/index.tsv
an tab separated value file conforming to Google cloud storage URL list data transfer format so you can export/backup local results to the cloud.- Download/sync peer to peer JSON results files from other Fortio servers (using their
index.tsv
URLs)
The report
mode is a readonly subset of the above directly on /
.
Fortio is written in the Go language and includes a scalable semi log histogram in stats.go and a periodic runner engine in periodic.go with specializations for http and grpc. The http/ package includes a very high performance specialized http 1.1 client. You may find fortio's logger useful as well.
You can run the histogram code standalone as a command line in histogram/, a basic echo http server in echosrv/, or both the http echo and GRPC ping server through fortio server
, the fortio command line interface lives in this top level directory fortio_main.go
With 5k qps: (includes envoy and mixer in the calls)
$ time fortio load -qps 5000 -t 60s -c 8 -r 0.0001 -H "Host: perf-cluster" http://benchmark-2:9090/echo
2017/07/09 02:31:05 Will be setting special Host header to perf-cluster
Fortio running at 5000 queries per second for 1m0s: http://benchmark-2:9090/echo
Starting at 5000 qps with 8 thread(s) [gomax 4] for 1m0s : 37500 calls each (total 300000)
2017/07/09 02:32:05 T004 ended after 1m0.000907812s : 37500 calls. qps=624.9905437680746
2017/07/09 02:32:05 T000 ended after 1m0.000922222s : 37500 calls. qps=624.9903936684861
2017/07/09 02:32:05 T005 ended after 1m0.00094454s : 37500 calls. qps=624.9901611965524
2017/07/09 02:32:05 T006 ended after 1m0.000944816s : 37500 calls. qps=624.9901583216429
2017/07/09 02:32:05 T001 ended after 1m0.00102094s : 37500 calls. qps=624.9893653892883
2017/07/09 02:32:05 T007 ended after 1m0.001096292s : 37500 calls. qps=624.9885805003184
2017/07/09 02:32:05 T003 ended after 1m0.001045342s : 37500 calls. qps=624.9891112105419
2017/07/09 02:32:05 T002 ended after 1m0.001044416s : 37500 calls. qps=624.9891208560392
Ended after 1m0.00112695s : 300000 calls. qps=4999.9
Aggregated Sleep Time : count 299992 avg 8.8889218e-05 +/- 0.002326 min -0.03490402 max 0.001006041 sum 26.6660543
# range, mid point, percentile, count
< 0 , 0 , 8.58, 25726
>= 0 < 0.001 , 0.0005 , 100.00, 274265
>= 0.001 < 0.002 , 0.0015 , 100.00, 1
# target 50% 0.000453102
WARNING 8.58% of sleep were falling behind
Aggregated Function Time : count 300000 avg 0.00094608764 +/- 0.0007901 min 0.000510522 max 0.029267604 sum 283.826292
# range, mid point, percentile, count
>= 0.0005 < 0.0006 , 0.00055 , 0.15, 456
>= 0.0006 < 0.0007 , 0.00065 , 3.25, 9295
>= 0.0007 < 0.0008 , 0.00075 , 24.23, 62926
>= 0.0008 < 0.0009 , 0.00085 , 62.73, 115519
>= 0.0009 < 0.001 , 0.00095 , 85.68, 68854
>= 0.001 < 0.0011 , 0.00105 , 93.11, 22293
>= 0.0011 < 0.0012 , 0.00115 , 95.38, 6792
>= 0.0012 < 0.0014 , 0.0013 , 97.18, 5404
>= 0.0014 < 0.0016 , 0.0015 , 97.94, 2275
>= 0.0016 < 0.0018 , 0.0017 , 98.34, 1198
>= 0.0018 < 0.002 , 0.0019 , 98.60, 775
>= 0.002 < 0.0025 , 0.00225 , 98.98, 1161
>= 0.0025 < 0.003 , 0.00275 , 99.21, 671
>= 0.003 < 0.0035 , 0.00325 , 99.36, 449
>= 0.0035 < 0.004 , 0.00375 , 99.47, 351
>= 0.004 < 0.0045 , 0.00425 , 99.57, 290
>= 0.0045 < 0.005 , 0.00475 , 99.66, 280
>= 0.005 < 0.006 , 0.0055 , 99.79, 380
>= 0.006 < 0.007 , 0.0065 , 99.82, 92
>= 0.007 < 0.008 , 0.0075 , 99.83, 15
>= 0.008 < 0.009 , 0.0085 , 99.83, 5
>= 0.009 < 0.01 , 0.0095 , 99.83, 1
>= 0.01 < 0.012 , 0.011 , 99.83, 8
>= 0.012 < 0.014 , 0.013 , 99.84, 35
>= 0.014 < 0.016 , 0.015 , 99.92, 231
>= 0.016 < 0.018 , 0.017 , 99.94, 65
>= 0.018 < 0.02 , 0.019 , 99.95, 26
>= 0.02 < 0.025 , 0.0225 , 100.00, 139
>= 0.025 < 0.03 , 0.0275 , 100.00, 14
# target 50% 0.000866935
# target 75% 0.000953452
# target 99% 0.00253875
# target 99.9% 0.0155152
Code 200 : 300000
Response Body Sizes : count 300000 avg 0 +/- 0 min 0 max 0 sum 0
Or you can get the data in JSON format (using -json result.json
)
Or graphically (through the http://localhost:8080/fortio/ web UI):
Simple form/UI:
Sample requests with responses delayed by 250us and 0.5% of 503 and 1.5% of 429 simulated http errors.
Run result:
Code 200 : 2939 (98.0 %)
Code 429 : 47 (1.6 %)
Code 503 : 14 (0.5 %)
Contributions whether through issues, documentation, bug fixes, or new features are most welcome !
Please also see Contributing to Istio and Getting started contributing to Fortio in the FAQ.
And make sure to go format and run those commands successfully before sending your PRs:
make test
make lint
make webtest
When modifying Javascript, check with standard:
standard --fix ui/static/js/fortio_chart.js
Our wiki and the Fortio FAQ (including for instance differences between fortio
and wrk
or httpbin
)