Performance - benchmarks?

[mindplay-dk]

Performance - benchmarks?

[weierophinney]

I did some benchmarking back in January, and found it magnitudes faster than using a framework, but it was an informal, one-off benchmark. I'll ask Zeev and Dmitry to do some benchmarks soon to see what they find.

I'd like to see your setup if I could, however; when I benchmarked before, the results were measured in a few microseconds, allowing thousands of requests per second. Admittedly, the middleware was just doing a "hello world" response, but the results should be indicative of the baseline performance.

[mindplay-dk]

Yep, just ran some quick benchmarks - a bare $request = ServerRequestFactory::fromGlobals(); was about 0.25 msec, while the "hello world" example in the README with Server and a MiddlewarePipe was 0.45 msec - so it's definitely fast enough.

My benchmark earlier today, as mentioned, was with ab, against Apache and NGINX respectively, both of which showed the same difference in overhead with/without the stack. The benchmark results above were done with a simple benchmark loop on the console, e.g. no web server overhead or bootstrapping between requests - so bare bones performance of the library looks good.

I have no idea what's causing this insane overhead when running under an actual web server - I'll have to look into it in the morning. Clearly something works very differently under a console as opposed to under NGINX or Apache... I'll let you know what I find out.

[mtymek]

I remember that initially ZF2 had some weird performance issue, that was solved by setting some specific headers in response object. Unfortunately I can't find related thread :(

[weierophinney]

@mtymek That's not likely the issue here. @mindplay-dk reported that running executing his middleware without stratigility was when he saw the issue; at that point, everything is the same with regards to the response object.

[weierophinney]

I've created a quick benchmark suite. It adds zend-diactoros and zend-stratigility as requirements, and then defines the following files under a public/ tree::

<?php
// public/middleware.php
return function ($req, $res, $next = null) {
    $res->getBody()->write('Hello world!');
    return $res;
};

<?php
// public/server-emitter.php
use Zend\Diactoros\Response;
use Zend\Diactoros\Response\SapiEmitter;
use Zend\Diactoros\ServerRequestFactory;

require __DIR__ . '/../vendor/autoload.php';

$request    = ServerRequestFactory::fromGlobals();
$response   = new Response();
$middleware = require __DIR__ . '/middleware.php';
$emitter    = new SapiEmitter();

$emitter->emit($middleware($request, $response));

<?php
// public/server-server.php
use Zend\Diactoros\Server;

require __DIR__ . '/../vendor/autoload.php';

$middleware = require __DIR__ . '/middleware.php';
$server     = Server::createServer(
    $middleware,
    $_SERVER,
    $_GET,
    $_POST,
    $_COOKIE,
    $_FILES
);
$server->listen();

<?php
// public/server-middleware.php
use Zend\Diactoros\Server;
use Zend\Stratigility\MiddlewarePipe;

require __DIR__ . '/../vendor/autoload.php';

$middleware = new MiddlewarePipe();
$middleware->pipe(require __DIR__ . '/middleware.php');
$server     = Server::createServer(
    $middleware,
    $_SERVER,
    $_GET,
    $_POST,
    $_COOKIE,
    $_FILES
);
$server->listen();

This allows testing three strategies:

• Executing the middleware directly and using the SapiEmitter to emit it.
• Creating a Server instance with the middleware, and listen()ing to it.
• Creating a Server instance that composes a Stratigility MiddlewarePipe that wraps the middleware, and listen()ing to it.

I am using Zend Server 8 with PHP 5.6.7 and Apache 2.4.10, with default configuration.

I then used Siege to benchmark. Siege is well-regarded as a more robust tool for benchmarking than ab, and is interesting in that it allows specifying a set of paths to traverse (complete with cookies!) if desired. For this case, I executed it with minimal settings on each of the three endpoints:

$ siege -b -t60S <url>

In each case, I hit my server's concurrency limits around 50s into the siege. The results were as follows:

Script	Transactions	Ave Response Time (s)	Transaction Rate	Concurrent Cxns
emitter	94582	0.01	1892	14.40
server	91548	0.01	1884	14.39
middleware	85846	0.01	1691	14.41

Siege measures response times in seconds, so it rounds to around 10ms. In previewing pages using Z-Ray, I was seeing around 100ms results, but this was likely due to, well, having Z-Ray injected into the resulting payload.

Essentially, using the Emitter and using Server are statistically the same. Wrapping the middleware in Stratigility's MiddlewarePipe loses around 10% in performance.

I pulled up the server-middleware.php example in Z-Ray to see where time is being spent.

The largest amount of time spent is with Composer, setting up autoloading; this accounts for >50% of execution. The next largest execution time is in ServerRequestFactory::fromGlobals(), which accounts for ~10% of execution. Stratigility's total functionality accounts for < 10% of execution, which includes instantiation, request/reponse decoration, and execution of middleware (which is around 50% of all Stratigility overhead, and which encompasses calls to Next).

The conclusion from this: Stratigility provides a very slim layer on top of PSR-7 to provide a middleware foundation. It does impose a peformance overhead, but it's minimal, and primarily due to increased number of objects in the object graph.

[GeeH]

Great work @weierophinney - thanks!

I've created a quick benchmark suite. It adds zend-diactoros and zend-stratigility

I'm still not 100% convinced those names aren't elaborate April Fool pranks.

[Ocramius]

Can we enable performance testing via something like
https://github.com/polyfractal/athletic ?

Marco Pivetta

http://twitter.com/Ocramius

http://ocramius.github.com/

On 28 May 2015 at 20:34, Gary Hockin [email protected] wrote:

Great work @weierophinney https://github.com/weierophinney - thanks!

I've created a quick benchmark suite. It adds zend-diactoros and
zend-stratigility

I'm still not 100% convinced those names aren't elaborate April Fool
pranks.

—
Reply to this email directly or view it on GitHub
#5 (comment)
.

[weierophinney]

@Ocramius wasn't aware of that tool! I'll start looking into a benchmark suite we can use with that.

[mindplay-dk]

If you want something you can use and understand in two minutes, try this - here's what I wrote earlier tonight benchmarking Diactoros:

<?php

use mindplay\benchpress\Benchmark;
use Zend\Diactoros\Server;
use Zend\Diactoros\ServerRequestFactory;
use Zend\Diactoros\Uri;
use Zend\Stratigility\Http\Request;
use Zend\Stratigility\Http\Response;
use Zend\Stratigility\MiddlewarePipe;

require dirname(__DIR__) . '/vendor/autoload.php';

$bench = new Benchmark();

$bench->add(
    'initialization',
    function () {
        $request = ServerRequestFactory::fromGlobals();
    }
);

$bench->add(
    'server',
    function () {
        $app = new MiddlewarePipe();

        $app->pipe('/', function (Request $req, Response $res, $next) {
            if ($req->getUri()->getPath() !== '/') {
                return $next($req, $res);
            }
            return $res->end('Hello world!');
        });

        $server = Server::createServer(
            $app,
            $_SERVER,
            $_GET,
            $_POST,
            $_COOKIE,
            $_FILES
        );

        $server->listen();
    }
);

$bench->add(
    'withMethod x 3',
    function () {
        $request = ServerRequestFactory::fromGlobals();

        $request
            ->withMethod('GET')
            ->withMethod('HEAD')
            ->withMethod('POST');
    }
);

$bench->add(
    'withUri',
    function () {
        $request = ServerRequestFactory::fromGlobals();
        $request = $request->withUri(new Uri('http://foo.com:8888/bar?bat=baz'));
    }
);

$bench->run();

Output:

initialization ........ 0.258 msec ..... 17.249 points ....... 57.83% ......... 1.00x
withMethod x 3 ........ 0.268 msec ..... 17.931 points ....... 60.12% ......... 1.04x
withUri ............... 0.298 msec ..... 19.946 points ....... 66.87% ......... 1.16x
server ................ 0.446 msec ..... 29.826 points ...... 100.00% ......... 1.73x

The "points" metric is relative to a measured baseline PHP language performance metric.

[weierophinney]

Benchmarking of this stuff needs to be done in a web environment, as that's where we'll see the real performance hit. This is why I used a web server + siege.

[mindplay-dk]

It's an alternative to e.g. Athletic, not to siege or apache bench etc.

[marc-mabe]

In my opinion there is an issue with this this kind of simple benchmarks as the hello wold app does nothing that will slow down this approve in real live. What I mean is any kind of IO waiting time like db queries fs operations that will effect all requests instead of only the one making this IO and on the other hand some kind of processes that takes CPU will also take time all requests have to wait instead of only the one making this process.

As a workaround you could start using special async extensions like eio but this will not work with all kinds of IO you need and are far less stable. Or as an alternative starting kind progresses like with pcntl.

That's only my 2 cent.

[weierophinney]

@marc-mabe The question was raised as to whether or not Stratigility was providing undue overhead; as the original poster noted. A "hello world" benchmark allows us to determine the overhead of Stratigility itself, and either rule it out or rule it in as a factor with regards to performance. In this case, we were able to essentially rule it out.

[mindplay-dk]

@marc-mabe we use this type of benchmark to benchmark our code for opportunities to optimize - finding bottlenecks in our own code is very useful, and things like waiting for IO are deliberately avoided in this type of benchmark, because it's not useful, we're interested in the performance of our code only, in this type of benchmark. We usually benchmark our applications for real-world performance at a later stage, after it's integrated, using Apache's ab (or something similar) and simulating many users, testing the full effects of the full HTTP round-trip throughput. If that helps :-)