Run PHP using the official integration API on AWS

[brefphp]

Run PHP using the official integration API on AWS

[mnapoli]

Here is a summary.

Intro

To have PHP support on Lambda with the new API we need 2 parts:

• PHP binary available in /opt/bin (auto-added to $PATH) (+ extensions somewhere), provided through a "layer"
• a bootstrap file in /opt (through a "layer") or /var/task (i.e. the lambda code)

Both are IMO loosely related. I'll talk about those separately.

PHP binary & extensions

The Stackery binary in their "layer" is disappointing for now, I consider it unusable at the moment.

Using Bref's scripts I have compiled PHP 7.2 and published a layer that provides the binary + extensions (+ a bootstrap file but you can ignore it). The ARN of the layer is arn:aws:lambda:us-east-1:416566615250:layer:php-72:4 if you want to try it (the version may change).

This part is IMO the "easiest" to do and less interesting. It's basically about compiling PHP and publishing an AWS layer.

Bootstrap file

Now THIS is where there are VERY interesting opportunities. This is were I'm incredibly excited!

The bootstrap file is basically the process manager of the lambda. When the lambda starts the bootstrap file is called (it can be a PHP script or anything executable). It is responsible for calling an HTTP API that stalls until an event is available to process by the Lambda.

When this HTTP calls finally returns the event data, the code of our PHP app should execute and process the event. Then the HTTP API should be called again to signal the end of the lambda's execution with the response to return.

Let's compare with PHP-FPM:

• PHP-FPM starts, starts a pool of PHP processes
• PHP-FPM waits for an HTTP request
• an HTTP request comes: PHP-FPM runs a PHP process to handle the request
• the PHP process returns a response, PHP-FPM returns the response

With Bref 0.2 (before the new integration):

• the lambda starts and executes the JS handler
• the JS handler waits for an event
• an event comes: JS executes a PHP process to handle the event
• the PHP process returns the response, JS returns that response

With the new Lambda integration:

• the lambda starts and executes bootstrap
• bootstrap waits for an event (calls the HTTP API which is blocking until an event is available)
• an event comes: bootstrap executes a PHP process to handle the event
• the PHP process returns the response, bootstrap returns that response

The last example is what can be done if we simply port Bref's behavior to the new runtime API.

But we can explore other solutions.

For example Stackery's runtime provides a bootstrap that runs the PHP built-in webserver (php -S ...). So the bootstrap file starts the PHP webserver and forwards the HTTP requests to it.

This is an idea worth exploring. However again I do not consider this bootstrap usable at the moment (because of its quality basically).

BUT we are not limited to those options. Here are a few I tried.

Scenario A

We run the PHP code in the same process as bootstrap, very similarly to what can be done with ReactPHP/Amp/Aerys/...

This is very fast, both for cold starts and warm requests! We can get response times below 10ms with that. However, just like when using such frameworks outside of lambda we have disadvantages: the memory is shared between all requests. That means we can have memory leaks, we have to be careful about global state, etc. Also a fatal error will kill the whole lambda (a new lambda will be started by AWS but that means a new cold start).

This is a very interesting option that can be worth proposing as an option, but it cannot be the solution that will work with all apps/frameworks.

Example of a bootstrap:

<?php
// ...
require __DIR__ . '/vendor/autoload.php';

// BOOT Symfony BEFORE a request comes in!
$kernel = new Kernel('prod', false);
$kernel->boot();
$symfonyAdapter = new SymfonyAdapter($kernel);

while (true) {
    // This is a blocking HTTP call until an event is available
    $event = waitForEventFromLambdaApi();

    $request = RequestFactory::fromLambdaEvent($event);
    // REUSE the same Symfony Kernel, meaning fast response time!
    $response = $symfonyAdapter->handle($request);
    $lambdaResponse = LambdaResponse::fromPsr7Response($response);

    signalSuccessToLambdaApi($lambdaResponse);
}

Scenario B

The bootstrap starts a sub-process (exec) every time an event needs to be processed.

That allows to protect the bootstrap process from failures of the children. This is basically what Bref does at the moment.

This is similar too to how PHP-FPM works (in the spirit at least).

Example of a bootstrap:

<?php
// ...
while (true) {
    // This is a blocking HTTP call until an event is available
    $event = waitForEventFromLambdaApi();

    $process = new Process(['/opt/bin/php', 'index.php', /* pass the event too */]);
    $process->setTimeout(null);
    // This waits for the process to finish
    $process->run();

    // [fetch response ...]

    signalSuccessToLambdaApi($lambdaResponse);
}

Example of a index.php that could be executed by that:

<?php
// ...
require __DIR__ . '/vendor/autoload.php';

// [fetch event from process args]

$kernel = new Kernel('prod', false);
$kernel->boot();
$symfonyAdapter = new SymfonyAdapter($kernel);
$request = RequestFactory::fromLambdaEvent($event);
$response = $symfonyAdapter->handle($request);
$lambdaResponse = LambdaResponse::fromPsr7Response($response);

// [return response to bootstrap somehow]

// DIE!
exit(0);

Scenario C

Just like B except bootstrap does not handle events: it immediately executes a sub-process. The PHP sub-process will call the integration HTTP API and wait for an event. That means that we can run code before waiting for an event. E.g. we can bootstrap Composer's autoloader and Symfony before a request comes in!

Example of a bootstrap:

<?php
// ...
while (true) {
    $process = new Process(['/opt/bin/php', 'index.php']);
    $process->setTimeout(null);
    // This waits for the process to finish (i.e. waits until an event has been processed)
    $process->run();
}

Example of a index.php that could be executed by that:

<?php
// ...
require __DIR__ . '/vendor/autoload.php';

// BOOT Symfony BEFORE a request comes in!
$kernel = new Kernel('prod', false);
$kernel->boot();
$symfonyAdapter = new SymfonyAdapter($kernel);

// This is a blocking HTTP call until an event is available
$event = waitForEventFromLambdaApi();

$request = RequestFactory::fromLambdaEvent($event);
$response = $symfonyAdapter->handle($request);
$lambdaResponse = LambdaResponse::fromPsr7Response($response);

signalSuccessToLambdaApi($lambdaResponse);

// DIE!
exit(0);

Scenario D

How about instead of creating a new process we fork the bootstrap process?

The app would bootstrap once in total, but still there is no shared state between events (because each event is processed by a fork).

Example of bootstrap:

<?php
// ...

require __DIR__ . '/vendor/autoload.php';

// BOOT Symfony ONLY ONCE for all the requests!
$kernel = new Kernel('prod', false);
$kernel->boot();
$symfonyAdapter = new SymfonyAdapter($kernel);

while (true) {
    $pid = pcntl_fork();
    if ($pid) {
        // Root process
        // Wait for the child to process the event
        pcntl_wait($status);
    } else {
        // Child process
        // Here the autoloader is already loaded and Symfony initialized!

        // This is a blocking HTTP call until an event is available
        $event = waitForEventFromLambdaApi();

        $request = RequestFactory::fromLambdaEvent($event);
        $response = $symfonyAdapter->handle($request);
        $lambdaResponse = LambdaResponse::fromPsr7Response($response);

        signalSuccessToLambdaApi($lambdaResponse);

        // The fork DIES! The root process will resume its execution
        exit(0);
    }
}

I find this scenario very interesting. This is something I've always wanted to try implement with PHP-FPM (boot the app before a request comes in) but was never able to because it requires knowing C.

Conclusion

With Lambda's execution model and API it is now possible to basically recreate PHP-FPM but with any language, without having to care about the load (because we handle only 1 event at a time). A new world of possibilities is opening! How about other ideas? How about workers or other types of events? How about websockets?

Performances?

Here are a few benchmarks I did. I admit being disappointed by the performances of the fork test, am I doing things wrong? WDYT?

There are lambda execution time for handling one HTTP event from API Gateway (cold start excluded):

• scenario A: 6ms without a framework (Slim framework, didn't try Symfony yet)
• scenario B: did not try
• scenario C: 95ms for Symfony, 75ms without a framework
• scenario D: 50ms for Symfony, 17ms for Slim, 13ms without a framework

Symfony performances are quite bad in general here, I don't know why (I did run in prod environment with cache generated). I don't trust my own benchmarks (was a bit tired) so don't take them too seriously.

I did not spend time measuring cold starts but they were between 100ms and 500ms.

[nealio82]

I cannot wait to try this - but I won’t have time until after Wednesday :( FTR, I love how quickly you’ve / we’ve adapted to the changes from AWS and how much better the project is becoming because of everyone’s excitement and enthusiasm.

[mnapoli]

For those following here: the benchmarks have spoken.

We'll have 2 stable runtimes:

• PHP CLI for simple lambdas, workers, cron, etc.
• PHP FPM for HTTP lambdas (API Gateway)

We'll have an extra experimental runtime that I'll document later when I get more time.

I've started building the PHP CLI runtime in #106. The second part is in #115.

[mnapoli]

You can start testing 🎉

See #116 if you are interested in testing!

[joelwurtz]

I see also other problems to solution A :

• If a script use register_shutdown_function it will not be called after his scripts end but rather when the process die
• If there is some memory leaks and objects in this memory leaks have actions on their destructor, they will only be called when the process die
• Some extensions may have shutdown actions when request is finished and they will only be called when the process die

Basically everything that does what this function do : https://github.com/php/php-src/blob/67e0138c0dfd966624223911a0821f6c294ad1c6/main/main.c#L1857 will not be done on the solution A

It may be ok in some uses case, but i think it's dangerous to be a default behavior. (Maybe Bref can have multiple bootstraping solution ?)

[sandrokeil]

This is really promising. 👍 Are you aware of swoole? It looks like a mix of Scenario A and D. Zend Expressive has support for Swoole. More details in this blog post.

[deleugpn]

I studied Lambda on the php scenario a bit during the last few months (sporadic research) and the reason I dropped most of my interest was not performance, but rather price. Although lambda is extremely cheap, when I realized that I had to support 1 million request per 5 days I concluded that Fargate is cheaper than Lambda.
The reason I bring this up is because I don't think the performance is that much relevant. Lambda will be fast enough to give a good user flow no matter the scenario / option and at the end of the day it is more likely for you to swap lambda for something else not because of the performance, but for the cost of running too much requests. Moving to Fargate or similar will allow the developer to be back with full opcache, preloading and any performance booster necessary.
Perhaps one thing you might consider (and I would totally help here) is to try and provide a seamless API that allow the user to deploy on Lambda and, when necessary, bref can help promote the project into a full blown ECS container on Fargate with the same bref deploy (by changing a config somewhere maybe). That could mean your primary focus for lambda is just be stable and easy while using Fargate for promoting performance-first projects.

Sorry if I sounded too crazy here, the idea just popped up and I got carried away 🤣

[mnapoli]

Great news! 🎉

I have created a repository to document the benchmarks and share the results: https://github.com/mnapoli/bref-bootstrap-benchmarks

I have published new numbers there now that I run with opcache and those are very interesting!

Solution D is actually twice faster than current Bref performances!

There is also now solution E (PHP-FPM), F (built-in webserver) and G (custom PHP SAPI).

@joelwurtz thinks solution G is doable (but requires to code it in C) and if it is, it might be very interesting in terms of performances! It would be the equivalent of PHP-FPM (which works with FastCGI) but to work with AWS Lambda's API. Let's see how we can get that ball rolling in the next days.

Please also help with some missing scenarios, I can run the benchmark myself. I'm not sure anyone will be able to provide E without having to compile PHP and all so it's not easy. But if someone can do F and also B that would be awesome. Running the benchmarks on a LAMP stack would be very helpful too to compare AWS Lambda to LAMP!

Also please if you can review the benchmarks, especially the Symfony code! I might be missing some optimizations! I have also opened 2 issues on that repo if you want to help.

[mnapoli]

Update:

• I managed to compile and publish my own PHP 7.2 runtime, with all the extensions currently provided by Bref (much better since PHP 7.1 is becoming EOL)
• Stackery's bootstrap file is very buggy and WIP, this first version is not usable at all, I don't know what is planned there
• I am exploring writing another bootstrap from scratch that does not use/require pcntl and forks, will report when I have more info

Building our own runtime doesn't sound so like a bad idea honestly, we already have a good build script.

[nealio82]

Do you have a WIP branch that we can try out & contribute to?

[nealio82]

If you've managed to create a PHP 7.2 runtime easily enough, with everything Bref currently supports, and less buggy than the 3rd party version, then I can't see any real wins from depending on another project.

[mnapoli]

If you've managed to create a PHP 7.2 runtime easily enough, with everything Bref currently supports, and less buggy than the 3rd party version, then I can't see any real wins from depending on another project.

@nealio82 agreed.

Also if you want to try out all that see https://gist.github.com/mnapoli/573e4f36a241e458fe9395b779f87511 This is very rough for now sorry about that.

For the next weeks I have 2 full days every week to work on all of that. There is something awesome to be done here I can feel it… Any help is welcome! I also tried diving into PHP's source code (mainly PHP FPM and PHP's built-in webserver). If only I knew more about C, I'm sure a PHP-FPM clone that would work with AWS's API and not FastCGI would kill everything.

[barryvdh]

Did you compare the performance to the current Bref implementation? I guess ideally, scenario B would be faster, or at least equal to the current Nodejs based version, but with the benefit of being in PHP and not having to ship the binary, right? Would that be a good thing to start with, getting this approach on par with the current Bref?

Good job!

[nealio82]

Maybe I don't understand Lambda's execution model well enough - but my first question is this: if we the bootstrap is happening before a request has come in & then waiting for requests, does that mean the Lambda function is running all the time? In that case, how does this differ from running an EC2 box with PHP installed on it? Or does the bootstrap happen once for each cold start when the container boots, but you're only charged by AWS between request and response when the function code is running?

Also, will scenario D still be relevant after PHP pre-loading becomes a thing? In userland we could pre-load the entire framework. https://wiki.php.net/rfc/preload

Should we document each method and allow the end-user to choose which scenario they want to implement? Maybe for some C would be the better option, whereas others might find A suits their needs the most.

[mnapoli]

@hectorj made me realize the example of scenario D wasn't optimal. I have edited it to reflect what I actually tested: I fork first then the child waits for the event (instead of forking after). That allows to save the overhead of forking in the response time. The example wasn't showing that so I edited my comment.

Did you compare the performance to the current Bref implementation?

@barryvdh No that's a good point! I just tested the same demo that I deployed earlier this year with Bref and I see an average execution time of 80ms. So better than scenario C (and logically B too), but worse than the fork option. That's surprising that the new integration is slower than the Node-based one 🤔 There must be something going on here.

What I should do is create a repository with all the test cases laid out so that it is automated, reproducible and reviewable.

does that mean the Lambda function is running all the time?

@nealio82 from what I understand yes, the container and the bootstrap script is running all the time (as soon as it is started ofc).

In that case, how does this differ from running an EC2 box with PHP installed on it?

You only pay for the actual execution time, not the time it's waiting for requests. Also scaling of each "worker"/container is handled by AWS (no limit compared to an EC2 machine).

Or does the bootstrap happen once for each cold start when the container boots

The bootstrap executable is started when the container boots, but stays alive. This is why in JS a lambda looks like this:

// init the lambda: this is the cold start (plus the container start time obviously)

exports.myHandler = function(event, context, callback) {
   // this is the part executed on each event
}

The Node process stays alive and waits for events to execute the callback. This is basically the same in PHP now if we create a bootstrap PHP file and a while loop that waits for events.

but you're only charged by AWS between request and response when the function code is running?

Exactly.

[nealio82]

The Node process stays alive and waits for events to execute the callback. This is basically the same in PHP now if we create a bootstrap PHP file and a while loop that waits for events.

Oh, I always assumed it just invoked a new process each time :S

[nealio82]

Could this also potentially open the door for some interesting stuff like installing the blackfire.io daemon?

[mnapoli]

Yep now that they've opened the engine and explained the internals everything makes much more sense! This is actually simple in the end: the container starts and executes bootstrap. I wonder if you can "hack" that to run long-running stuff and if they stop your process if you abuse it somehow… Anyway not the topic here!

Could this also potentially open the door for some interesting stuff like installing the blackfire.io daemon?

Exactly! (at least that's what I understood!) This is why they boast New Relic integration (and other monitoring tools) with the layer thing: you can add the New Relic layer which will add the daemon binary. I guess then it's up to you to start it in your bootstrap. Clever!

Oh and I just want to note before I forget: I did not compile and enable opcache in the new layer! This is maybe what is affecting performances so much 🤔 (they are enabled in Bref, but there is a warning in my Symfony-deployed lambda with Bref that opcache is not loading… I need to fix this and publish it. Or if anyone wants to do it go ahead!)

[deleugpn]

About the current performance being better than the native layer, remember that the current bref is basically a web server written in NodeJS and delegating requests to php binary.
This reasoning was what made me show people that this is quite good and not such a bad hack. If you think about it, php ALWAYS relied on a web server written on another language (apache, nginx).
It's not that surprising that booting up a Node process is faster than booting up a layer with native php because in the end you still need an event to grab the request and run php.
While not helpful for the layer thing, here's something you might want to play with: spiral/roadrunner.
It's a web server for php written in golang . Go has native support on Lambda.
Remember the native language only matter for the web server and we don't have a lot of web servers written in php AFAIK (excluding react). So perhaps bref could also provision a golang lambda with roadrunner and run php behind it.

[mnapoli]

It's not that surprising that booting up a Node process is faster than booting up a layer with native php because in the end you still need an event to grab the request and run php.

Yes that's right!

Roadrunner is indeed something to consider but AFAIK not different from React/Aerys/Amp/PHP-PM in the sense that a single PHP process will handle many requests/events. So there is shared memory to care about (memory leaks, etc.). This is the same thing as solution A, except with another language involved. I don't see a benefit over solution A 🤔

[deleugpn]

I didn't think roadrunner would keep the php process up. I though it was more like Apache: spawn children processes that will be the event (take requests) and run php.
That being the case it is indeed not so relevant.
OTOH if goland is better at managing processes (idk, just speculating), perhaps it could still be a better option than process managers written in php itself.

[barryvdh]

If you're using something like https://github.com/php-pm/php-pm for option A, there is the benefit of already having the framework integrations (Symfony, Laravel etc). But I think it probably still needs to be optional, because of shared memory (doesn't always work well enough).

[mnapoli]

@barryvdh I'm not sure I see what we would gain with php-pm, it already works with all (major) frameworks right now.

[barryvdh]

I meant if you want to go with option A, which shares the Symfony/Laravel etc bootstrap code. Right now it works because you start a 'fresh' php process, so no shared memory/containers etc, right? php-pm would be faster after the cold-start, because it doesn't have to autoload/boot the container etc. But I think you have to make sure some container stuff is reset etc. But I might just be misunderstanding what Bref/php-pm exactly does exactly.

[OskarStark]

Did you mean „no concurrent requests“ instead of „no current requests“ in the PR header?

[mnapoli]

@OskarStark yes I fixed that thanks.

php-pm would be faster after the cold-start, because it doesn't have to autoload/boot the container etc.

@barryvdh yes, but that's the same with solution A because the bootstrap script runs in a loop: it bootstrap Symfony once and then handles request in the same PHP process forever. So you have to be careful about resetting everything on every loop (1 loop iteration = 1 event to process).

[joelwurtz]

@mnapoli php-pm will do worker restart when memory usage is too high, so i think this is the better choice

[mnapoli]

It may be ok in some uses case, but i think it's dangerous to be a default behavior.

@joelwurtz completely agree, these are exactly the same problems as any other long running web app with PHP-PM/React/Amp/etc. So yes it will not be the default. But it's great that some people already running these technologies (i.e. they have apps developed with that in mind) can benefit of AWS Lambda.

So yeah, A is not the default but will be offered (as the benefits are very real).

Now the default solution could be B, C, D or something else. I think there is something worth exploring with the forks, and I wonder if there's not something else to do with PHP-FPM: can we take the code of PHP-FPM and make it work with Lambda's integration instead of FastCGI?

We could even do that in another language (Go, even PHP?). I think the key thing here is understand how PHP-FPM reuses the same PHP processes but without them sharing memory between requests. How do they "reset" the memory of those processes? How do PHP-FPM work?

This is key to avoid the overhead of booting a PHP process on every event.

[joelwurtz]

This is basically how php fpm work (may be not 100% right, but closes)

FPM Start a master process

1. Master process does the init stuff (init modules / ...)
2. Master process fork himself in order to create fpm workers
3. Master process enter in an event loop where it's only job is to shutdown / startup workers

Child process :

1. Init stuff for child process
2. Listen to a fastcgi request
3. Execute php_request_startup to prepare php vm
4. Execute script with the request
5. Execute php_request_shutdown to clean php vm
6. Exit if he has to shutdown (max requests / max time / ....)
7. Go back to step 2

You can see the child request loop here : https://github.com/php/php-src/blob/67e0138c0dfd966624223911a0821f6c294ad1c6/sapi/fpm/fpm/fpm_main.c#L1878

[barryvdh]

Instead of replication FPM, can't you use it directly? Or does that remove the performance gain?
https://stackoverflow.com/questions/30238602/how-to-execute-a-php-file-using-a-php5-fpm-pool-socket

[mnapoli]

@sandrokeil yes, I place it in the same category as A and the related solutions (Amp, React, PHP-PM, etc.). The reason for that is that there is no isolation between events/requests. (unless I'm mistaken).

Solution D allows to have a complete separation between requests as the whole state of the PHP process is reset every time.

@barryvdh yes good point, let's call this Scenario E :) (we have to test it if we want the comparison to be serious).

I suspect it won't have awesome performances because we need a bootstrap that receives the AWS event (written in PHP or something else) and that forwards that to PHP-FPM's FastCGI interface, which would in turn forward that to PHP. But maybe we'll have a good surprise.

[mnapoli]

@deleugpn that's an interesting idea but clearly out of scope for now ^^ Maybe later once everything is stable!

And TBH I don't reach the same conclusions as you regarding pricing but I'd rather keep this thread on topic 😉 so let's discuss that at another time.

[barryvdh]

Sorry if I'm not making sense, I have little experience with Lambda, so not sure how the execution model works for cold/hot starts. I've did some testing on my local Mac with a very simplistic test case using https://github.com/hollodotme/fast-cgi-client

Scenario B = runExec or runProcess,
Scenario E = runFastcgi

This seems to reduce the overhead of calling a new process pretty much.

$ php scenarios.php 
runExec COLD: 92.12ms
runExec HOT: 86.6ms
runProcess COLD: 95.06ms
runProcess HOT: 90.22ms
runFastcgi COLD: 3.55ms
runFastcgi HOT: 0.47ms

See script: https://gist.github.com/barryvdh/75667b91f4cd9820ae9c746d752166b7

So obviously that depends on how many times the VM is re-used etc and perhaps on some configuration (I guess you would just want 1 fpm worker in this case)

(Note; a cold start is probably not really a cold start here, because opcache etc loads things already and php fpm is already running)

[barryvdh]

I don't really have time to create the benchmark, still need to setup docker/AWS SAM etc. But this is what I tried for the webserver:

$server = new Process("php -S localhost:8000 'index.php'");
$server->setTimeout(null);
$server->start(function ($type, $output) {
    if ($type === Process::ERR) {
        echo($output);
        exit(1);
    }
});

// Wait for the server to start
sleep(1);

register_shutdown_function(function() use($server) {
    $server->stop();
});

From there it should be easy to use Guzzle or any other PSR-7 client to send the request (with modified host perhaps) to localhost). Response times seem pretty fast, but downside is that I can't really seem to reliably detect if the webserver is started, so you have to wait a while (1 sec is what stackery does). But I can't detect the output, only retry messages of the connection is refused. But not ideal.