Comments (25)
Hi!
I personally chose to redirect Twig's cache to config/packages/prod/twig.yaml
to be able to warm up other bundles.
# config/packages/prod/twig.yaml
twig:
cache: '/tmp/cache/twig'
I know it's a temporary solution but I preferer doing this because making /tmp
the default Symfony cache directory increases the cold start from 500ms to 2 seconds.
I hope Twig's team will work on this problem to comply with the Symfony4 best practices (var/cache should be read-only and warmable).
from bref.
I reverted my changes in Kernel.php
and did the following:
β bref-symfony-demo git:(master) β composer update
Loading composer repositories with package information
Updating dependencies (including require-dev)
Prefetching 2 packages π΅
- Downloading (100%)
Package operations: 2 installs, 20 updates, 0 removals
- Installing matomo/ini (2.0.0) Loading from cache
- Updating symfony/process (v4.0.9 => v4.1.1) Loading from cache
- Updating zendframework/zend-diactoros (1.7.1 => 1.8.1) Loading from cache
- Updating aws/aws-sdk-php (3.55.9 => 3.62.12) Loading from cache
- Updating symfony/http-foundation (v4.0.9 => v4.1.1) Loading from cache
- Installing symfony/polyfill-ctype (v1.8.0) Loading from cache
- Updating symfony/yaml (v4.0.9 => v4.1.1) Loading from cache
- Updating symfony/finder (v4.0.9 => v4.1.1) Loading from cache
- Updating symfony/filesystem (v4.0.9 => v4.1.1) Loading from cache
- Updating symfony/console (v4.0.9 => v4.1.1) Loading from cache
- Updating mnapoli/bref (0.2.4 => 0.2.18) Loading from cache
- Updating symfony/routing (v4.0.9 => v4.1.1) Loading from cache
- Updating symfony/event-dispatcher (v4.0.9 => v4.1.1) Loading from cache
- Updating symfony/debug (v4.0.9 => v4.1.1) Loading from cache
- Updating symfony/http-kernel (v4.0.9 => v4.1.1) Loading from cache
- Updating symfony/dependency-injection (v4.0.9 => v4.1.1) Loading from cache
- Updating symfony/config (v4.0.9 => v4.1.1) Loading from cache
- Updating symfony/cache (v4.0.9 => v4.1.1) Loading from cache
- Updating symfony/framework-bundle (v4.0.9 => v4.1.1) Loading from cache
- Updating symfony/twig-bridge (v4.0.9 => v4.1.1) Loading from cache
- Updating symfony/twig-bundle (v4.0.9 => v4.1.1) Loading from cache
- Updating symfony/dotenv (v4.0.9 => v4.1.1) Loading from cache
Writing lock file
Generating autoload files
What about running composer global require symfony/thanks && composer thanks now?
This will spread some π by sending a β
to the GitHub repositories of your fellow package maintainers.
Executing script cache:clear [OK]
β bref-symfony-demo git:(master) β php vendor/bin/bref deploy
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service .zip file to S3 (22.52 MB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
..
Serverless: Stack update finished...
Service Information
service: bref-demo-symfony
stage: dev
region: eu-west-3
stack: bref-demo-symfony-dev
api keys:
None
endpoints:
ANY - https://j1wj4kj13e.execute-api.eu-west-3.amazonaws.com/dev
ANY - https://j1wj4kj13e.execute-api.eu-west-3.amazonaws.com/dev/{proxy+}
functions:
main: bref-demo-symfony-dev-main
Deployment success
8/8 [ββββββββββββββββββββββββββββ] 1 min%
I see the same exception(s) related to the cache again
[STDERR] 2018-07-10T12:19:17+00:00 [critical] Uncaught Exception: Unable to write in the cache directory (/var/task/var/cache/prod/twig/dc).
2018-07-10T12:19:17+00:00 [critical] Uncaught PHP Exception RuntimeException: "Unable to write in the cache directory (/var/task/var/cache/prod/twig/dc)." at /var/task/vendor/twig/twig/lib/Twig/Cache/Filesystem.php line 59
[STDERR] 2018-07-10T12:19:17+00:00 [critical] Exception thrown when handling an exception (RuntimeException: Unable to write in the cache directory (/var/task/var/cache/prod/twig/dc). at /var/task/vendor/twig/twig/lib/Twig/Cache/Filesystem.php line 59)
[STDERR] 2018-07-10T12:19:17+00:00 [critical] Uncaught Exception: Unable to write in the cache directory (/var/task/var/cache/prod/twig/dc).
Fatal error: Uncaught RuntimeException: Unable to create the cache directory (/var/task/var/cache/prod/twig/ba). in /var/task/vendor/twig/twig/lib/Twig/Cache/Filesystem.php:55
Stack trace:
#0 /var/task/vendor/twig/twig/lib/Twig/Environment.php(369): Twig_Cache_Filesystem->write('/var/task/var/c...', '<?php\n\n/* hello...')
#1 /var/task/vendor/twig/twig/lib/Twig/Environment.php(289): Twig_Environment->loadTemplate('hello.html.twig')
#2 /var/task/vendor/symfony/framework-bundle/Controller/ControllerTrait.php(224): Twig_Environment->render('hello.html.twig', Array)
#3 /var/task/src/Controller/HomeController.php(12): Symfony\Bundle\FrameworkBundle\Controller\Controller->render('hello.html.twig')
#4 /var/task/vendor/symfony/http-kernel/HttpKernel.php(149): App\Controller\HomeController->index()
#5 /var/task/vendor/symfony/http-kernel/HttpKernel.php(66): Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object(Symfony\Component\HttpFoundation\Request), 1)
#6 /var/task/vendor/symfony/http-kernel/Kernel.php(188): Symfony\Compone in /var/task/vendor/twig/twig/lib/Twig/Cache/Filesystem.php on line 59
from bref.
Workaround documentation added in #42
from bref.
I'll be closing this issue since it was opened in 2019, and since then we have the Symfony bridge that should take care of the cache π
from bref.
Regarding the memory size problem, you could try increasing the memory limit of PHP by customizing the php.ini
config, see https://github.com/mnapoli/bref#php-configuration to see how to set php.ini
flags. However 128Mb seems already quite much, I'm surprised Symfony would consume that much.
Presumably setting the cache dir to Lambda's local /tmp kind-of defeats the point pre-warming the cache with the hooks in the .bref.yml file? The object is to avoid launching the application without the cache already in place, right?
Agreed, this is not really efficient in production that's why it's better to pre-generate the cache.
I get the same behaviour with cloning the mnapoli/bref-symfony-demo repo and running bref deploy
This is surprising, but since I wrote (and tested) bref-symfony-demo we have added some extensions to PHP (mainly opcache, the other extensions are disabled by default), maybe that could be it? I don't really know much more right now I'll try to dig in.
from bref.
The first thing I tried was to increase the memory_limit
(I should have mentioned that above, sorry!)
I think the cache is not being pre-warmed during the deploy process, because presumably it should exist and be readable if it has been!?
As mentioned, if you set the cache dir to the writeable /tmp/
dir then the problem disappears and PHP doesn't complain about memory any more
from bref.
OK, I'll try to reproduce that tonight and see how it goes. Thanks for opening the issue!
from bref.
We use our own kernel here, but since we also use symfony/dependency-injection
we had to find a solution to pre-warm the cache, which essentially means to build and dump the container.
The only way we found to do so is to use relative paths.
For instance, we have a parameter %cache_dir%
. Its value is always ./var/cache
. This allows us to dump the container before uploading it on lambda.
Our kernel always check that the current working directory is $_ENV['LAMBDA_TASK_ROOT']
.
This works and we don't build the container on each execution. I think you could do something similar with the Symfony kernel.
That beeing said, if you have a better solution, I would be very interested.
from bref.
@nealio82 I've updated (composer update
) everything to the latest versions on the symfony-demo project and redeployed, it still works: https://7oaryq3rzl.execute-api.eu-west-3.amazonaws.com/dev I don't understand what breaks for you π€
@t-geindre Yes! I had issues with absolute/relative paths when deploying the API platform demo for my benchmarks. In the end I gave up and stored the cache in /tmp
because I was not measuring the cold starts, so I just had to warmup the application and that's it. But this is not ideal at all.
from bref.
from bref.
Interestingly, if I clone the mnapoli/bref-symfony-demo
repo, I get a different (and nicer) exception message about the cache.
The symfony/website-skeleton
still gives me out of memory, but the SF demo now specifically mentions the cache.
I made a screencast here: https://youtu.be/Ar3USl8h8Ug
from bref.
Thanks for the screencast! Could you try one last time mnapoli/bref-symfony-demo
by running composer update
before bref deploy
to install the last version of Bref?
from bref.
If I add the cache
param to config/packages/twig.yml
with the value of either /tmp/...
or false
, then everything works again.
twig:
paths: ['%kernel.project_dir%/templates']
debug: '%kernel.debug%'
strict_variables: '%kernel.debug%'
cache: '/tmp/twig'
see https://twig.symfony.com/doc/2.x/api.html#environment-options and http://symfony.com/doc/current/reference/configuration/twig.html#cache
However, I don't like setting /tmp/...
in the twig config, and I don't know enough about Twig to tell if disabling the cache is a bad idea.
I also don't get why Twig needs to write to the cache if the templates are pre-compiled!?
from bref.
OK I think I remember now where the problem came from (when I tried deploying the API platform demo).
Some things in the Symfony cache are using absolute paths, as @t-geindre mentioned. You need to tweak Symfony and its config to force it to use relative paths (because the paths on your machine and on AWS lambda are different).
That works, except for Twig that IIRC uses realpath()
as some point in the code (I think it's here). Because of that call, a relative path is turned into an absolute path and it breaks things. I don't remember the details anymore though.
Also it is important to note that not all the Symfony cache is pregenerated, for example you have to put Doctrine's cache (which is not pregenerated) in /tmp
manually.
Here is an example of what I did for Doctrine in config/packages/doctrine.yaml
:
parameters:
writable_cache_dir: '/tmp/cache'
[β¦]
doctrine:
orm:
[β¦]
metadata_cache_driver:
cache_provider: metadata_cache
result_cache_driver:
cache_provider: result_cache
query_cache_driver:
cache_provider: query_cache
[β¦]
doctrine_cache:
providers:
metadata_cache:
aliases: [doctrine.orm.default_metadata_cache]
file_system:
directory: "%writable_cache_dir%/doctrine/metadata"
result_cache:
aliases: [doctrine.orm.default_result_cache]
file_system:
directory: "%writable_cache_dir%/doctrine/result"
query_cache:
aliases: [doctrine.orm.default_query_cache]
file_system:
directory: "%writable_cache_dir%/doctrine/query"
I did the same for the user cache:
framework:
cache:
directory: "%writable_cache_dir%/app"
from bref.
Ok, so I assume that means at some point we have to have a writeable filesystem in order for Twig to work properly?
However, that raises 2 questions:
- Should the
Kernel.php
in the documentation be updated to change the cache dir to/tmp
to make sure Twig can always write to it? - Why does your example work for you and not for me? You saw in the screencast that I made no adjustments to the example repo's code!
from bref.
Should the Kernel.php in the documentation be updated to change the cache dir to /tmp to make sure Twig can always write to it?
That could be a temporary solution. I still hope to find the correct configuration to apply to solve that (or maybe fix the problem in Twig/Symfony). But yeah it's better to have helpful documentation for now.
Why does your example work for you and not for me? You saw in the screencast that I made no adjustments to the example repo's code!
Yes I don't know how to explain that either :/
from bref.
from bref.
Does anyone here has a public repository to reproduce this?
from bref.
from bref.
For those reading this issue, I reproduced it when writing a simple Symfony 4 application, starting with Flex. I just have a base template (the layout) and a few pages that extend from that. I don't have time right now to create a whole repository to reproduce that but that can maybe be a starting point for others.
from bref.
Weirdly I tried yesterday; starting with Symfony 4 & extending a base template, then passing in some variables to render. I didnβt see the issue at all. (And iβm pretty sure the first time I saw the issue I didnβt even need to extend a base template)
from bref.
Thanks that's a much better solution! I'll try to update the documentation in the coming weeks (I should have more time than this month), if anyone wants to get on this though feel free.
from bref.
@thibaudlemaire by any chance do you know why it adds a 4x overhead? Have you looked into that with Blackfire, for example? The Symfony best practice is to override it in the Kernel
class: https://symfony.com/doc/current/configuration/override_dir_structure.html#override-the-cache-directory
from bref.
When you override the cache directory, pre-warmed cache cannot be used because either it's warmed up in your local /tmp
dir that is not packed by sam package
command, or during Symfony execution it tries to retrieve cache from the lambda container's /tmp
directory that is empty on first request. I think we cannot apply blindly what's explained in the link you gave.
The 4x overhead is due to the cache warmup when a Lambda container is created. I don't know exactly what is compiled at the first execution but I suppose : Doctrine proxies, annotations, Twig views, Kernel config, services container, etc.
That's why we need to make Symfony use the pre-warmed cache. To do so, I thought about two solutions :
- Use
/tmp
as cache directory and copy (or symlink) pre-warmed cache files on lambda container initialisation. I don't know exactly how to do it. A solution may be to add this feature to Bref's event handler to make it copy (or symlink) cache on first invoke. - Or use the default Symfony cache directory (
./var/cache
), warm it up before deployment so thatsam package
pack cache too. This should be possible for all bundles because Symfony recommends not to write in the cache directory after deployment. For bundles that are not compliant with this best practice (like Twig), we still can override the cache directory to make them write in/tmp
.
I use the second one because it's easier. But it's not ideal because twig views still need to be compiled on first invoke.
from bref.
When you override the cache directory, pre-warmed cache cannot be used because either it's warmed up in your local
/tmp
dir that is not packed bysam package
command, or during Symfony execution it tries to retrieve cache from the lambda container's/tmp
directory that is empty on first request. I think we cannot apply blindly what's explained in the link you gave.
Ah, of course! π€¦ββ
from bref.
Related Issues (20)
- Feedback for βCron tasks on AWS Lambdaβ HOT 1
- Handle "Response too big" in Console functions HOT 2
- PlanetScale removed the Hobby tier
- Long running CLI commands repeat every 2 minutes HOT 1
- Troubleshooting Bref Docker Deployment with Octane Handler
- Symfony CLI DATABASE_URL Not Found HOT 1
- Can I fire putJobSuccessResult event from AWS Lambda which is developed with Bref? HOT 1
- Feedback for βCustom domain namesβ HOT 1
- I am getting this issue :-
- OPENSSL_3.3.0' not found (required by /opt/lib/libssl.so.3) HOT 10
- Stopping 1.x maintenance
- Feedback for βServerless Laravel - Getting startedβ HOT 3
- NOTICE since upgrading to v2 HOT 1
- Brefv2 500 Errors Not In Logs HOT 5
- PHP ext-intl is required on layer 8.3 HOT 1
- CURL SSL_ERROR_SYSCALL calling another Lambda HOT 7
- Env always return null after update packages to bref 2 version
- max_execution_time not working HOT 3
- Laravel 11, PHP 8.3 mysql driver not enabled, also can't use postgress
- Bref Serverless v4 HOT 12
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bref.