Giter Site home page Giter Site logo

Comments (25)

thibaudlemaire avatar thibaudlemaire commented on August 16, 2024 4

Hi!
I personally chose to redirect Twig's cache to config/packages/prod/twig.yaml to be able to warm up other bundles.

# config/packages/prod/twig.yaml
twig:
    cache: '/tmp/cache/twig'

I know it's a temporary solution but I preferer doing this because making /tmp the default Symfony cache directory increases the cold start from 500ms to 2 seconds.
I hope Twig's team will work on this problem to comply with the Symfony4 best practices (var/cache should be read-only and warmable).

from bref.

nealio82 avatar nealio82 commented on August 16, 2024 1

I reverted my changes in Kernel.php and did the following:

➜  bref-symfony-demo git:(master) βœ— composer update
Loading composer repositories with package information
Updating dependencies (including require-dev)

Prefetching 2 packages 🎡
  - Downloading (100%)

Package operations: 2 installs, 20 updates, 0 removals
  - Installing matomo/ini (2.0.0) Loading from cache
  - Updating symfony/process (v4.0.9 => v4.1.1) Loading from cache
  - Updating zendframework/zend-diactoros (1.7.1 => 1.8.1) Loading from cache
  - Updating aws/aws-sdk-php (3.55.9 => 3.62.12) Loading from cache
  - Updating symfony/http-foundation (v4.0.9 => v4.1.1) Loading from cache
  - Installing symfony/polyfill-ctype (v1.8.0) Loading from cache
  - Updating symfony/yaml (v4.0.9 => v4.1.1) Loading from cache
  - Updating symfony/finder (v4.0.9 => v4.1.1) Loading from cache
  - Updating symfony/filesystem (v4.0.9 => v4.1.1) Loading from cache
  - Updating symfony/console (v4.0.9 => v4.1.1) Loading from cache
  - Updating mnapoli/bref (0.2.4 => 0.2.18) Loading from cache
  - Updating symfony/routing (v4.0.9 => v4.1.1) Loading from cache
  - Updating symfony/event-dispatcher (v4.0.9 => v4.1.1) Loading from cache
  - Updating symfony/debug (v4.0.9 => v4.1.1) Loading from cache
  - Updating symfony/http-kernel (v4.0.9 => v4.1.1) Loading from cache
  - Updating symfony/dependency-injection (v4.0.9 => v4.1.1) Loading from cache
  - Updating symfony/config (v4.0.9 => v4.1.1) Loading from cache
  - Updating symfony/cache (v4.0.9 => v4.1.1) Loading from cache
  - Updating symfony/framework-bundle (v4.0.9 => v4.1.1) Loading from cache
  - Updating symfony/twig-bridge (v4.0.9 => v4.1.1) Loading from cache
  - Updating symfony/twig-bundle (v4.0.9 => v4.1.1) Loading from cache
  - Updating symfony/dotenv (v4.0.9 => v4.1.1) Loading from cache
Writing lock file
Generating autoload files

What about running composer global require symfony/thanks && composer thanks now?
This will spread some πŸ’–  by sending a β˜…  to the GitHub repositories of your fellow package maintainers.

Executing script cache:clear [OK]

➜  bref-symfony-demo git:(master) βœ— php vendor/bin/bref deploy
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service .zip file to S3 (22.52 MB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
..
Serverless: Stack update finished...
Service Information
service: bref-demo-symfony
stage: dev
region: eu-west-3
stack: bref-demo-symfony-dev
api keys:
  None
endpoints:
  ANY - https://j1wj4kj13e.execute-api.eu-west-3.amazonaws.com/dev
  ANY - https://j1wj4kj13e.execute-api.eu-west-3.amazonaws.com/dev/{proxy+}
functions:
  main: bref-demo-symfony-dev-main
Deployment success
 8/8 [β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘]  1 min%

I see the same exception(s) related to the cache again

[STDERR] 2018-07-10T12:19:17+00:00 [critical] Uncaught Exception: Unable to write in the cache directory (/var/task/var/cache/prod/twig/dc).
2018-07-10T12:19:17+00:00 [critical] Uncaught PHP Exception RuntimeException: "Unable to write in the cache directory (/var/task/var/cache/prod/twig/dc)." at /var/task/vendor/twig/twig/lib/Twig/Cache/Filesystem.php line 59
[STDERR] 2018-07-10T12:19:17+00:00 [critical] Exception thrown when handling an exception (RuntimeException: Unable to write in the cache directory (/var/task/var/cache/prod/twig/dc). at /var/task/vendor/twig/twig/lib/Twig/Cache/Filesystem.php line 59)
[STDERR] 2018-07-10T12:19:17+00:00 [critical] Uncaught Exception: Unable to write in the cache directory (/var/task/var/cache/prod/twig/dc).
Fatal error: Uncaught RuntimeException: Unable to create the cache directory (/var/task/var/cache/prod/twig/ba). in /var/task/vendor/twig/twig/lib/Twig/Cache/Filesystem.php:55
Stack trace:
#0 /var/task/vendor/twig/twig/lib/Twig/Environment.php(369): Twig_Cache_Filesystem->write('/var/task/var/c...', '<?php\n\n/* hello...')
#1 /var/task/vendor/twig/twig/lib/Twig/Environment.php(289): Twig_Environment->loadTemplate('hello.html.twig')
#2 /var/task/vendor/symfony/framework-bundle/Controller/ControllerTrait.php(224): Twig_Environment->render('hello.html.twig', Array)
#3 /var/task/src/Controller/HomeController.php(12): Symfony\Bundle\FrameworkBundle\Controller\Controller->render('hello.html.twig')
#4 /var/task/vendor/symfony/http-kernel/HttpKernel.php(149): App\Controller\HomeController->index()
#5 /var/task/vendor/symfony/http-kernel/HttpKernel.php(66): Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object(Symfony\Component\HttpFoundation\Request), 1)
#6 /var/task/vendor/symfony/http-kernel/Kernel.php(188): Symfony\Compone in /var/task/vendor/twig/twig/lib/Twig/Cache/Filesystem.php on line 59

from bref.

nealio82 avatar nealio82 commented on August 16, 2024 1

Workaround documentation added in #42

from bref.

mnapoli avatar mnapoli commented on August 16, 2024 1

I'll be closing this issue since it was opened in 2019, and since then we have the Symfony bridge that should take care of the cache πŸŽ‰

from bref.

mnapoli avatar mnapoli commented on August 16, 2024

Regarding the memory size problem, you could try increasing the memory limit of PHP by customizing the php.ini config, see https://github.com/mnapoli/bref#php-configuration to see how to set php.ini flags. However 128Mb seems already quite much, I'm surprised Symfony would consume that much.

Presumably setting the cache dir to Lambda's local /tmp kind-of defeats the point pre-warming the cache with the hooks in the .bref.yml file? The object is to avoid launching the application without the cache already in place, right?

Agreed, this is not really efficient in production that's why it's better to pre-generate the cache.

I get the same behaviour with cloning the mnapoli/bref-symfony-demo repo and running bref deploy

This is surprising, but since I wrote (and tested) bref-symfony-demo we have added some extensions to PHP (mainly opcache, the other extensions are disabled by default), maybe that could be it? I don't really know much more right now I'll try to dig in.

from bref.

nealio82 avatar nealio82 commented on August 16, 2024

The first thing I tried was to increase the memory_limit (I should have mentioned that above, sorry!)

I think the cache is not being pre-warmed during the deploy process, because presumably it should exist and be readable if it has been!?

As mentioned, if you set the cache dir to the writeable /tmp/ dir then the problem disappears and PHP doesn't complain about memory any more

from bref.

mnapoli avatar mnapoli commented on August 16, 2024

OK, I'll try to reproduce that tonight and see how it goes. Thanks for opening the issue!

from bref.

t-geindre avatar t-geindre commented on August 16, 2024

We use our own kernel here, but since we also use symfony/dependency-injection we had to find a solution to pre-warm the cache, which essentially means to build and dump the container.

The only way we found to do so is to use relative paths.

For instance, we have a parameter %cache_dir%. Its value is always ./var/cache. This allows us to dump the container before uploading it on lambda.

Our kernel always check that the current working directory is $_ENV['LAMBDA_TASK_ROOT'].

This works and we don't build the container on each execution. I think you could do something similar with the Symfony kernel.

That beeing said, if you have a better solution, I would be very interested.

from bref.

mnapoli avatar mnapoli commented on August 16, 2024

@nealio82 I've updated (composer update) everything to the latest versions on the symfony-demo project and redeployed, it still works: https://7oaryq3rzl.execute-api.eu-west-3.amazonaws.com/dev I don't understand what breaks for you πŸ€”

@t-geindre Yes! I had issues with absolute/relative paths when deploying the API platform demo for my benchmarks. In the end I gave up and stored the cache in /tmp because I was not measuring the cold starts, so I just had to warmup the application and that's it. But this is not ideal at all.

from bref.

nealio82 avatar nealio82 commented on August 16, 2024

from bref.

nealio82 avatar nealio82 commented on August 16, 2024

Interestingly, if I clone the mnapoli/bref-symfony-demo repo, I get a different (and nicer) exception message about the cache.

The symfony/website-skeleton still gives me out of memory, but the SF demo now specifically mentions the cache.

I made a screencast here: https://youtu.be/Ar3USl8h8Ug

from bref.

mnapoli avatar mnapoli commented on August 16, 2024

Thanks for the screencast! Could you try one last time mnapoli/bref-symfony-demo by running composer update before bref deploy to install the last version of Bref?

from bref.

nealio82 avatar nealio82 commented on August 16, 2024

If I add the cache param to config/packages/twig.yml with the value of either /tmp/... or false, then everything works again.

twig:
    paths: ['%kernel.project_dir%/templates']
    debug: '%kernel.debug%'
    strict_variables: '%kernel.debug%'
    cache: '/tmp/twig'

see https://twig.symfony.com/doc/2.x/api.html#environment-options and http://symfony.com/doc/current/reference/configuration/twig.html#cache

However, I don't like setting /tmp/... in the twig config, and I don't know enough about Twig to tell if disabling the cache is a bad idea.

I also don't get why Twig needs to write to the cache if the templates are pre-compiled!?

from bref.

mnapoli avatar mnapoli commented on August 16, 2024

OK I think I remember now where the problem came from (when I tried deploying the API platform demo).

Some things in the Symfony cache are using absolute paths, as @t-geindre mentioned. You need to tweak Symfony and its config to force it to use relative paths (because the paths on your machine and on AWS lambda are different).

That works, except for Twig that IIRC uses realpath() as some point in the code (I think it's here). Because of that call, a relative path is turned into an absolute path and it breaks things. I don't remember the details anymore though.


Also it is important to note that not all the Symfony cache is pregenerated, for example you have to put Doctrine's cache (which is not pregenerated) in /tmp manually.

Here is an example of what I did for Doctrine in config/packages/doctrine.yaml:

parameters:
    writable_cache_dir: '/tmp/cache'

[…]

doctrine:
    orm:
        […]
        metadata_cache_driver:
            cache_provider: metadata_cache
        result_cache_driver:
            cache_provider: result_cache
        query_cache_driver:
            cache_provider: query_cache

[…]

doctrine_cache:
    providers:
        metadata_cache:
            aliases: [doctrine.orm.default_metadata_cache]
            file_system:
                directory: "%writable_cache_dir%/doctrine/metadata"
        result_cache:
            aliases: [doctrine.orm.default_result_cache]
            file_system:
                directory: "%writable_cache_dir%/doctrine/result"
        query_cache:
            aliases: [doctrine.orm.default_query_cache]
            file_system:
                directory: "%writable_cache_dir%/doctrine/query"

I did the same for the user cache:

framework:
    cache:
        directory: "%writable_cache_dir%/app"

from bref.

nealio82 avatar nealio82 commented on August 16, 2024

Ok, so I assume that means at some point we have to have a writeable filesystem in order for Twig to work properly?

However, that raises 2 questions:

  1. Should the Kernel.php in the documentation be updated to change the cache dir to /tmp to make sure Twig can always write to it?
  2. Why does your example work for you and not for me? You saw in the screencast that I made no adjustments to the example repo's code!

from bref.

mnapoli avatar mnapoli commented on August 16, 2024

Should the Kernel.php in the documentation be updated to change the cache dir to /tmp to make sure Twig can always write to it?

That could be a temporary solution. I still hope to find the correct configuration to apply to solve that (or maybe fix the problem in Twig/Symfony). But yeah it's better to have helpful documentation for now.

Why does your example work for you and not for me? You saw in the screencast that I made no adjustments to the example repo's code!

Yes I don't know how to explain that either :/

from bref.

nealio82 avatar nealio82 commented on August 16, 2024

from bref.

mnapoli avatar mnapoli commented on August 16, 2024

Does anyone here has a public repository to reproduce this?

from bref.

nealio82 avatar nealio82 commented on August 16, 2024

from bref.

mnapoli avatar mnapoli commented on August 16, 2024

For those reading this issue, I reproduced it when writing a simple Symfony 4 application, starting with Flex. I just have a base template (the layout) and a few pages that extend from that. I don't have time right now to create a whole repository to reproduce that but that can maybe be a starting point for others.

from bref.

nealio82 avatar nealio82 commented on August 16, 2024

Weirdly I tried yesterday; starting with Symfony 4 & extending a base template, then passing in some variables to render. I didn’t see the issue at all. (And i’m pretty sure the first time I saw the issue I didn’t even need to extend a base template)

from bref.

mnapoli avatar mnapoli commented on August 16, 2024

Thanks that's a much better solution! I'll try to update the documentation in the coming weeks (I should have more time than this month), if anyone wants to get on this though feel free.

from bref.

nealio82 avatar nealio82 commented on August 16, 2024

@thibaudlemaire by any chance do you know why it adds a 4x overhead? Have you looked into that with Blackfire, for example? The Symfony best practice is to override it in the Kernel class: https://symfony.com/doc/current/configuration/override_dir_structure.html#override-the-cache-directory

from bref.

thibaudlemaire avatar thibaudlemaire commented on August 16, 2024

When you override the cache directory, pre-warmed cache cannot be used because either it's warmed up in your local /tmp dir that is not packed by sam package command, or during Symfony execution it tries to retrieve cache from the lambda container's /tmp directory that is empty on first request. I think we cannot apply blindly what's explained in the link you gave.

The 4x overhead is due to the cache warmup when a Lambda container is created. I don't know exactly what is compiled at the first execution but I suppose : Doctrine proxies, annotations, Twig views, Kernel config, services container, etc.

That's why we need to make Symfony use the pre-warmed cache. To do so, I thought about two solutions :

  • Use /tmp as cache directory and copy (or symlink) pre-warmed cache files on lambda container initialisation. I don't know exactly how to do it. A solution may be to add this feature to Bref's event handler to make it copy (or symlink) cache on first invoke.
  • Or use the default Symfony cache directory (./var/cache), warm it up before deployment so that sam package pack cache too. This should be possible for all bundles because Symfony recommends not to write in the cache directory after deployment. For bundles that are not compliant with this best practice (like Twig), we still can override the cache directory to make them write in /tmp.

I use the second one because it's easier. But it's not ideal because twig views still need to be compiled on first invoke.

from bref.

nealio82 avatar nealio82 commented on August 16, 2024

When you override the cache directory, pre-warmed cache cannot be used because either it's warmed up in your local /tmp dir that is not packed by sam package command, or during Symfony execution it tries to retrieve cache from the lambda container's /tmp directory that is empty on first request. I think we cannot apply blindly what's explained in the link you gave.

Ah, of course! πŸ€¦β€β™‚

from bref.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.