Navigating in the categories tree leads to errors in Redis: "Redis session exceeded concurrent connections"

[netzkollektiv]

Navigating in the categories tree leads to errors in Redis: "Redis session exceeded concurrent connections"

[hostep]

Thanks for shedding a bit of light onto the problem!

There is no need to wait on confirmation, the issue was already confirmed by many people in this thread. To repeat: you shouldn't try to reproduce the Redis error. You just need to have a big category tree, with many levels, then open the inspector in your browser and see the hundreds of simultaneous ajax requests going to catalog/category/categoriesJson. This is bad design and needs to be reworked to only a single ajax request if possible.

[Atul-glo35265]

Hi @netzkollektiv @hostep
Thanks for your contribution and finding the issue.

This particular code is responsible for getting the children of categories at level 3 or further. If we comment out that foreach loop, the triangle button will become invisible and the user won't be able to traverse to categories at level 4. Please refer attached screenshot.

We need to check the feasibility of optimizing the code for this issue. Once the issue is confirmed from our internal community team, we will investigate it further.

[engcom-Delta]

Hi @netzkollektiv ,

Verified the issue on 2.4-develop and 2.4.7 and we are not able to reproduce this issue.

Preconditions and environment

1.Redis enabled

Steps to reproduce:-

1.Go to the backend
2.Navigate to Manage Categories
3.Open some child-nodes in categories tree by clicking on the arrow next to the category in the tree
4.Watch the logs and network logs

Redis enabled

Admin

Manage Categories

Open some child-nodes in categories tree

logs

Please refer the attached screenshots and let us know if we missed anything.

[netzkollektiv]

Add categories until you have a huge category tree with many subordinate categories

You need a lot of categories to reproduce the issue, not just 4. See also comment of @hostep

[hostep]

Indeed, that's a workaround for the Redis problem, it won't help with hosting companies that have rate limiters setup though.

The refactoring to the category tree that was done in Magento 2.4.7 should be taken a close look at and refactored again so it doesn't require hundreds of ajax calls to get the necessary info.

When I first looked at this issue, removing the foreach here also solved the issue, but I have no idea if this has repercussions...

[engcom-Bravo]

Hi @hostep,

Thanks for your update.

We have verified the issue in Latest 2.4-develop instance and the issue is reproducible.Kindly refer the screenshots.

Hence Confirming the issue.

Thanks.

[m2-assistant]

Hi @netzkollektiv. Thank you for your report.
To speed up processing of this issue, make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, Add a comment to the issue:

• @magento give me 2.4-develop instance - upcoming 2.4.x release
• For more details, review the Magento Contributor Assistant documentation.
• Add a comment to assign the issue: @magento I am working on this
• To learn more about issue processing workflow, refer to the Code Contributions.

---

Join Magento Community Engineering Slack and ask your questions in #github channel.
⚠️ According to the Magento Contribution requirements, all issues must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting.
🕙 You can find the schedule on the Magento Community Calendar page.
📞 The triage of issues happens in the queue order. If you want to speed up the delivery of your contribution, join the Community Contributions Triage session to discuss the appropriate ticket.

[m2-assistant]

Hi @engcom-Delta. Thank you for working on this issue.
In order to make sure that issue has enough information and ready for development, please read and check the following instruction: 👇

• 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).
• 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue.
• 3. Add Area: XXXXX label to the ticket, indicating the functional areas it may be related to.
• 4. Verify that the issue is reproducible on 2.4-develop branch
  Details

  - Add the comment @magento give me 2.4-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.4-develop branch, please, add the label Reproduced on 2.4.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!
• 5. Add label Issue: Confirmed once verification is complete.
• 6. Make sure that automatic system confirms that report has been added to the backlog.

[hostep]

Also running into this problem over here. In our case it not only happens when clicking the arrow, but also when just clicking a deeper level of category itself, in that case it's even worse, we sometimes see > 70 similar ajax requests happening simultaneous of which a lot are failing.
One of our shops with this problem is hosted on hypernode.com and we notice we get a bunch of HTTP status codes 429 back, because their rate limiter sees way too many requests happening at the same time from the same source IP.

I think the problem got introduced by 67f1b8f which was released in Magento 2.4.7 and higher, more specifically in this piece of code.

I'm not exactly sure what this code is supposed to do, but it sounds wrong to me to perform tens if not hundreds of similar ajax requests for info when you can try to get the necessary info as well within a single request.

Maybe @Atul-glo35265 is around to try to explain how that code is supposed to work and if there isn't a way to optimise it so it doesn't send all those ajax requests in a loop, but bundle them up in a single request somehow?
/cc @kandy

[kandy]

@hostep Are you sure that this issue is not reproducible on version < 2.4.7?
My assumption that this issue caused by switching from http1 to http2 that allow up to 100 parallel request to server

Usually BFF (like graphql) responsibly for batching requests, so implementing batching like /tree/child/?ids=[1,2,3,4] is optimal for now

[lamskoy]

This happens in 2.4.7 and 2.4.7-p1 for me
Lot of XHR requests coming to PageBuilder previews

Stack trace:

[2024-06-25T15:32:16.627508+00:00] main.CRITICAL: Cm\RedisSession\ConcurrentConnectionsExceededException in /home/www/vhosts/magento/vendor/colinmollenhour/php-redis-session-abstract/src/Cm/RedisSession/Handler.php:531
Stack trace:
#0 /home/www/vhosts/magento/vendor/magento/framework/Session/SaveHandler/Redis.php(84): Cm\RedisSession\Handler->read()
#1 /home/www/vhosts/magento/vendor/magento/framework/Session/SaveHandler.php(213): Magento\Framework\Session\SaveHandler\Redis->read()
#2 /home/www/vhosts/magento/vendor/magento/framework/Session/SaveHandler.php(126): Magento\Framework\Session\SaveHandler->callSafely()
#3 [internal function]: Magento\Framework\Session\SaveHandler->read()
#4 /home/www/vhosts/magento/vendor/magento/framework/Session/SessionManager.php(187): session_start()
#5 /home/www/vhosts/magento/generated/code/Magento/Backend/Model/Auth/Session/Interceptor.php(32): Magento\Framework\Session\SessionManager->start()
#6 /home/www/vhosts/magento/vendor/magento/framework/Session/SessionManager.php(124): Magento\Backend\Model\Auth\Session\Interceptor->start()
#7 /home/www/vhosts/magento/vendor/magento/module-backend/Model/Auth/Session.php(104): Magento\Framework\Session\SessionManager->__construct()
#8 /home/www/vhosts/magento/generated/code/Magento/Backend/Model/Auth/Session/Interceptor.php(14): Magento\Backend\Model\Auth\Session->__construct()
#9 /home/www/vhosts/magento/vendor/magento/framework/ObjectManager/Factory/AbstractFactory.php(121): Magento\Backend\Model\Auth\Session\Interceptor->__construct()
#10 /home/www/vhosts/magento/vendor/magento/framework/ObjectManager/Factory/Compiled.php(108): Magento\Framework\ObjectManager\Factory\AbstractFactory->createObject()
#11 /home/www/vhosts/magento/vendor/magento/framework/ObjectManager/Factory/Compiled.php(150): Magento\Framework\ObjectManager\Factory\Compiled->create()
#12 /home/www/vhosts/magento/vendor/magento/framework/ObjectManager/Factory/Compiled.php(79): Magento\Framework\ObjectManager\Factory\Compiled->get()
#13 /home/www/vhosts/magento/vendor/magento/framework/ObjectManager/Factory/Compiled.php(150): Magento\Framework\ObjectManager\Factory\Compiled->create()
#14 /home/www/vhosts/magento/vendor/magento/framework/ObjectManager/Factory/Compiled.php(79): Magento\Framework\ObjectManager\Factory\Compiled->get()
#15 /home/www/vhosts/magento/vendor/magento/framework/ObjectManager/Factory/Compiled.php(150): Magento\Framework\ObjectManager\Factory\Compiled->create()
#16 /home/www/vhosts/magento/vendor/magento/framework/ObjectManager/Factory/Compiled.php(125): Magento\Framework\ObjectManager\Factory\Compiled->get()
#17 /home/www/vhosts/magento/vendor/magento/framework/ObjectManager/Factory/Compiled.php(86): Magento\Framework\ObjectManager\Factory\Compiled->parseArray()
#18 /home/www/vhosts/magento/vendor/magento/framework/ObjectManager/ObjectManager.php(73): Magento\Framework\ObjectManager\Factory\Compiled->create()
#19 /home/www/vhosts/magento/vendor/magento/framework/App/FrontController.php(109): Magento\Framework\ObjectManager\ObjectManager->get()
#20 /home/www/vhosts/magento/generated/code/Magento/Framework/App/FrontController/Interceptor.php(14): Magento\Framework\App\FrontController->__construct()
#21 /home/www/vhosts/magento/vendor/magento/framework/ObjectManager/Factory/AbstractFactory.php(121): Magento\Framework\App\FrontController\Interceptor->__construct()
#22 /home/www/vhosts/magento/vendor/magento/framework/ObjectManager/Factory/Compiled.php(108): Magento\Framework\ObjectManager\Factory\AbstractFactory->createObject()
#23 /home/www/vhosts/magento/vendor/magento/framework/ObjectManager/ObjectManager.php(73): Magento\Framework\ObjectManager\Factory\Compiled->create()
#24 /home/www/vhosts/magento/vendor/magento/framework/App/Http.php(115): Magento\Framework\ObjectManager\ObjectManager->get()
#25 /home/www/vhosts/magento/vendor/magento/framework/Interception/Interceptor.php(58): Magento\Framework\App\Http->launch()
#26 /home/www/vhosts/magento/vendor/magento/framework/Interception/Interceptor.php(138): Magento\Framework\App\Http\Interceptor->___callParent()
#27 /home/www/vhosts/magento/vendor/magento/module-application-performance-monitor/Plugin/ApplicationPerformanceMonitor.php(38): Magento\Framework\App\Http\Interceptor->Magento\Framework\Interception\{closure}()
#28 /home/www/vhosts/magento/vendor/magento/framework/Interception/Interceptor.php(135): Magento\ApplicationPerformanceMonitor\Plugin\ApplicationPerformanceMonitor->aroundLaunch()
#29 /home/www/vhosts/magento/vendor/magento/framework/Interception/Interceptor.php(153): Magento\Framework\App\Http\Interceptor->Magento\Framework\Interception\{closure}()
#30 /home/www/vhosts/magento/generated/code/Magento/Framework/App/Http/Interceptor.php(23): Magento\Framework\App\Http\Interceptor->___callPlugins()
#31 /home/www/vhosts/magento/vendor/magento/framework/App/Bootstrap.php(264): Magento\Framework\App\Http\Interceptor->launch()
#32 /home/www/vhosts/magento/pub/index.php(30): Magento\Framework\App\Bootstrap->run()
#33 {main}

Next Magento\Framework\Exception\LocalizedException: Redis session exceeded concurrent connections in /home/www/vhosts/magento/vendor/magento/framework/Session/SaveHandler/Redis.php:86
Stack trace:
#0 /home/www/vhosts/magento/vendor/magento/framework/Session/SaveHandler.php(213): Magento\Framework\Session\SaveHandler\Redis->read()
#1 /home/www/vhosts/magento/vendor/magento/framework/Session/SaveHandler.php(126): Magento\Framework\Session\SaveHandler->callSafely()
#2 [internal function]: Magento\Framework\Session\SaveHandler->read()
#3 /home/www/vhosts/magento/vendor/magento/framework/Session/SessionManager.php(187): session_start()
#4 /home/www/vhosts/magento/generated/code/Magento/Backend/Model/Auth/Session/Interceptor.php(32): Magento\Framework\Session\SessionManager->start()
#5 /home/www/vhosts/magento/vendor/magento/framework/Session/SessionManager.php(124): Magento\Backend\Model\Auth\Session\Interceptor->start()
#6 /home/www/vhosts/magento/vendor/magento/module-backend/Model/Auth/Session.php(104): Magento\Framework\Session\SessionManager->__construct()
#7 /home/www/vhosts/magento/generated/code/Magento/Backend/Model/Auth/Session/Interceptor.php(14): Magento\Backend\Model\Auth\Session->__construct()
#8 /home/www/vhosts/magento/vendor/magento/framework/ObjectManager/Factory/AbstractFactory.php(121): Magento\Backend\Model\Auth\Session\Interceptor->__construct()
#9 /home/www/vhosts/magento/vendor/magento/framework/ObjectManager/Factory/Compiled.php(108): Magento\Framework\ObjectManager\Factory\AbstractFactory->createObject()
#10 /home/www/vhosts/magento/vendor/magento/framework/ObjectManager/Factory/Compiled.php(150): Magento\Framework\ObjectManager\Factory\Compiled->create()
#11 /home/www/vhosts/magento/vendor/magento/framework/ObjectManager/Factory/Compiled.php(79): Magento\Framework\ObjectManager\Factory\Compiled->get()
#12 /home/www/vhosts/magento/vendor/magento/framework/ObjectManager/Factory/Compiled.php(150): Magento\Framework\ObjectManager\Factory\Compiled->create()
#13 /home/www/vhosts/magento/vendor/magento/framework/ObjectManager/Factory/Compiled.php(79): Magento\Framework\ObjectManager\Factory\Compiled->get()
#14 /home/www/vhosts/magento/vendor/magento/framework/ObjectManager/Factory/Compiled.php(150): Magento\Framework\ObjectManager\Factory\Compiled->create()
#15 /home/www/vhosts/magento/vendor/magento/framework/ObjectManager/Factory/Compiled.php(125): Magento\Framework\ObjectManager\Factory\Compiled->get()
#16 /home/www/vhosts/magento/vendor/magento/framework/ObjectManager/Factory/Compiled.php(86): Magento\Framework\ObjectManager\Factory\Compiled->parseArray()
#17 /home/www/vhosts/magento/vendor/magento/framework/ObjectManager/ObjectManager.php(73): Magento\Framework\ObjectManager\Factory\Compiled->create()
#18 /home/www/vhosts/magento/vendor/magento/framework/App/FrontController.php(109): Magento\Framework\ObjectManager\ObjectManager->get()
#19 /home/www/vhosts/magento/generated/code/Magento/Framework/App/FrontController/Interceptor.php(14): Magento\Framework\App\FrontController->__construct()
#20 /home/www/vhosts/magento/vendor/magento/framework/ObjectManager/Factory/AbstractFactory.php(121): Magento\Framework\App\FrontController\Interceptor->__construct()
#21 /home/www/vhosts/magento/vendor/magento/framework/ObjectManager/Factory/Compiled.php(108): Magento\Framework\ObjectManager\Factory\AbstractFactory->createObject()
#22 /home/www/vhosts/magento/vendor/magento/framework/ObjectManager/ObjectManager.php(73): Magento\Framework\ObjectManager\Factory\Compiled->create()
#23 /home/www/vhosts/magento/vendor/magento/framework/App/Http.php(115): Magento\Framework\ObjectManager\ObjectManager->get()
#24 /home/www/vhosts/magento/vendor/magento/framework/Interception/Interceptor.php(58): Magento\Framework\App\Http->launch()
#25 /home/www/vhosts/magento/vendor/magento/framework/Interception/Interceptor.php(138): Magento\Framework\App\Http\Interceptor->___callParent()
#26 /home/www/vhosts/magento/vendor/magento/module-application-performance-monitor/Plugin/ApplicationPerformanceMonitor.php(38): Magento\Framework\App\Http\Interceptor->Magento\Framework\Interception\{closure}()
#27 /home/www/vhosts/magento/vendor/magento/framework/Interception/Interceptor.php(135): Magento\ApplicationPerformanceMonitor\Plugin\ApplicationPerformanceMonitor->aroundLaunch()
#28 /home/www/vhosts/magento/vendor/magento/framework/Interception/Interceptor.php(153): Magento\Framework\App\Http\Interceptor->Magento\Framework\Interception\{closure}()
#29 /home/www/vhosts/magento/generated/code/Magento/Framework/App/Http/Interceptor.php(23): Magento\Framework\App\Http\Interceptor->___callPlugins()
#30 /home/www/vhosts/magento/vendor/magento/framework/App/Bootstrap.php(264): Magento\Framework\App\Http\Interceptor->launch()
#31 /home/www/vhosts/magento/pub/index.php(30): Magento\Framework\App\Bootstrap->run()
#32 {main} {"report_id":"054a95716720fcc93be4f07fb1d91d9ae7089fd62f426b0feb7ebf9aca5fd7ee","exception":"[object] (Magento\\Framework\\Exception\\LocalizedException(code: 0): Redis session exceeded concurrent connections at /home/www/vhosts/magento/vendor/magento/framework/Session/SaveHandler/Redis.php:86)
[previous exception] [object] (Cm\\RedisSession\\ConcurrentConnectionsExceededException(code: 0):  at /home/www/vhosts/magento/vendor/colinmollenhour/php-redis-session-abstract/src/Cm/RedisSession/Handler.php:531)"} []

Catalog has about 3k of categories and single store in my case
Redis (almost default) config from app/etc/env.php:

'session' => [
        'save' => 'redis',
        'redis' => [
            'host' => 'localhost',
            'port' => '6379',
            'password' => '',
            'timeout' => '2.5',
            'persistent_identifier' => '',
            'database' => '3',
            'compression_threshold' => '2048',
            'compression_library' => 'gzip',
            'log_level' => '4',
            'max_concurrency' => '6',
            'break_after_frontend' => '5',
            'break_after_adminhtml' => '30',
            'first_lifetime' => '600',
            'bot_first_lifetime' => '60',
            'bot_lifetime' => '7200',
            'disable_locking' => '0',
            'min_lifetime' => '60',
            'max_lifetime' => '2592000',
            'sentinel_master' => '',
            'sentinel_servers' => '',
            'sentinel_connect_retries' => '5',
            'sentinel_verify_master' => '0'
        ]
    ],

Doubt that setting bigger value for max_concurrency is a good idea, but can be a workaround

[hostep]

@hostep Are you sure that this issue is not reproducible on version < 2.4.7?
My assumption that this issue caused by switching from http1 to http2 that allow up to 100 parallel request to server
Usually BFF (like graphql) responsibly for batching requests, so implementing batching like /tree/child/?ids=[1,2,3,4] is optimal for now

I can double check with 2.4.6, but since the category tree code was refactored in 2.4.7, I don't think it will trigger on anything lower than 2.4.7.
Simultaneous request with HTTP/2 is good for dealing with static assets by the browser (images, scripts, css, ...). So one connection that fetches the html (outputted by a single php process) and then fetching all the other assets simultaneously over 1 and the same TCP connection (which have nothing to do with php usually).
But in this case, you are sending tens if not hundreds of ajax requests (almost) simultaneously, which the server than has to start processing with php, and validating the sessions for each request to make sure you are authenticated and then fetching, per tiny request, the little info it needs. This seems not a good behavior in my opinion, way too much overhead, way too many php processes that are doing things. Just fetch the info you need with one single ajax request and deal with the big response in the frontend js code.

As far as batching goes, I don't think that happens here at the moment. If you indeed would batch like 50 things in a single request, that should significantly improve the situation.

@lamskoy: could you open a new issue for your problem? It may sound like the same issue at a glance, but it definitely isn't the same problem (unless those preview calls are generated by the ajax calls discussed in this issue?).

[lamskoy]

Add categories until you have a huge category tree with many subordinate categories

You need a lot of categories to reproduce the issue, not just 4. See also comment of @hostep

I have around 2700 categories in tree. This happens

[lucasjor]

I'm having the same issue, same error message. Strangely it is not being consistent along all our environments, even with all of them having the same Redis configuration.

[hostep]

The issue is not the Redis configuration, the issue is those hundreds (depending on the amount of categories) of concurrent calls to that categoriesJson url, @kandy I hope you understand it better by now?

[LinkImaging]

Also, seeing this issue. As usual, Magento fails to test at scale. This is probably the 20th time I've seen an issue triggered when there are thousands of records and Magento says "We tested with 2 records and it's fine".

Why shouldn't everyone here just move to OroCommerce, BigCommerce, Shopify, Webflow, or one of the other platforms? Magento2 has drastically reduced in quality over the last 5 years.

[engcom-Delta]

Hi @netzkollektiv ,

Verified the issue on 2.4.7 again.

Preconditions

1.Redis enabled

Steps to reproduce:-

1.Go to the backend
2.Navigate to Manage Categories
3.Add categories until you have a huge category tree with many subordinate categories
4.Open some child-nodes in categories tree by clicking on the arrow next to the category in the tree
5.Watch the logs and network logs

Actual Result:

We are getting "Redis session exceeded concurrent connections" error and we can see many XHR requests coming to PageBuilder previews.

admin panel

"Redis session exceeded concurrent connections" error on logs

network logs

Could you please let us know for network logs , what kind of error you have reported as it is not mentioned in the issue above.

[nola-adrien]

Hello,
I have the same issue with another customer with a "big" category tree.

When we want to open the sub-categories there are more than 450 ajax requests to /admin/catalog/category/categoriesJson
Lot of them get a 500 error : "Redis session exceeded concurrent connections"

Why is this issue status is "needs update" ? The issue seems pretty clear to me.

Has someone a temporary fix ?

[lamskoy]

Has someone a temporary fix ?

increase max_concurrency value inside app/etc/env.php. Default value is 6

'session' => [
        'save' => 'redis',
        'redis' => [
            'host' => 'localhost',
            'port' => '6379',
       ...
            'max_concurrency' => '6',
       ...
        ]
    ],

Bad practice, but could help

[github-jira-sync-bot]

✅ Jira issue https://jira.corp.adobe.com/browse/AC-12571 is successfully created for this GitHub issue.

[m2-assistant]

✅ Confirmed by @engcom-Bravo. Thank you for verifying the issue.
Issue Available: @engcom-Bravo, You will be automatically unassigned. Contributors/Maintainers can claim this issue to continue. To reclaim and continue work, reassign the ticket to yourself.

[hostep]

Potential fix: #39018, thanks to @sheepfy! (I haven't tested it yet)

[engcom-Bravo]

Hello,

Internal team has started to work on it

Thanks