A possible issue on systems that are under heavy load with timeout errors in the distributed cache

[ikyriak]

A possible issue on systems that are under heavy load with timeout errors in the distributed cache

[ikyriak]

Hello @william-keller,

The related improvement has been included in the 1.0.0 RC.01 release.

Please read the README.md and CHANGELOG.md files because this release has breaking changes.

You can find the 1.0.0 RC.01 NuGet packages here.

If you have any suggestions or comments please, do not hesitate to send them 😉

[william-keller]

Hi @ikyriak ... how are you doing?

We've finally solved this problem by adding this code in our startup.cs file:

ThreadPool.SetMinThreads(100, 100);

It was probaly a thread poll issue

Many Thanks

[william-keller]

Hi @ikyriak
We found a scenario when this error occurs more frequently

Try to run locally (2 or more) docker containers with the same application/controller, such as a scalable cloud enviroment
Then, try to send 2 post requests with the same idempotencykey many times

[ikyriak]

Hello @william-keller

The standard IDistributedCache interface doesn't support an atomic command to GetOrSet a cached value. For that reason, the current implementation cannon prevent multiple concurrent requests and guarantee that only one will be executed. So, they will be both executed in a scenario of two concurrent requests with the same idempotency key.

Additionally, there is the scenario of a Distributed Cache failure (e.g., timeout) when setting the controller action's result. In such a case, we would need a retry mechanism on the server instance (that has the result) and reject the subsequent requests (in the instances that don't have the result). So, we should use a local memory cache.

The good news is that we could use the FusionCache implementation, which supports both the GetOrSet method and several fail-safe mechanisms. I have started writing the tests for these scenarios, and then I will try to use the FusionCache.

[ikyriak]

Hello @william-keller,

That's very interesting.
I am happy that you solved it and informed us about your solution.

Have a nice day,