AkkaSystem unclean termination

[Zetanova]

AkkaSystem unclean termination

[Arkatufus]

I can't really reproduce the bug, maybe there are other things that causes the actor system to fail?

[Aaronontheweb]

Terminate and/or CoordinatedShutdown a system in OOM situation.

So I missed this - but this is largely an unhandleable situation and CoordinatedShutdown won't run correctly because processes are aborted when this occurs. Catastrophic runtime failures can't be handled gracefully through the normal pathways we use to handle graceful terminations. The solve here is to fix the OOM.

[Aaronontheweb]

See https://learn.microsoft.com/en-us/dotnet/api/system.outofmemoryexception?view=net-7.0 for a fuller explanation on what you can do to log this type of error (Environment.FailFast), but there are no tools to handle it once it gets going.

[Zetanova]

Its not that OOM exception should be handled explicitly,
but system.WhenTerminated should be completed even after an exception in system.Terminate() itself.
The callbacks registered in System.RegisterOnTermination(StopApplication) could/should be executed even after an exception in CoordinatedShutdown

If not system.Terminate() and/or CoordinatedShutdown will break the system state and system.WhenTerminated never completes and no System.RegisterOnTermination(StopApplication) callbacks get executed.

My end result was that my LiveHealthCheck on system.WhenTerminated was successful
but the ReadyHealthCheck on the akka cluster was in a failure state. (Kubernetes)

[Zetanova]

In my opinion system.Terminate() should have the same behavior as Dispose() that even when it internaly throws the end result of the instance is disposed

[Aaronontheweb]

but system.WhenTerminated should be completed even after an exception in system.Terminate() itself.
The callbacks registered in System.RegisterOnTermination(StopApplication) could/should be executed even after an exception in CoordinatedShutdown

Ok, that is fixable. We can do that.

[Arkatufus]

@Zetanova can you look at #6967 and check if I missed anything, I could not reproduce your error.

[Aaronontheweb]

Any updates on this @Zetanova ?