[FEATURE REQUEST] Graceful Shutdown

[temporalio]

[FEATURE REQUEST] Graceful Shutdown

[rustatian]

@changwuf31 When it's merged spiral-modules/roadrunner-binary#11 you will have the ability to configure the grace period in seconds, minutes, and hours. Additionally, like in the Kubernetes, when you sent SIGINT or SIGTERM to the RR2 it will activate the grace period and let RR2 finish gracefully. During the graceful shutdown, if RR2 receive a second signal, it will be forcibly closed with a non-zero error code.

[wolfy-j]

Yes, it does. The plugin will attempt to complete in-flight activities before shutting down. However, under load, some workflow tasks might miss the worker, which is fine since the server will handle it properly.

[wolfy-j]

Graceful shutdown use similar behavior as hot reload command rr reset.

[changwuf31]

so, does that mean, we can use this on kubernetes:

lifecycle:
  preStop:
    exec:
      command: [
        # Gracefully shutdown roadrunner temporal
        "rr", "reset"
      ]

[wolfy-j]

You can send SIGTERM or SIGINT to RR to ask for the shutdown. Reset will recreate the worker pool. If this is not convenient we can expose the "stop" command.

[changwuf31]

On nginx case

              # SIGTERM triggers a quick exit; gracefully terminate instead
              command: ["/usr/sbin/nginx","-s","quit"]

Ref: https://ubuntu.com/blog/avoiding-dropped-connections-in-nginx-containers-with-stopsignal-sigquit

If SIGTERM on roadrunner is a graceful terminate, then there is no need for a "stop" command,
Does sending SIGTERM to roadrunner, will make the temporal workers:

• stop asking for new jobs
• process the current running jobs
• and stop gracefully
  ?

Btw, Is there a local job queue ?

[rustatian]

Hey @changwuf31 . Thanks for the question :)

1. RR2 (via the Endure container) does not support the graceful shutdown at the moment. When you send SIGTERM/SIGINT signal to the RR2 it ask the container to start the Stop routine. Container topologically asks vertices (plugins) to Stop. If some vertex did not respond during the configurable (10s by default) time, Endure will show the message with such vertices and exit from the root RR2 process which will lead to an exit from all child processes.
2. It will be easy to implement, but could you please file us a ticket to the main RR2 repository https://github.com/spiral/roadrunner/issues ?

[rustatian]

ref: roadrunner-server/roadrunner#555

[changwuf31]

1. RR2 (via the [Endure](https://github.com/spiral/endure) container) does not support the graceful shutdown at the moment. When you send `SIGTERM/SIGINT` signal to the RR2 it ask the container to start the `Stop` routine. Container topologically asks vertices (plugins) to Stop. If some vertex did not respond during the configurable (10s by default) time, `Endure` will show the message with such vertices and exit from the root RR2 process which will lead to an exit from all child processes.

If so, is it correct if I say, we just need to call endure.SetStopTimeOut(600 * time.Second) on roadrunner-binary ?

[rustatian]

@changwuf31 Yep, but we will provide graceful shutdown later as well.