NGINX started multiple times after certificate renewal check

[Nyffeneggerr]

NGINX started multiple times after certificate renewal check

[JonasAlfredsson]

Reading some more configs it might be worth to try setting worker_shutdown_timeout, it seems to give the workers a hard deadline before they are killed. It will cause some minor hiccups for the connected clients, but if they handle reconnects properly it should not be an issue.

[JonasAlfredsson]

Nginx has one "main" process, and then one worker per CPU (if set to automatic).

This is how it looks on my 8 core machine:

ps fax | grep nginx

    1 pts/0    Ss+    0:00 /bin/bash /scripts/start_nginx_certbot.sh
   38 pts/0    S+     0:00 nginx: master process nginx -g daemon off;
  411 pts/0    S+     0:00  \_ nginx: worker process
  412 pts/0    S+     0:00  \_ nginx: worker process
  413 pts/0    S+     0:00  \_ nginx: worker process
  414 pts/0    S+     0:00  \_ nginx: worker process
  415 pts/0    S+     0:00  \_ nginx: worker process
  417 pts/0    S+     0:00  \_ nginx: worker process
  418 pts/0    S+     0:00  \_ nginx: worker process
  419 pts/0    S+     0:00  \_ nginx: worker process
   39 pts/0    S+     0:00 /bin/bash /scripts/start_nginx_certbot.sh

After a reload Nginx starts new worker processes with the new configuration settings, but lets the old processes shut down in a controlled fashion. This means that if there are clients that holds a TCP connection open the old worker will remain active until all of its connections are terminated: https://www.ruby-forum.com/t/memory-usage-doubles-on-reload/239590/2

[JonasAlfredsson]

How often do you reload, and what type of clients connect to you?
Because it seems weird if they hold a connection open for the 8 days reload cycle that is default.

[Nyffeneggerr]

Ah yes that would explain the behavior to me. We are controlling a huge amount of sensors that maintain a persistent connection (Websocket) to our service (trough NGINX). These connections remain open as long as the VM is available (until restart/crash or whatever).
What do you think, would it make sense in this case to increase the renewal interval? Or do you see some other solution?

[Nyffeneggerr]

Or would it be possible to force the termination of the "old" process?

[JonasAlfredsson]

Depends on what you value here.

Let's Encrypt certificates are valid for 90 days, so you could probably set it to 85 days or so, but read this first.
However, this still has the issue that your clients will continue to hold that connection open, so if they do not restart/reconnect during another 80 days you will run in to the same issue again.

I don't find any information on how to "forcefully" reload Nginx, so the next best thing would perhaps be to restart the container, which terminates everything and starts up again with new certificates. This means that you will have some downtime (seconds) but if your clients can handle that, then it would work.

[JonasAlfredsson]

You could probably write some own logic which tries to kill long lived processes, but I believe that is out of scope of the scripts I provide.

[Nyffeneggerr]

Makes totally sense to me, thanks a lot for your help Jonas! Btw. I really appreciate your work! Thanks a lot and have a good day.

[Nyffeneggerr]

Thanks again, I will definitely give that setting a try. And if not working, I will ensure to restart the container once per month.

[Nyffeneggerr]

Just as an information, I tested the worker_shutdown_timeout and it works perfectly. The devices are disconnected as desired and the memory 'leak' is gone. Everything fine, thanks a lot for your help!

[JonasAlfredsson]

Glad to hear that!