Graceful shutdown

[pkieltyka]

Graceful shutdown

[pkieltyka]

@cyx yea exactly, it's important to preempt certain code blocks properly and that is where the valve package comes in. I'm going to update it's README soon, but the thought is its designed as an extension to context.Context, that adds an additional valve signal that when stopped will tell non-preemptive code blocks to stop processing and finish up, aka, the taps are turning off, stop what you’re doing.

it's already first-class to chi since it just relies on context.Context. It's not specific to just http servers, its useful for any kind of server. I have however moved the valve package to https://github.com/pressly/valve

I thought to write something like github.com/tylerb/graceful from scratch and having it right as a part of chi/graceful - but there is no point, I'd just be ripping their code for what does a very fine job. The missing part was signaling the shutdown and having proper code to let non-preemptable code finish. At this present time, I recommend the approach https://github.com/pressly/chi/blob/master/_examples/graceful/main.go but I'm always open to feedback and future ideas. Let me know how it works for all :)

[pkieltyka]

it's also likely someone will solve this outside of chi using Go 1.7's request context and it will "just work" with chi since we only depend on the stdlib net/http and context.

[VojtechVitek]

We should discuss the Graceful Shutdown workflow:

On SIGINT/SIGTERM (or any other registered signal):

1a. Close listening socket and refuse any incoming connections.
1b. or .. Start responding with HTTP 503.
2. Disable keep-alive connections (how about http2?).
3. Start time.After(softTimeout) to let active requests finish.
4. When softTimeout expires, close the root context. Start time.After(hardTimeout).
5. Wait for all active requests to finish (middleware adding/removing value from channel or atomic.Uint64)
6. When hardTimeout expires, exit program.

• softTimeout lets active requests finish their work without closing the context chain

I think all of the above except for 1a and 2 can be solved by middleware.

[kanocz]

"1a" variant is much better (allows to use SO_REUSEPORT and start second instance)
P.S.: it's possible just to use https://github.com/tylerb/graceful (or get some inspiration there)

[cyx]

Yep, right now we're using goji/graceful, and it appears to work, so I'm curious what @pkieltyka is thinking re: his comment that what we're doing wouldn't work. Maybe subtle race conditions?

[pkieltyka]

@cyx btw, this is what I was thinking https://github.com/goware/valve/blob/master/_example/main.go

its agnostic to chi, but of course works well with it

[cyx]

@pkieltyka nice, I see what you mean now -- you're very precise in the teardown process so I guess the way we're doing it now will be hand-waivy about things at best.

Excited to see something first class in chi?