Broker crashes, running out of file descriptors

[RalfJung]

Broker crashes, running out of file descriptors

[RalfJung]

Looking at the process (/proc/$PID/fd) right now (with about 70 active connections) shows 1019 file descriptors. 144 are anon_inode:[timerfd], and 800 of them are pipe:[...]. 75 are socket:[...]. So it seems these pipes (whatever they are) are actually much worse than the timerfds.

[RalfJung]

Grepping for pipes shows that the hooks use a pipe. And they never seem to close it.

What is the reason not to use subprocess.check_output?

[kostko]

What is the reason not to use subprocess.check_output?

That would block the event loop. We need non-blocking file descriptors, which we register in the event loop, which uses epoll.

It looks like HookProcess.close doesn't close the pipe file descriptors after unregistering them from the event loop. Could you try adding self.process.stdout.close() (and the same for stderr) at the end of close?

[RalfJung]

Could you try adding self.process.stdout.close() (and the same for stderr) at the end of close?

Already done and rolled out to one server and confirmed to get rid of the hundreds of pipes. :) See #59.

[RalfJung]

After 12h of uptime, we now have the following FD usage:

# ls -lah | fgrep socket -c
143
root@gw3:/proc/6638/fd# ls -lah | fgrep timer -c
278

Looks like indeed the big leak got fixed.