API freeze when task do not complete

[devo-ps]

API freeze when task do not complete

[zbal]

Caused by: https://github.com/Wiredcraft/pipelines/blob/dev/pipelines/api/server.py#L69

Approaches:

• add timeout on tasks
• add dedicated thread to maintain the API up and running despite the workers being stucked

@JuhaS - are the threads being handling the pipelines? or the tasks?

[zbal]

Had again the issue - it seems to be related in fact with the reading of logs - that could be "breaking" the server ...

https://github.com/Wiredcraft/pipelines/blob/dev/pipelines/api/server.py#L380-L387

Wondering if a bad file would block the process...

[JuhaS]

Probably because we don't currently close a file after every line, so it stays locker: https://github.com/Wiredcraft/pipelines/blob/dev/pipelines/plugins/bash_executor.py#L64

We should probably open/close it after every line (not sure of the speed though).

[zbal]

The likely root cause: moby/moby#30405 -- when running pipelines in a container, when there is barely no network traffic (e.g. no one on the UI and a webhook triggered every couple of hours), the 1st REQ through an nginx will often be welcomed with a 502 ...

This could be alleviated by using a reverse proxy that enables some HTTP health check to "force" some traffic to occur. (e.g. haproxy or openresty). We've switched to haproxy on a setup and are looking forward to improvements (so far we haven't experienced anymore issue).

[JuhaS]

@zbal is there something we can do to prevent or detect this? Should we close this issue?

[zbal]

Closing - this is not a pipelines related issue.