Flaky test cases in docker tests

[nroi]

Flaky test cases in docker tests

[nroi]

For what it's worth, I've been experiencing freezes in the service lately (haven't run the test suite).

I'm pretty confident that the freezes described in this issue where caused by the docker setup, and not related to any bug in the actual Flexo code, and I'm also confident that this issue has now been fixed. So what you're describing is probably another issue.

I've created a separate issue from your description. If you gain any more information about this issue, like a pattern that it always occurs in a specific situation, please let me know, because these sorts of things are difficult to troubleshoot.

[nroi]

The problem seems to be that, while our docker containers are running, certain actions can only be executed with a delay. For example, if NGINX is running on localhost – even without docker – then the following will show that connection establishment is sometimes delayed when our docker-compose tests are running:

while true; do echo -n '' | nc -N 127.0.0.1 80; date +%s%N; sleep 0.1; done | ./analyze_nanos.py

[nroi]

It seems to be unrelated to sockets:

while true; do echo -n '' | date +%s%N; sleep 0.1; done | ./analyze_nanos.py

Gave the following output when the docker tests were running:

>>> Threshold exceeded at 2020-12-06 18:45:54.307145: 1.493492608
>>> Threshold exceeded at 2020-12-06 18:45:58.851240: 4.544129866

[nroi]

Another example:

while true; do date +%s%N; sleep 0.1; done | ./analyze_nanos.py

>>> Threshold exceeded at 2020-12-06 19:11:48.709007: 2.51274226

[nroi]

while true; do strace --absolute-timestamps=format:unix,precision:ns /bin/true  2>&1; done | ./analyze_nanos.py

>>> Threshold exceeded at 2020-12-06 21:32:27.167433: 19.471243619918823
1607286727.695725213 +++ exited with 0 +++

1607286747.166968830 execve("/bin/true", ["/bin/true"], 0x7ffc168da2e8 /* 61 vars */) = 0

[aude]

For what it's worth, I've been experiencing freezes in the service lately (haven't run the test suite).

I've not been able to get any response with HTTP, and the log shows nothing happens. Even restarting flexo doesn't work immediately. Then, after maybe 1s-2m, it just starts responding again like nothing happened.

Haven't looked into it, so could definitely be something local, though thought I'd mention anyway just in case.