Stop button in the UI has different behavior than `Ctrl+C`-ing the same process

[deref]

Stop button in the UI has different behavior than `Ctrl+C`-ing the same process

[brandonbloom]

I'd feel better if we figured out what was going on :)

[brandonbloom]

I think we can consider this fixed now, but it's going to be a little while still before our next release. Thanks for your patience.

[brandonbloom]

Hi @siggisim,

Could you please use ps ax -o 'pid,ppid,pgid,command' to look at the process id and process group ids for the server spawned by bazel? I'm curious if bazel's spawn is creating an isolated process group or not.

cc @BenElgar and @kendru who may have more insight in to this problem.

We'll get to the bottom of this & see if there is anything we can do to make exo more robust when killing processes here.

Thanks for the report!
-- Brandon

[siggisim]

Here are the two processes, I believe the first is the bazel run and the second is our server:

21847  8849 21847 /usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/Python /usr/local/bin/bazel run enterprise/server/cmd/executor:executor -- --monitoring_port=9091 --executor.docker_socket=
21848 21847 21847 /private/var/tmp/_bazel_siggi/e1a494dcbcb7b7235bc655bacb26413a/execroot/buildbuddy/bazel-out/darwin-fastbuild/bin/enterprise/server/cmd/executor/executor_/executor --config_file=enterprise/config/executor.local.yaml --port=8888 --monitoring_port=9091 --max_shutdown_duration=3s --monitoring_port=9091 --executor.docker_socket=

[siggisim]

To complicate things slightly further, I'm using https://github.com/bazelbuild/bazelisk - that's why that first process is Python.

[brandonbloom]

OK, looks like they are in the same process group. Thanks for the details! That says to me it should work.

Going to assign to @BenElgar to try to suss out what's happening here.

[BenElgar]

Hi @siggisim! Sorry, for the slow response. I've spent some time trying to reproduce the issue using the same Procfile and haven't had any luck: the exec process seems to successfully stop via the UI and port 9091 is no longer bound.

My working theory is that you're encountering some UI bug whereby the UI is failing to call the stop process API. Do you reckon you could try it again but include any output from the console in your browser's dev tools and any relevant network requests? You should have one POST request to "http://localhost:4000/_exo/workspace/stop-component" with a 200 response.

[siggisim]

Hey @BenElgar - thanks for looking into it!

Not seeing any weirdness in the Network tab, I see stop-component called with a 200 response code.

Maybe I'm just running an older exo version, and it's been fixed since. I'm running exo 2021.07.30.

When testing, did you wait until 9091 got bound before killing? It'd probably take a while because it has to do a build first.

[BenElgar]

Yes, I'm using the same version of exo as you.

I don't manage to do a successful build due to some environmental issue[0] but the port does seem to be bound before I attempt to stop it:

❯ lsof -i -P -n | grep 9091
executor  82675  ben    3u  IPv6 263532      0t0  TCP *:9091 (LISTEN)

and the port is no longer bound once the process is stopped via the UI. I'm going to try and get the build working to see if that helps me reproduce.

[0] I get the following error:

18:18:30  app ERROR: /home/ben/Programming/buildbuddy/enterprise/app/BUILD.bazel:80:4: Executing genrule //enterprise/app:sha failed: (Exit 127): bash failed: error executing command /bin/bash -c ... (remaining 1 argument(s) skipped)
18:18:30  app /bin/bash: line 1: shasum: command not found
18:18:30  app xargs: shasum: No such file or directory
18:18:30  app Target //enterprise/server/cmd/server:buildbuddy failed to build

[siggisim]

Hm okay, seems like if that port is being bound you're good. Seems like just the app is failing to build because shasum isn't installed, but the executor is building alright.

Might just be something Mac related then (I'm on a Mac). Feel free to close, I'm happy to work around it.

[brandonbloom]

We've reproduced this. Trying to fix. Thanks :)

[brandonbloom]

+ @kendru for visibility.

In investigating this, found all sorts of problems, but haven't fixed this actual problem yet!

Various fixes are now on main: https://github.com/deref/exo/compare/5f8f1d7..6b2cf25

Still trying to root cause the main issue.

[brandonbloom]

OK, I've totally overhauled how this works and I think it's fixed. Changes here:
https://github.com/deref/exo/compare/5f8f1d7..65eef975961fe8d76b88c0a7e218f94257e81d61

My brain is fried tonight though, so @BenElgar and @kendru please review these changes and sanity the behavior of things.