Deadlock when using async input and pipes.

[boostorg]

Deadlock when using async input and pipes.

[klemens-morgenstern]

Weird. Alright, I'll look into it. It might actually be the same bug for all your issues.

[klemens-morgenstern]

I think it's a bug in the async stdin version, because (I fear) that doesn't close the pipe when done. Thus cat stays active since the pipe is open.

Can you test using a different input method? i won't be able to look into it before next week.

[davisking]

I've found a number of other seemingly related deadlocks. Like maybe #65. As well as this:

    std::string data = "some data to send to cat";
    std::future<std::string> result;
    boost::asio::io_service ios;
    boost::process::pipe p;
    opstream out;
    spawn("cat", std_in<out, std_out>p);
    spawn("cat", std_in<p, std_out>result, ios);

    std::thread t([&]() { out << data << std::flush; out.pipe().close(); });
    // run blocks forever.
    ios.run();
    t.join();

However, this slightly different version works as expected:

    std::string data = "some data to send to cat";
    std::future<std::string> result;
    boost::asio::io_service ios;
    boost::process::pipe p;
    opstream out;
    spawn("cat", std_in<out, std_out>result, ios);

    std::thread t([&]() { out << data << std::flush; out.pipe().close(); });
    ios.run();
    t.join();

Maybe they are all caused by the same underlying issue. I am not sure.

[klemens-morgenstern]

I assume you're on linux, right?
Can you try out what happens if you change spawn to async_system and put some dummy handler in?

I.e. async_system(ios, [](const boost::system::error_code & ec, int ret){}, ("cat", std_in<out, std_out>p);

[davisking]

Yeah, using Ubuntu 16.04.

I tried replacing spawn with async_system and it still hangs just like before.

[davisking]

Sweet, thanks :)

Yeah, that's my suspicion as well.

[klemens-morgenstern]

Do you use boost 1.68 or the current HEAD of the git repo?

[davisking]

I'm using Boost 1.66 along with the code from https://github.com/boostorg/process as it was on Jan 1st.

[klemens-morgenstern]

Ok, that should, but could you use the currend develop-HEAD just to be sure?

[davisking]

Just grabbed the develop-HEAD of https://github.com/boostorg/process and got the same results, same things work and same things still hang.

[LuisAyuso]

I am experimenting sort of the same issue. Although my code only binds stderr and stdout output streams, whenever I execute the context run function it will hang in there even when the child process is gone.

something like:

    child = boost::process::child(path, args, group, bp::std_out > stdout_stream, bp::std_err > stderr_stream, ios);
[..]
    ios.run();
    child.wait();

When I submit a kill signal to the child process, the parent process is still waiting for IO. child.wait() is never executed.

Is this the same issue?
If so, could we rename this thread to remove the input keyword?

[klemens-morgenstern]

@LuisAyuso I would need more about your use-case - do the streams refer to an async_pipe?

[LuisAyuso]

my stream objects are a custom class, they wrap an async_pipe, hold a buffer to accumulate output, and re-register the callback after every read.
I noticed that the read callback is never invoked after child process dead, not even to notify an error.

[klemens-morgenstern]

Weird, which system are you on? Normally you should get an EOF.

What do you call to start the read operation?

[LuisAyuso]

Sorry for fragmented information, let's see if we manage to have a proper report ;)
I am using Ubuntu 18.04, with gcc 7.4.0
The Boost version is 1.69

I have been cleaning the code so I can show something here:

    ba::io_context ios;

    std::vector<char> stdout_buffer(1024);
    auto              out_buffer{ba::buffer(stdout_buffer)};
    bp::async_pipe    std_out{ios};
    std::function<void(const boost::system::error_code &ec, size_t s)> stdout_callback;
    stdout_callback = [&](const boost::system::error_code &ec, size_t s) {
        if (ec)
            return;
        auto v = std::string_view(&stdout_buffer.front(), s);
        fmt::print("stdout: {}", v);
        std_out.async_read_some(out_buffer, stdout_callback);
    };

    std::vector<char> stderr_buffer(1024);
    auto              err_buffer{ba::buffer(stderr_buffer)};
    bp::async_pipe    std_err{ios};
    std::function<void(const boost::system::error_code &ec, size_t s)> stderr_callback;
    stderr_callback = [&](const boost::system::error_code &ec, size_t s) {
        if (ec)
            return;
        auto v = std::string_view(&stdout_buffer.front(), s);
        fmt::print("stderr: {}", v);
        std_err.async_read_some(err_buffer, stderr_callback);
    };

    std_out.async_read_some(out_buffer, stdout_callback);
    std_err.async_read_some(err_buffer, stderr_callback);

    child =
          bp::child(path, args, group, bp::std_out > std_out, bp::std_err > std_err, ios);

    ios.run();
    child.wait();

The behavior of this code is the same as the one reported originally.

• ios.run() blocks event when the child process is dead.
• child.wait() is never executed if the child process is externally killed

[klemens-morgenstern]

This is weird indeed. Can you add bp::on_exit to check if this gets called?

Also, where does the group come from?

[LuisAyuso]

Group is currently default initialized. (bp::group)

on_exit works, I can use it now to shut down the io context (ios.stop()) and let the program proceed to child.wait()
I can use now this as a workaround, it is way better than my original one (which was to put io.run in a separate thread)

[klemens-morgenstern]

The better workaround would be to close the pipes from the on-exit handler.

Can you try the current master branch and see if it fixes the issue?

[xd009642]

So I encountered this issue today, but I had also encountered the same effect in some rust async code in tokio. So because of the possibility they could be the same issue here is my associated issue and more importantly the PR that fixed it tokio-rs/tokio#2255 tokio-rs/tokio#2218