servo/ipc-channel

A multiprocess drop-in replacement for Rust channels

ipc-channel

📚 Documentation 📚

Overview

ipc-channel is an implementation of the Rust channel API (a form of communicating sequential processes, CSP) over the native OS abstractions. Under the hood, this API uses Mach ports on the Mac and file descriptor passing over Unix sockets on Linux. The serde library is used to serialize values for transport over the wire.

As much as possible, ipc-channel has been designed to be a drop-in replacement for Rust channels. The mapping from the Rust channel APIs to ipc-channel APIs is as follows:

• channel() → ipc::channel().unwrap()
• Sender<T> → ipc::IpcSender<T> (requires T: Serialize)
• Receiver<T> → ipc::IpcReceiver<T> (requires T: Deserialize)

Note that both IpcSender<T> and IpcReceiver<T> implement Serialize and Deserialize, so you can send IPC channels over IPC channels freely, just as you can with Rust channels.

The easiest way to make your types implement Serialize and Deserialize is to use the serde_macros crate from crates.io as a plugin and then annotate the types you want to send with #[derive(Deserialize, Serialize]). In many cases, that's all you need to do—the compiler generates all the tedious boilerplate code needed to save and restore instances of your types.

In order to bootstrap an IPC connection across processes, you create an instance of the IpcOneShotServer type, register a global name, pass that name into the client process (perhaps with an environment variable or command line flag), and connect to the server in the client. See cross_process_embedded_senders() in test.rs for an example of how to do this using Unix fork() to spawn the process.

Major missing features

• Servers only accept one client at a time. This is fine if you simply want to use this API to split your application up into a fixed number of mutually untrusting processes, but it's not suitable for implementing a system service. An API for multiple clients may be added later if demand exists for it.

ipc-channel's Issues

How about using shared-memory to implement ipc-channel.

Refer to http://doc.qt.io/qt-5/qtcore-ipc-sharedmemory-example.html

Trailing data with `bytes_channel` on MacOS

As evidenced by 988a503 , when transferring raw data with a length not a multiple of 4 through a bytes_channel, on the MacOS backend, the data is padded with 0s. Sooner or later this is going to bite someone.

benchmark tests: bench_size does not compile

After 20ed7d2 bench_size needs to be updated as it currently fails to compile at line 40 because Cell<T> doesn't implement Sync.

allow access to the underlying fd to register it in a mio event loop

hi,

for various reasons I have a project that heavily uses mio, and I want my subprocesses to get orders through ipc_channel, and be notified of new orders in the same event loop they use.
For this, I need to implement the mio::Evented trait on IpcReceiver. With access to the underlying file descriptor (it would be very platform dependent, of course), I'd be able to easily integrate it. Unfortunately IpcReceiver.os_receiver and OsIpcReceiver.fd are private.

I don't know if it's something you would like to support, I'd be happy to hear your thoughts about it.

Consider returning more meaningful error values than ()

When IPC channel creation fails, we get

thread 'http_loader' panicked at 'called `Result::unwrap()` on an `Err` value: ()', src/libcore/result.rs:732"

Move to Servo org, set up travis, homu.

Preserve sending order upon receiving

Do I understand correctly that ipc-channel doesn't guarantee that the messages (sent from the same sender) are getting received in the same order? I can observe that on OSX with my application. (Edit: nope, turns out to be late night programming syndrome). If that's the case, is there any way/alternative for me to enforce the order?

Zero-copy support

Is it possible to do zero-copy with ipc-channel today? I've seen OsIpcSharedMemory, but I'm not sure it can be used directly for this purpose.

If it's not possible right now, do you plan to support it? I'm testing some ideas for interprocess video transfer, and zero-copy would be really helpful.

Thanks

ipc-channel tests truncate received data to expected size

I don't understand this -- every test truncates the received data to the expected size, e.g.:

https://github.com/servo/ipc-channel/blob/master/src/platform/test.rs#L28
https://github.com/servo/ipc-channel/blob/master/src/platform/test.rs#L139

Is it expected/allowed that you can receive more data than you send, or even a bigger buffer?!

I had a bug where I was accidentally sending data twice, and these tests were happily passing.

Publish new release?

There was an update to the rand dependency and the version number was updated, but no new crate was actually published. Could a new release be published?

One-shot server hangs indefinetely if client process died

Code looks like this:

                    let (server, server_name) = IpcOneShotServer::new().unwrap();

                    let mut child = Command::new(ciri_path)
                        .arg("--log")
                        .arg(ciri_logging_level)
                        .arg("--server")
                        .arg(server_name)
                        .arg("discover")
                        .arg(entry.path())
                        .env("CIRI_FIXTURE_DIR", ciri_fixture_dir)
                        .env("RUST_BACKTRACE", "1")
                        .spawn()
                        .unwrap();

                    let (_, (suite_name, test_name)): (_, (String, String)) =
                        server.accept().unwrap();

                    child.wait().unwrap();

My child panics in the main thread and dies w/o sending anything. This code then hangs on server.accept().

Is it expected? Should I take care of it on my side?

macos: Make sure to decrement the ref count of the mach port on ChannelClosed

Similar to what is seen in #96 we should ensure that we do not have mach port leaks on ChannelClosed.

We should also use an auto-incrementing number as an id instead of the local_port cast to a u64(here and here) to avoid a race condition if the mach port number is reused. I do not know mach ports well enough to know if this is necessary yet.

Related to: #96 #105

Note suggesting to use mkstemp or mkdtemp instead of mktemp

FYI, I get the following note when building on ARM when everything is being linked at the end. Don't recall if I saw it elsewhere.

note: /local/servo/components/servo/target/debug/deps/libipc_channel-948af358b3c092a6.rlib(ipc_channel-948af358b3c092a6.0.o): In function `platform::linux::_$LT$impl$GT$::new::he43c9a17ef03c572fbb':
/home/username/.cargo/git/checkouts/ipc-channel-62de6a747162919c/master/platform/linux/mod.rs:433: warning: the use of `mktemp' is dangerous, better use `mkstemp' or `mkdtemp'

Relax Debug bounds on IpcReceiver and IpcSender

Their Debug impls shouldn't require T: Debug.

testy test test issue for notification

Use memfd for IPCSharedMemory on Linux

Currently on linux IPCSharedMemory is backed by a file in /tmp, which is often not tempfs, and ends up doing unnecessary disk IO.

If possible, we should use memfd_create() to take an existing memory buffer and make it into a file descriptor (which we can then share).

cc @antrik @pcwalton

Intermittent thread 'platform::test::cross_process_sender_transfer' panicked at 'assertion failed: `(left == right)` (left: `2`, right: `1`)', platform/test.rs:276

Maybe allow ipc-channel to build using rust stable on windows

Right now if I try to build ipc-channel using stable rust (1.12.1 d4f39402a 2016-10-19) on windows, it fails with this error:

error[E0554]: #[feature] may not be used on the stable release channel
  --> C:\Users\kats\zspace\mozilla-git\third_party\rust\ipc-channel\src\lib.rs:11:4
   |
11 |                   feature(mpsc_select))]
   |                   ^^^^^^^^^^^^^^^^^^^^^

I'm not entirely sure what mpsc_select is, because that token doesn't appear anywhere else in the repo. Can this be removed, or can be rewritten using stuff that compiles with stable rust?

/cc @Ms2ger (who originally added the line in f4fef7d)
/cc @jrmuizel as FYI

ipc-channel makes assumptions about OS buffer sizes

ipc-channel seems to make assumptions about underlying OS buffer sizes; the tests want to send()/recv() in the same thread on the same channel, e.g.: https://github.com/servo/ipc-channel/blob/master/src/platform/test.rs#L97 . This is a problem; is this intended in the API, or is it unintentional due to the way the tests are written?

I'm going to end up needing to queue writes to happen from another thread for every sender as a result of this for the windows impl (probably a work queue).

Set up android, mac, windows testing

CC @edunham @larsbergstrom @metajack

Fix warnings

warning: unused extern crate [-W unused-extern-crates]
  --> /Users/nox/src/servo/../ipc-channel/lib.rs:22:1
22 |> extern crate uuid;
   |> ^^^^^^^^^^^^^^^^^^

warning: value assigned to `allocated_buffer` is never read, #[warn(unused_assignments)] on by default
   --> /Users/nox/src/servo/../ipc-channel/platform/macos/mod.rs:597:29
597 |>                             allocated_buffer = None;
    |>                             ^^^^^^^^^^^^^^^^

Assertion failure when cloning a channel when the receiver no longer exists

thread 'ScriptTask PipelineId { namespace_id: PipelineNamespaceId(1), index: PipelineIndex(0) }' panicked at 'assertion failed: mach_sys::mach_port_mod_refs(mach_task_self(), self.port,
                             MACH_PORT_RIGHT_SEND, 1) == KERN_SUCCESS', /Users/jdm/.cargo/git/checkouts/ipc-channel-d95a23d1f1577bfc/master/platform/macos/mod.rs:292
* thread #6: tid = 0x26ded7, 0x000000010297fe90 servo`rust_panic, stop reason = breakpoint 1.1
  * frame #0: 0x000000010297fe90 servo`rust_panic
    frame #1: 0x000000010296e792 servo`sys_common::unwind::begin_unwind_inner::h1e338bc64cf56d9dr8r + 466
    frame #2: 0x0000000102606728 servo`ipc_channel::sys_common::unwind::begin_unwind<&str>(msg=(data_ptr = "assertion failed: mach_sys::mach_port_mod_refs(mach_task_self(), self.port,\n                             MACH_PORT_RIGHT_SEND, 1) == KERN_SUCCESS", length = 145), file_line=0x0000000104113d00) + 120 at mod.rs:214
    frame #3: 0x0000000102605cff servo`ipc_channel::platform::macos::MachSender.Clone::clone(self=0x0000000125a12790) + 143 at <std macros>:3
    frame #4: 0x0000000100fd22c5 servo`script::ipc::IpcSender<T>.Clone::clone(self=0x0000000125a12790) + 21 at ipc.rs:122
    frame #5: 0x0000000100fd08a3 servo`fnfn + 787 at script_task.rs:513
    frame #6: 0x0000000100fd04f8 servo`fnfn + 104 at task.rs:52
    frame #7: 0x0000000100fd0448 servo`script::boxed::F.FnBox<A>::call_box(self=0x0000000115c43280, args=<unavailable>) + 104 at boxed.rs:516
    frame #8: 0x000000010079511c servo`script::boxed::Box<FnBox<A, Output = R>+ Send + 'a>.FnOnce<A>::call_once(self=Box<FnBox<()>> at 0x0000000125a12bc0, args=<unavailable>) + 60 at boxed.rs:532
    frame #9: 0x0000000100794cee servo`fnfn + 78 at mod.rs:280
    frame #10: 0x0000000100794c9a servo`script::sys_common::unwind::try::try_fn<closure>(opt_closure=0x0000000125a12cf0) + 58 at mod.rs:146
    frame #11: 0x000000010297fe89 servo`__rust_try + 9
    frame #12: 0x000000010297d32f servo`sys_common::unwind::try::inner_try::hc13d8e198528cd0fZ4r + 111
    frame #13: 0x0000000100794c04 servo`script::sys_common::unwind::try<closure>(f=closure at 0x0000000125a12d08) + 100 at mod.rs:116
    frame #14: 0x0000000100794a7c servo`fnfn + 412 at mod.rs:280
    frame #15: 0x000000010079539d servo`script::boxed::F.FnBox<A>::call_box(self=0x0000000115c24440, args=<unavailable>) + 77 at boxed.rs:516
    frame #16: 0x0000000102981f6e servo`sys::thread::_$LT$impl$GT$::new::thread_start::h906ddce3a93d4852ksw + 142
    frame #17: 0x00007fff982c6899 libsystem_pthread.dylib`_pthread_body + 138
    frame #18: 0x00007fff982c672a libsystem_pthread.dylib`_pthread_start + 137
    frame #19: 0x00007fff982cafc9 libsystem_pthread.dylib`thread_start + 13

Make IpcSender::send take a reference

For IPC, a sender never needs to take ownership of the sent value.

Gate on appveyor

CC @larsbergstrom

Option<char> values are corrupted when sent over an ipc channel

This test fails on OS X:

#[test]
fn simple_char() {
    let ch = Some('a');
    let (tx, rx) = ipc::channel().unwrap();
    tx.send(ch).unwrap();
    let received_ch = rx.recv().unwrap();
    assert_eq!(ch, received_ch);
}

yields

---- test::simple_char stdout ----
    thread 'test::simple_char' panicked at 'assertion failed: `(left == right)` (left: `Some('a')`, right: `Some('\u{0}')`)', test.rs:394
note: Run with `RUST_BACKTRACE=1` for a backtrace.


failures:
    test::simple_char

test result: FAILED. 46 passed; 1 failed; 0 ignored; 0 measured

ipc-channel can deadlock on Linux

When a recv() is reading a fragmented message, the Linux implementation pushes any interleaved fragments from other messages back to the same file descriptor via sendmsg().

However, since that is being sent to the same fd, if the send buffer is full, deadlock occurs - because the send is occurring on the same thread that should be draining from that fd.

This causes a lot of timeouts in the webrender reftests, and (I suspect) is also responsible for a lot of the Linux timeouts we see on the build machines.

For example, see the backtrace below - thread 72 is blocked in sendmsg since the send buffer is full, but this is called inside of the recv() function for the same fd, so the send buffer will never be emptied.

(gdb) thread 72
[Switching to thread 72 (Thread 0x7fd63a3ff700 (LWP 9553))]
#0  0x00007fd644c875bd in sendmsg () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0  0x00007fd644c875bd in sendmsg () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007fd646b5ec41 in platform::linux::recv::h0111fbf8cc03dac2Mxb ()
#2  0x00007fd646b5d34d in platform::linux::UnixReceiver::recv::h76d3513f4fe3a26e2Ra ()
#3  0x00007fd6467af31f in render_backend::RenderBackend::run::h33e3f2df27cc03b3ULo ()
#4  0x00007fd646803bd8 in sys_common::unwind::try::try_fn::h13009946449717054376 ()
#5  0x00007fd64754d959 in __rust_try ()
#6  0x00007fd647549b2c in sys_common::unwind::try::inner_try::h62f00e3d34dbb509D3s ()
#7  0x00007fd646804d8b in boxed::F.FnBox$LT$A$GT$::call_box::h7120465612878348244 ()
#8  0x00007fd64754fc64 in sys::thread::Thread::new::thread_start::hb3a0514b9d87669e1mx ()
#9  0x00007fd642ed3182 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fd644c8647d in clone () from /lib/x86_64-linux-gnu/libc.so.6

Further restrict the underlying FDs in UnixSender and UnixReceiver

UnixSender could use shutdown(SHUT_RD)

UnixReceiver could use shutdown(SHUT_WR)

Test case `cross_process_two_step_transfer_spawn()` fails on MacOS

This needs some digging.

(The comment in the source code suggests that it might be a kernel problem -- but I rather doubt it: a lot of the complexity of the Mach IPC system comes precisely from trying very hard to cover such corner cases...)

IpcSender should consistently be Sync or !Sync.

It seems like it's Sync on linux and !Sync on windows right now.

Write a test for #30

Should be straightforward - make a channel, send a value on the sender, drop the sender, and try to receive it from the receiver.

mach implementation needs to break large messages into pieces

servo/servo#13054 and servo/servo#10390 demonstrate clear cases where mach imposes size limits on the messages sent. We should break messages that exceed these limits into pieces and reconstruct them on the receiver's end.

Doesn't compile with musl

I've tried to fix the compile issue on alpine/musl and ran into this problem:

libc::msghdr has two fields that don't exist on non-musl, __pad1 and __pad2. Since there is no construcutor and these fields are private, msghdr can not be created.

https://github.com/rust-lang/libc/blob/master/src/unix/notbsd/linux/musl/b64/mod.rs

This currently blocks a musl port due to src/platform/unix/mod.rs. Everything else is trivial to fix.

Allow one-shot routes

It would be nice to have one-shot routes instead of requiring callers to do option dances.

This first came up in servo/servo#6884.

Add support for seL4 IPC

I'm leaving this largely as a to-do for myself. Once there is some amount of std support for seL4, this crate would be an easy port to it.

Consider multiplexing everything over one socket on Unix

I'm worried about the file descriptor limit being an ever-present failure mode. Chromium and Gecko multiplex everything over one socket, and I'm beginning to think we should too.

test test test

irc notification?

Intermittent "Connection reset by peer" for in-process channel in a recv() with a queued message but no sender

We got a panic in this code in Servo:

226                 Command::Exit(result) => {
227                     result.send(()).unwrap();
228                     break;
229                 }
...
382     pub fn exit(&self) {
383         let (response_chan, response_port) = ipc::channel().unwrap();
384         self.chan.send(Command::Exit(response_chan)).unwrap();
385         response_port.recv().unwrap();
386     }

The panic:

  │ thread 'Constellation' panicked at 'called `Result::unwrap()` on an `Err` value: Error { repr: Os { code: 104, message: "Connection reset by peer" } }', ../src/libcore/result.rs:688

This is on Linux.

mio regressed using a debugger with ipc-channel

I ended up doing a build of Servo using ipc-channel master, and discovered that hitting a breakpoint in a debugger caused a ton of "System call interrupted" panics from ipc-channel after resuming. We're going to need to address this before we can use any revision past 357abb9 in Servo.

Reduce copying when sending

This came out of discussions in #138.

The proposal there is to allow sending borrowed data, e.g. a method send_clone(&data) with the same semantics assend(data.clone()) but removing the unnecessary copying.

This deals with borrowing as long as the types match up, but doesn't deal with the case where the types disagree, e.g. sending a &[&T] on a channel of type IpcSender<Vec<T>>. There is some discussion in #138 about approaches to this.

cc @nox @antrik

Make android OSes use the unix module instead of inprocess

As of #19, android OSes do not currently use the unix module, but instead use inprocess.

This may have some overlap with the metabug servo/servo#8771.

• servo/servo#8762 - Partly addressed by #140 (the removal of mkstemp)
• servo/servo#8571 - Would need to be tested

`IpcBytesSender::recv()` truncates sometimes

See servo/webrender#991.

Unexpected drop

Consider the following program:

#[macro_use]
extern crate serde_derive;
extern crate ipc_channel;

#[derive(Serialize, Deserialize)]
struct Foo;

impl Drop for Foo {
    fn drop(&mut self) {
        println!("drop");
    }
}

fn main() {
    let (sender, receiver) = ipc_channel::ipc::channel().unwrap();
    let foo = Foo;
    sender.send(foo).unwrap(); //`foo` consumed here
    let _foo2 = receiver.recv().unwrap(); //`_foo2` is expected to drop here
}

From the user perspective, value gets consumed by sender and then reproduced by the receiver. This is expressed as move semantics by the API but doesn't actually behave this way: the program drops Foo twice here.

I understand that with serialization involved it becomes more difficult to say when the value is destroyed and when it's not. I do find this rather critical, since the user code may have unintended effects in Drop, and debugging cases like this is not fun. Can the implementation make sure to avoid dropping an object on send, and only drop it if the send failed? Alternatively, something should prevent/warn the user from ever having destructors on the types being sent.

cc @glennw

Intermittent assertions on Linux when using WR

I'm still seeing regular assertions on Linux when using WR - don't have a repro yet sorry. It seems to happen within a minute or so of running - perhaps a resource leak?

thread 'LayoutThread PipelineId { namespace_id: PipelineNamespaceId(0), index: PipelineIndex(0) }' panicked at 'assertion failed: fd >= 0', /home/projects/work/servo/servo/.cargo/git/checkouts/ipc-channel-3840821e54ea11a6/master/platform/linux/mod.rs:765

Compile time failure

When using the latest nightly rustc and compiling on the master branch I get the following.

error[E0282]: type annotations needed
   --> src/ipc.rs:155:42
    |
155 |                     let mut serializer = bincode::Serializer::new(&mut bytes);
    |                         --------------   ^^^^^^^^^^^^^^^^^^^^^^^^ cannot infer type for `E`
    |                         |
    |                         consider giving `serializer` a type
error: aborting due to previous error
error: Could not compile `ipc-channel`.

Windows support

Just creating this issue to keep track of implementing Windows support.

Store an Arc internally

We have had cases where Arc<IpcSender<T>> was added to curb rampant file handle cloning in Servo. Why not do this by default and have IpcSender store an Arc?

Intermittent failure in test::router_big_data on linux

failures:

---- test::router_big_data stdout ----

	thread 'test::router_big_data' panicked at 'called `Result::unwrap()` on an `Err` value: "SendError(..)"', /buildslave/rust-buildbot/slave/nightly-dist-rustc-linux/build/src/libcore/result.rs:870

failures:

    test::router_big_data

It may be related to these other messages that appeared in the log:

thread '<unnamed>' panicked at 'Readable event for unknown token: Token(135)', src/platform/unix/mod.rs:503

note: Run with `RUST_BACKTRACE=1` for a backtrace.

thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: Error { repr: Os { code: 32, message: "Broken pipe" } }', /buildslave/rust-buildbot/slave/nightly-dist-rustc-linux/build/src/libcore/result.rs:870

router tests are non-deterministic because of shared router

All of the tests use the global ROUTER RouterProxy, meaning that test runs end up depending on test order, number of threads, etc.

If everything's working okay, this isn't an issue -- but right now I'm chasing down an issue where every router test passes individually, and they will often pass when run in sequence, but every once in a while one will fail with a hard-to-track-down unwrap() panic after a previous one succeeded.

irc notifications should include issues

Multiple senders

I'm probably missing it, but where's the code to prevent multiple senders from having the messages interleaved?

i.e. on unix the messages are not sent in a single write call so what code keeps the next sender on a different thread from starting it's writes before the previous sender has finished?