servo / ipc-channel Goto Github PK
View Code? Open in Web Editor NEWA multiprocess drop-in replacement for Rust channels
License: Apache License 2.0
A multiprocess drop-in replacement for Rust channels
License: Apache License 2.0
Similar to what is seen in #96 we should ensure that we do not have mach port leaks on ChannelClosed
.
We should also use an auto-incrementing number as an id instead of the local_port
cast to a u64
(here and here) to avoid a race condition if the mach port number is reused. I do not know mach ports well enough to know if this is necessary yet.
For IPC, a sender never needs to take ownership of the sent value.
ipc-channel seems to make assumptions about underlying OS buffer sizes; the tests want to send()/recv() in the same thread on the same channel, e.g.: https://github.com/servo/ipc-channel/blob/master/src/platform/test.rs#L97 . This is a problem; is this intended in the API, or is it unintentional due to the way the tests are written?
I'm going to end up needing to queue writes to happen from another thread for every sender as a result of this for the windows impl (probably a work queue).
After 20ed7d2 bench_size
needs to be updated as it currently fails to compile at line 40 because Cell<T>
doesn't implement Sync
.
As of #19, android OSes do not currently use the unix
module, but instead use inprocess
.
This may have some overlap with the metabug servo/servo#8771.
mkstemp
)irc notification?
Their Debug
impls shouldn't require T: Debug
.
All of the tests use the global ROUTER RouterProxy, meaning that test runs end up depending on test order, number of threads, etc.
If everything's working okay, this isn't an issue -- but right now I'm chasing down an issue where every router test passes individually, and they will often pass when run in sequence, but every once in a while one will fail with a hard-to-track-down unwrap()
panic after a previous one succeeded.
I ended up doing a build of Servo using ipc-channel master, and discovered that hitting a breakpoint in a debugger caused a ton of "System call interrupted" panics from ipc-channel after resuming. We're going to need to address this before we can use any revision past 357abb9 in Servo.
I don't understand this -- every test truncates the received data to the expected size, e.g.:
https://github.com/servo/ipc-channel/blob/master/src/platform/test.rs#L28
https://github.com/servo/ipc-channel/blob/master/src/platform/test.rs#L139
Is it expected/allowed that you can receive more data than you send, or even a bigger buffer?!
I had a bug where I was accidentally sending data twice, and these tests were happily passing.
failures:
---- test::router_big_data stdout ----
thread 'test::router_big_data' panicked at 'called `Result::unwrap()` on an `Err` value: "SendError(..)"', /buildslave/rust-buildbot/slave/nightly-dist-rustc-linux/build/src/libcore/result.rs:870
failures:
test::router_big_data
It may be related to these other messages that appeared in the log:
thread '<unnamed>' panicked at 'Readable event for unknown token: Token(135)', src/platform/unix/mod.rs:503
note: Run with `RUST_BACKTRACE=1` for a backtrace.
thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: Error { repr: Os { code: 32, message: "Broken pipe" } }', /buildslave/rust-buildbot/slave/nightly-dist-rustc-linux/build/src/libcore/result.rs:870
It seems like it's Sync
on linux and !Sync
on windows right now.
FYI, I get the following note when building on ARM when everything is being linked at the end. Don't recall if I saw it elsewhere.
note: /local/servo/components/servo/target/debug/deps/libipc_channel-948af358b3c092a6.rlib(ipc_channel-948af358b3c092a6.0.o): In function `platform::linux::_$LT$impl$GT$::new::he43c9a17ef03c572fbb': /home/username/.cargo/git/checkouts/ipc-channel-62de6a747162919c/master/platform/linux/mod.rs:433: warning: the use of `mktemp' is dangerous, better use `mkstemp' or `mkdtemp'
thread 'ScriptTask PipelineId { namespace_id: PipelineNamespaceId(1), index: PipelineIndex(0) }' panicked at 'assertion failed: mach_sys::mach_port_mod_refs(mach_task_self(), self.port,
MACH_PORT_RIGHT_SEND, 1) == KERN_SUCCESS', /Users/jdm/.cargo/git/checkouts/ipc-channel-d95a23d1f1577bfc/master/platform/macos/mod.rs:292
* thread #6: tid = 0x26ded7, 0x000000010297fe90 servo`rust_panic, stop reason = breakpoint 1.1
* frame #0: 0x000000010297fe90 servo`rust_panic
frame #1: 0x000000010296e792 servo`sys_common::unwind::begin_unwind_inner::h1e338bc64cf56d9dr8r + 466
frame #2: 0x0000000102606728 servo`ipc_channel::sys_common::unwind::begin_unwind<&str>(msg=(data_ptr = "assertion failed: mach_sys::mach_port_mod_refs(mach_task_self(), self.port,\n MACH_PORT_RIGHT_SEND, 1) == KERN_SUCCESS", length = 145), file_line=0x0000000104113d00) + 120 at mod.rs:214
frame #3: 0x0000000102605cff servo`ipc_channel::platform::macos::MachSender.Clone::clone(self=0x0000000125a12790) + 143 at <std macros>:3
frame #4: 0x0000000100fd22c5 servo`script::ipc::IpcSender<T>.Clone::clone(self=0x0000000125a12790) + 21 at ipc.rs:122
frame #5: 0x0000000100fd08a3 servo`fnfn + 787 at script_task.rs:513
frame #6: 0x0000000100fd04f8 servo`fnfn + 104 at task.rs:52
frame #7: 0x0000000100fd0448 servo`script::boxed::F.FnBox<A>::call_box(self=0x0000000115c43280, args=<unavailable>) + 104 at boxed.rs:516
frame #8: 0x000000010079511c servo`script::boxed::Box<FnBox<A, Output = R>+ Send + 'a>.FnOnce<A>::call_once(self=Box<FnBox<()>> at 0x0000000125a12bc0, args=<unavailable>) + 60 at boxed.rs:532
frame #9: 0x0000000100794cee servo`fnfn + 78 at mod.rs:280
frame #10: 0x0000000100794c9a servo`script::sys_common::unwind::try::try_fn<closure>(opt_closure=0x0000000125a12cf0) + 58 at mod.rs:146
frame #11: 0x000000010297fe89 servo`__rust_try + 9
frame #12: 0x000000010297d32f servo`sys_common::unwind::try::inner_try::hc13d8e198528cd0fZ4r + 111
frame #13: 0x0000000100794c04 servo`script::sys_common::unwind::try<closure>(f=closure at 0x0000000125a12d08) + 100 at mod.rs:116
frame #14: 0x0000000100794a7c servo`fnfn + 412 at mod.rs:280
frame #15: 0x000000010079539d servo`script::boxed::F.FnBox<A>::call_box(self=0x0000000115c24440, args=<unavailable>) + 77 at boxed.rs:516
frame #16: 0x0000000102981f6e servo`sys::thread::_$LT$impl$GT$::new::thread_start::h906ddce3a93d4852ksw + 142
frame #17: 0x00007fff982c6899 libsystem_pthread.dylib`_pthread_body + 138
frame #18: 0x00007fff982c672a libsystem_pthread.dylib`_pthread_start + 137
frame #19: 0x00007fff982cafc9 libsystem_pthread.dylib`thread_start + 13
Consider the following program:
#[macro_use]
extern crate serde_derive;
extern crate ipc_channel;
#[derive(Serialize, Deserialize)]
struct Foo;
impl Drop for Foo {
fn drop(&mut self) {
println!("drop");
}
}
fn main() {
let (sender, receiver) = ipc_channel::ipc::channel().unwrap();
let foo = Foo;
sender.send(foo).unwrap(); //`foo` consumed here
let _foo2 = receiver.recv().unwrap(); //`_foo2` is expected to drop here
}
From the user perspective, value gets consumed by sender and then reproduced by the receiver. This is expressed as move semantics by the API but doesn't actually behave this way: the program drops Foo
twice here.
I understand that with serialization involved it becomes more difficult to say when the value is destroyed and when it's not. I do find this rather critical, since the user code may have unintended effects in Drop
, and debugging cases like this is not fun. Can the implementation make sure to avoid dropping an object on send, and only drop it if the send failed? Alternatively, something should prevent/warn the user from ever having destructors on the types being sent.
cc @glennw
servo/servo#13054 and servo/servo#10390 demonstrate clear cases where mach imposes size limits on the messages sent. We should break messages that exceed these limits into pieces and reconstruct them on the receiver's end.
We have had cases where Arc<IpcSender<T>>
was added to curb rampant file handle cloning in Servo. Why not do this by default and have IpcSender store an Arc?
warning: unused extern crate [-W unused-extern-crates]
--> /Users/nox/src/servo/../ipc-channel/lib.rs:22:1
22 |> extern crate uuid;
|> ^^^^^^^^^^^^^^^^^^
warning: value assigned to `allocated_buffer` is never read, #[warn(unused_assignments)] on by default
--> /Users/nox/src/servo/../ipc-channel/platform/macos/mod.rs:597:29
597 |> allocated_buffer = None;
|> ^^^^^^^^^^^^^^^^
Is it possible to do zero-copy with ipc-channel today? I've seen OsIpcSharedMemory, but I'm not sure it can be used directly for this purpose.
If it's not possible right now, do you plan to support it? I'm testing some ideas for interprocess video transfer, and zero-copy would be really helpful.
Thanks
Should be straightforward - make a channel, send a value on the sender, drop the sender, and try to receive it from the receiver.
There was an update to the rand
dependency and the version number was updated, but no new crate was actually published. Could a new release be published?
This came out of discussions in #138.
The proposal there is to allow sending borrowed data, e.g. a method send_clone(&data)
with the same semantics assend(data.clone())
but removing the unnecessary copying.
This deals with borrowing as long as the types match up, but doesn't deal with the case where the types disagree, e.g. sending a &[&T]
on a channel of type IpcSender<Vec<T>>
. There is some discussion in #138 about approaches to this.
Just creating this issue to keep track of implementing Windows support.
I've tried to fix the compile issue on alpine/musl and ran into this problem:
libc::msghdr
has two fields that don't exist on non-musl, __pad1
and __pad2
. Since there is no construcutor and these fields are private, msghdr
can not be created.
https://github.com/rust-lang/libc/blob/master/src/unix/notbsd/linux/musl/b64/mod.rs
This currently blocks a musl port due to src/platform/unix/mod.rs. Everything else is trivial to fix.
We got a panic in this code in Servo:
226 Command::Exit(result) => {
227 result.send(()).unwrap();
228 break;
229 }
...
382 pub fn exit(&self) {
383 let (response_chan, response_port) = ipc::channel().unwrap();
384 self.chan.send(Command::Exit(response_chan)).unwrap();
385 response_port.recv().unwrap();
386 }
The panic:
โ thread 'Constellation' panicked at 'called `Result::unwrap()` on an `Err` value: Error { repr: Os { code: 104, message: "Connection reset by peer" } }', ../src/libcore/result.rs:688
This is on Linux.
This test fails on OS X:
#[test]
fn simple_char() {
let ch = Some('a');
let (tx, rx) = ipc::channel().unwrap();
tx.send(ch).unwrap();
let received_ch = rx.recv().unwrap();
assert_eq!(ch, received_ch);
}
yields
---- test::simple_char stdout ----
thread 'test::simple_char' panicked at 'assertion failed: `(left == right)` (left: `Some('a')`, right: `Some('\u{0}')`)', test.rs:394
note: Run with `RUST_BACKTRACE=1` for a backtrace.
failures:
test::simple_char
test result: FAILED. 46 passed; 1 failed; 0 ignored; 0 measured
Right now if I try to build ipc-channel using stable rust (1.12.1 d4f39402a 2016-10-19) on windows, it fails with this error:
error[E0554]: #[feature] may not be used on the stable release channel
--> C:\Users\kats\zspace\mozilla-git\third_party\rust\ipc-channel\src\lib.rs:11:4
|
11 | feature(mpsc_select))]
| ^^^^^^^^^^^^^^^^^^^^^
I'm not entirely sure what mpsc_select is, because that token doesn't appear anywhere else in the repo. Can this be removed, or can be rewritten using stuff that compiles with stable rust?
/cc @Ms2ger (who originally added the line in f4fef7d)
/cc @jrmuizel as FYI
It would be nice to have one-shot routes instead of requiring callers to do option dances.
This first came up in servo/servo#6884.
I'm probably missing it, but where's the code to prevent multiple senders from having the messages interleaved?
i.e. on unix the messages are not sent in a single write call so what code keeps the next sender on a different thread from starting it's writes before the previous sender has finished?
Currently on linux IPCSharedMemory is backed by a file in /tmp
, which is often not tempfs, and ends up doing unnecessary disk IO.
If possible, we should use memfd_create()
to take an existing memory buffer and make it into a file descriptor (which we can then share).
I'm still seeing regular assertions on Linux when using WR - don't have a repro yet sorry. It seems to happen within a minute or so of running - perhaps a resource leak?
thread 'LayoutThread PipelineId { namespace_id: PipelineNamespaceId(0), index: PipelineIndex(0) }' panicked at 'assertion failed: fd >= 0', /home/projects/work/servo/servo/.cargo/git/checkouts/ipc-channel-3840821e54ea11a6/master/platform/linux/mod.rs:765
I'm leaving this largely as a to-do for myself. Once there is some amount of std
support for seL4, this crate would be an easy port to it.
UnixSender
could use shutdown(SHUT_RD)
UnixReceiver
could use shutdown(SHUT_WR)
I'm worried about the file descriptor limit being an ever-present failure mode. Chromium and Gecko multiplex everything over one socket, and I'm beginning to think we should too.
This needs some digging.
(The comment in the source code suggests that it might be a kernel problem -- but I rather doubt it: a lot of the complexity of the Mach IPC system comes precisely from trying very hard to cover such corner cases...)
As evidenced by 988a503 , when transferring raw data with a length not a multiple of 4 through a bytes_channel
, on the MacOS backend, the data is padded with 0
s. Sooner or later this is going to bite someone.
When IPC channel creation fails, we get
thread 'http_loader' panicked at 'called `Result::unwrap()` on an `Err` value: ()', src/libcore/result.rs:732"
hi,
for various reasons I have a project that heavily uses mio, and I want my subprocesses to get orders through ipc_channel, and be notified of new orders in the same event loop they use.
For this, I need to implement the mio::Evented
trait on IpcReceiver
. With access to the underlying file descriptor (it would be very platform dependent, of course), I'd be able to easily integrate it. Unfortunately IpcReceiver.os_receiver
and OsIpcReceiver.fd
are private.
I don't know if it's something you would like to support, I'd be happy to hear your thoughts about it.
Do I understand correctly that ipc-channel
doesn't guarantee that the messages (sent from the same sender) are getting received in the same order? I can observe that on OSX with my application. (Edit: nope, turns out to be late night programming syndrome). If that's the case, is there any way/alternative for me to enforce the order?
See servo/webrender#991.
When using the latest nightly rustc and compiling on the master branch I get the following.
error[E0282]: type annotations needed
--> src/ipc.rs:155:42
|
155 | let mut serializer = bincode::Serializer::new(&mut bytes);
| -------------- ^^^^^^^^^^^^^^^^^^^^^^^^ cannot infer type for `E`
| |
| consider giving `serializer` a type
error: aborting due to previous error
error: Could not compile `ipc-channel`.
Code looks like this:
let (server, server_name) = IpcOneShotServer::new().unwrap();
let mut child = Command::new(ciri_path)
.arg("--log")
.arg(ciri_logging_level)
.arg("--server")
.arg(server_name)
.arg("discover")
.arg(entry.path())
.env("CIRI_FIXTURE_DIR", ciri_fixture_dir)
.env("RUST_BACKTRACE", "1")
.spawn()
.unwrap();
let (_, (suite_name, test_name)): (_, (String, String)) =
server.accept().unwrap();
child.wait().unwrap();
My child
panics in the main thread and dies w/o sending anything. This code then hangs on server.accept()
.
Is it expected? Should I take care of it on my side?
When a recv() is reading a fragmented message, the Linux implementation pushes any interleaved fragments from other messages back to the same file descriptor via sendmsg().
However, since that is being sent to the same fd, if the send buffer is full, deadlock occurs - because the send is occurring on the same thread that should be draining from that fd.
This causes a lot of timeouts in the webrender reftests, and (I suspect) is also responsible for a lot of the Linux timeouts we see on the build machines.
For example, see the backtrace below - thread 72 is blocked in sendmsg since the send buffer is full, but this is called inside of the recv() function for the same fd, so the send buffer will never be emptied.
(gdb) thread 72
[Switching to thread 72 (Thread 0x7fd63a3ff700 (LWP 9553))]
#0 0x00007fd644c875bd in sendmsg () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0 0x00007fd644c875bd in sendmsg () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fd646b5ec41 in platform::linux::recv::h0111fbf8cc03dac2Mxb ()
#2 0x00007fd646b5d34d in platform::linux::UnixReceiver::recv::h76d3513f4fe3a26e2Ra ()
#3 0x00007fd6467af31f in render_backend::RenderBackend::run::h33e3f2df27cc03b3ULo ()
#4 0x00007fd646803bd8 in sys_common::unwind::try::try_fn::h13009946449717054376 ()
#5 0x00007fd64754d959 in __rust_try ()
#6 0x00007fd647549b2c in sys_common::unwind::try::inner_try::h62f00e3d34dbb509D3s ()
#7 0x00007fd646804d8b in boxed::F.FnBox$LT$A$GT$::call_box::h7120465612878348244 ()
#8 0x00007fd64754fc64 in sys::thread::Thread::new::thread_start::hb3a0514b9d87669e1mx ()
#9 0x00007fd642ed3182 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007fd644c8647d in clone () from /lib/x86_64-linux-gnu/libc.so.6
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.