smol-rs/async-io

Async I/O and timers

async-io's Issues

avoid unnecessary poll `source.ready().await` when events happen immediately

ticks indicate whether event have happened, after record ticks we register event to reactor and return Pending. if event happen immediately, tick will be:

let tick = self
            .reactor
            .ticker
            .fetch_add(1, Ordering::SeqCst)
            .wrapping_add(1);

because reactor block on wait, which will cause ticks.0 equal to tick when we wake this task. and we reregister event to reactor and wake again. this time ticks.0 increase 1, and we return Poll::Ready now.

IMO ticks.0 is useless maybe we can just use ticks.1 to indicate whether event has happened. or we can set state[dir].tick = tick + 1.

RUSTSEC-2018-0017: `tempdir` crate has been deprecated; use `tempfile` instead

tempdir crate has been deprecated; use tempfile instead

Details
Status	unmaintained
Package	tempdir
Version	0.3.7
URL	rust-lang-deprecated/tempdir#46
Date	2018-02-13

The tempdir crate has been deprecated
and the functionality is merged into tempfile.

See advisory page for additional details.

Use NT reserve objects

They can improve robustness under certain conditions, as they avoid late failures

implement FusedFuture for Timer

Support for Handles on Windows

Windows has two kinds file descriptors: Handle and Socket. It seems smol::Async currently only support Socket but not yet Handle, which is needed to abstract over APIs such as named pipes. What is the plan for Handle-based APIs in smol?

async-io reactor seems to cause I/O to hung with TLS

I have a simple TLS proxy which is using futures_lite::io::copy to copy source to dest and dest to source. The test does 10 iteration of sending message to between client and server. However, only first iteration works, during second iteration, client send seems to be block

Oct 01 04:48:53.968 DEBUG proxy: client: sleep to give server chance to come up    
Oct 01 04:48:53.968 DEBUG proxy: server: binding    
Oct 01 04:48:53.968 DEBUG proxy: server: successfully binding. waiting for incoming    
Oct 01 04:48:54.168 DEBUG proxy: client: trying to connect    
Oct 01 04:48:54.177 DEBUG proxy: client: got connection. waiting    
Oct 01 04:48:54.177 DEBUG proxy: client: loop 0 sending test message    
Oct 01 04:48:54.178 DEBUG proxy: server: new incoming connection    
Oct 01 04:48:54.178 DEBUG proxy: server: loop 0, received from client: 8 bytes    
Oct 01 04:48:54.178 DEBUG proxy: sever: send back reply: message0reply    
Oct 01 04:48:54.178 DEBUG proxy: client: loop 0, received reply back    
Oct 01 04:48:54.178 DEBUG proxy: client: loop 1 sending test message  

To reproduce this:

git clone https://github.com/infinyon/flv-tls-proxy
cd flv-tls-proxy
RUST_LOG=proxy=debug  cargo test test_proxy

In trace mode, log shows following:

TRACE tokio_util::codec::framed_read: frame decoded from buffer    
Oct 01 04:52:52.420 DEBUG proxy: client: loop 0, received reply back    
Oct 01 04:52:52.420 DEBUG proxy: client: loop 1 sending test message    
Oct 01 04:52:52.420 TRACE tokio_util::codec::framed_write: flushing framed transport    
Oct 01 04:52:52.420 TRACE tokio_util::codec::framed_write: writing; remaining=8    
Oct 01 04:52:52.420 TRACE async_io::driver: main_loop: sleeping for 1000 us    
Oct 01 04:52:52.420 TRACE polling::epoll: interest: epoll_fd=3, fd=8, ev=Event { key: 2, readable: true, writable: false }    
Oct 01 04:52:52.420 TRACE polling::epoll: new events: epoll_fd=3, res=1    
Oct 01 04:52:52.420 TRACE tokio_util::codec::framed_write: framed transport flushed    
Oct 01 04:52:52.420 TRACE polling::epoll: interest: epoll_fd=3, fd=4, ev=Event { key: 18446744073709551615, readable: true, writable: false }    
Oct 01 04:52:52.420 TRACE tokio_util::codec::framed_read: attempting to decode a frame    
Oct 01 04:52:52.420 TRACE async_io::reactor: react: 1 ready wakers    
Oct 01 04:52:52.420 TRACE async_io::driver: block_on: stops hogging the reactor    
Oct 01 04:52:52.420 TRACE polling: Poller::notify()    
Oct 01 04:52:52.420 TRACE polling::epoll: notify: epoll_fd=3, event_fd=4    
Oct 01 04:52:52.420 TRACE async_io::driver: main_loop: notified    
Oct 01 04:52:52.420 TRACE async_io::driver: block_on: waiting on I/O    
Oct 01 04:52:52.420 TRACE async_io::driver: main_loop: sleeping for 50 us    
Oct 01 04:52:52.420 TRACE async_io::reactor: process_timers: 0 ready wakers    
Oct 01 04:52:52.420 TRACE polling::epoll: interest: epoll_fd=3, fd=7, ev=Event { key: 1, readable: true, writable: false }    
Oct 01 04:52:52.420 TRACE polling: Poller::wait(_, None)    
Oct 01 04:52:52.420 TRACE async_io::driver: block_on: sleep until notification    
Oct 01 04:52:52.420 TRACE polling::epoll: interest: epoll_fd=3, fd=9, ev=Event { key: 3, readable: true, writable: false }    
Oct 01 04:52:52.420 TRACE polling::epoll: wait: epoll_fd=3, timeout=None    
Oct 01 04:52:52.421 TRACE async_io::driver: main_loop: sleeping for 75 us    
Oct 01 04:52:52.421 TRACE polling::epoll: interest: epoll_fd=3, fd=5, ev=Event { key: 18446744073709551615, readable: true, writable: false }    
Oct 01 04:52:52.421 TRACE async_io::driver: block_on: sleep until notification    
Oct 01 04:52:52.421 TRACE polling::epoll: new events: epoll_fd=3, res=1    
Oct 01 04:52:52.421 TRACE polling::epoll: interest: epoll_fd=3, fd=4, ev=Event { key: 18446744073709551615, readable: true, writable: false }    
Oct 01 04:52:52.421 TRACE async_io::reactor: process_timers: 0 ready wakers    
Oct 01 04:52:52.421 TRACE async_io::driver: main_loop: sleeping for 100 us    
Oct 01 04:52:52.421 TRACE async_io::reactor: react: 0 ready wakers    
Oct 01 04:52:52.421 TRACE async_io::driver: block_on: stops hogging the reactor    
Oct 01 04:52:52.421 TRACE async_io::driver: main_loop: notified    
Oct 01 04:52:52.421 TRACE async_io::driver: main_loop: waiting on I/O    
Oct 01 04:52:52.421 TRACE async_io::reactor: process_timers: 0 ready wakers    
Oct 01 04:52:52.421 TRACE polling: Poller::wait(_, None)    
Oct 01 04:52:52.421 TRACE polling::epoll: wait: epoll_fd=3, timeout=None    
Oct 01 04:52:52.421 TRACE polling::epoll: interest: epoll_fd=3, fd=5, ev=Event { key: 18446744073709551615, readable: true, writable: false }  

make `TcpStream` return by `Async<TcpListener>::accept()` nonblocking mode?

it seems that TcpStream return by accept is not in noblocking mode.

tcp_connect test failed on FreeBSD on Cirrus CI

https://cirrus-ci.com/task/5088564308869120

---- tcp_connect stdout ----
thread 'tcp_connect' panicked at 'assertion failed: `(left == right)`
  left: `TimedOut`,
 right: `ConnectionRefused`', tests/async.rs:57:9
stack backtrace:
   0: rust_begin_unwind
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/panicking.rs:575:5
   1: core::panicking::panic_fmt
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/panicking.rs:65:14
   2: core::panicking::assert_failed_inner
   3: core::panicking::assert_failed
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/panicking.rs:203:5
   4: async::tcp_connect::{{closure}}
             at ./tests/async.rs:57:9
   5: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/future/mod.rs:91:19
   6: futures_lite::future::block_on::{{closure}}
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/futures-lite-1.12.0/src/future.rs:89:27
   7: std::thread::local::LocalKey<T>::try_with
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/thread/local.rs:446:16
   8: std::thread::local::LocalKey<T>::with
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/std/src/thread/local.rs:422:9
   9: futures_lite::future::block_on
             at /.cargo/registry/src/github.com-1ecc6299db9ec823/futures-lite-1.12.0/src/future.rs:79:5
  10: async::tcp_connect
             at ./tests/async.rs:38:5
  11: async::tcp_connect::{{closure}}
             at ./tests/async.rs:37:1
  12: core::ops::function::FnOnce::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:251:5
  13: core::ops::function::FnOnce::call_once
             at /rustc/90743e7298aca107ddaa0c202a4d3604e29bfeb6/library/core/src/ops/function.rs:251:5

note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
failures:
    tcp_connect

test result: FAILED. 12 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 75.10s

error: test failed, to rerun pass `--test async`

Cutting a release to allow use of polling 2.6.0

Polling 2.6.0 depends on windows-sys 0.45.0, but async-io 1.12.0 depends on windows-sys 0.42.0. Would it be possible to make a new release of async-io, which has already migrated from windows-sys to rustix? That would let us use async-io on some of the new platforms supported by polling.

Port to windows-sys

async-io/Cargo.toml

Line 39 in 823d6f1

winapi = { version = "0.3.9", features = ["winsock2"] }

Related: smol-rs/async-process#27, smol-rs/polling#46

Interval API

It would be great if Timer was not just a Future but also a Stream so that it could act as a "ticker" delivering events at regular intervals.

Proposed API:

impl Timer {
    fn interval(period: Duration) -> Timer;
    fn interval_at(start: Instant, period: Duration) -> Timer;

    fn set_interval(&mut self, period: Duration);
    fn set_interval_at(&mut self, start: Instant, period: Duration);
}

impl Stream for Timer {
    type Item = Instant;
}

Thoughts or suggestions?

`Timer` is paused on Android in "doze" state

Because std::time::Instant is implemented using libc::clock_gettime(CLOCK_MONOTONIC) and this time does not advance on Android while the app is in "doze" state, the timer may be effectively paused on Android: https://users.rust-lang.org/t/std-now-with-android/41774

One way to fix this is to use SystemTime instead and handle time occasionally going backwards properly. Another is to lift this issue to the standard library so it uses Android-specific ioctl on /dev/alarm as described here: https://users.rust-lang.org/t/std-now-with-android/41774/2

[Question]: Why Registration does not contain a `BorrowedFd` instead of a `RawFd`?

When looking at the Registration type for unix , I stumbled across the required invariant:

This describes a valid file descriptor that has not been closed. It will not be closed while this object is alive.

and it looks a lot to what is guarantee by the BorrowedFd type:

This has a lifetime parameter to tie it to the lifetime of something that owns the file descriptor. For the duration of that lifetime, it is guaranteed that nobody will close the file descriptor.

So, I thought naively this is maybe a valid design that will avoid the unsafe and ensure the invariant is respected:

pub struct Registration<'fd> {
    fd: BorrowedFd<'fd>,
}

But I'm sure you are well aware about BorrowdFd, so my question is, why have you decided to use a RawFd and unsafe instead of a BorrowedFd?

bump up libc version

async-io/Cargo.toml

Line 31 in 153c732

libc = "0.2.77"

It seems that libc has moved on a lot of new versions, and the versions beyond 0.2.121 no longer can co-exist with the version (up to libc v0.2.117) used by async-io.

Is there any plan to bump up the libc version? Thanks.

Use rustix' use-libc-auxv feature?

I ran into the problem that rustix panics when using async-io inside a 32-bit Linux sandbox with no /proc available. Turns out that the default feature use-libc-auxv of rustix would avoid that. Is the feature intentionally disabled?

Linker error in stable-x86_64-pc-windows-gnu toolchain

Since async-std switched to smol I'm unable to compile my project in the stable-gnu toolchain. It's working perfectly in stable-msvc as well as native linux compilation.

I'm getting the following error:
error: linking with `x86_64-w64-mingw32-gcc` failed: exit code: 1

And multiple missing functions like this:
In function `smol::reactor::sys::Reactor::new::h4fc4018e596093ae':
...\src\github.com-1ecc6299db9ec823\smol-0.1.18\src/reactor.rs:741: undefined reference to `epoll_create1'

For the functions epoll_create1, epoll_ctl, and epoll_wait.

I'm using the default compiler that comes with installing the toolchain with rustup. Maybe I'm missing a library?

Avoid spawning the async-io thread if possible

In most programs that use async-io, the user will simply run block_on at the top level. However, this will still spawn the async-io thread despite it not being necessary. Would it be possible to avoid that?

Why poller add Event::none(key) for registers an I/O?

https://github.com/smol-rs/async-io/blob/master/src/reactor.rs#L109

support multiple async-io instances

There are rare occasions (like when running with threads in different network namespaces during a network simulation) where it would be beneficial to explicitly create an async-io reactor to make sure that the reactor runs in the correct network namespace

Support waitable objects on Windows 10

There is an (undocumented) NtAssociateWaitCompletionPacket syscall that can be used for the purpose.

support tokio reactor

I am very fond of async-io and use it in many projects. However tokio users aren't very happy with the additional thread. Is there any way that we can wrap the tokio evented trait or whatever it's called these days and add a tokio feature flag?

expose register_reader and register_writer so that manual futures can be implemented

since rust doesn't support self referential structs there needs to be an alternative to async fn next(&self). Since self isn't owned there is no way to get it into a struct so that it can be polled.

Please consider supporting AsFd

Currently, most methods async_io::Async requires the type it wraps to implement AsRawFd, or AsRawSocket on Windows.

The standard library now supports the safe alternative AsFd (or AsSocket on Windows), and async-io has some support for this. However, it doesn't seem possible to create an Async<T> for a type that implements AsFd but not AsRawFd.

I have types I'd love to use Async with, and I'd like to drop the implementation of AsRawFd for those types. (This would also eliminate unsafe code.)

Please consider adding a constructor for Async<T> where T: AsFd.

Performance issue with `Timer`

This 'regression' appears a while ago in criner, and apparently went unnoticed when switching from tokio::time::Delay to async_io::Timer.

The picture above shows a criner process doing nothing, consuming 10 to 15% of CPU for displaying 4 progress bars which update once per second. The expensive part of this used to be the drawing itself, but now it appears to be dominated by the condvar in parking.

In smol 0.1.5 using smol::Timer, this looked like that:

All that shows up is the drawing of the TUI.

Having had a look myself I am absolutely puzzled as to why that is - the code looks very similar.

Any hint at how to solve this would be greatly appreciated.
Thank you

Meta

• 'utils.rs' on a criner branch that works as expected, without performance issues
  • It's noteworthy that criner didn't change in the relevant portions of the code responsible for the timer invocations see utils.rs.

• the version of 'utils.rs' that suffers from 'slow' timers

read/write flags should remove EPOLLERR

EPOLLERR
              Error condition happened on the associated file descriptor.
              This event is also reported for the write end of a pipe when
              the read end has been closed.

              epoll_wait(2) will always report for this event; it is not
              necessary to set it in events when calling epoll_ctl().

io_uring support

Mostly for tech porn reasons, it's new and shiny. Apparently it's also faster. Not sure if it fits in async-io, because it works very different from epoll from what I understand by reading withoutboats blog.

Missing waker.wake events

I was going though a libp2p tutorial and found that I couldn't reliably run libp2p-lookup against libp2p-relay_v2. The libp2p-lookup would work the first time I ran libp2p-relay_v2 after a reboot. But libp2p-lookup would fail if I stopped and restarted libp2p-relay_v2. Here is a discussion on libp2p repo. Initially I thought it was a performance problem as debug builds could fail but release builds worked. Also, I didn't see the problem on my desktop machine only on a very small Digital Ocean VM with 1CPU and 1GB ram. Also, recently I tested a tokio build of libp2p-relay_v2 and it worked fine.

By using RUST_LOG=trace the "timeout" symptom would "always" fail and I started adding copious amounts of logging trying to narrow down the problem. To shorten the story, the problem appears to be that async-io can sometimes miss waker.wake events and is resolved by making a one line change to async-io in Source::poll_ready.

I change:

state[dir].ticks = Some((Reactor::get().ticker(), state[dir].tick));

to:

state[dir].ticks = Some((state[dir].tick, 0)

My analysis of the problem is that using the current Reactor::ticker value in ticks is not always correct. It is my understanding that when placing a ticker value into Direction::ticks there is a guarantee that the waker.wake call has completed and the associated task has been run. The problem is the increment of the ticker value is in ReactorLock::react is typically running on a different CPU thread than the Source::poll_ready (called by poll_readable, poll_writeable), which is saving the ticker value in ticks as show in the above code snippet.

Because the increment and use of ticker value are on different threads the required guarantee can not be full filled and I'm proposing the above fix which only uses the a current state[dir].tick and not the ticker value. Another possible solution is to set state[dir].ticks = None but the proposed solution is a "smaller" change so I've chosen it.

Of course I fully expect async-io experts may provide other solutions or even identify that problem as something else all together. But, at the moment, this solution does resolve the problem.

Async<TcpListener>::accept on OS X may miss connection attempts

It appears that Async::<TcpListener>::accept may have a bug on OS X where it can miss connection attempts. I was able to reduce this bug down to this:

#[test]
fn tcp_connect_bug() {
    for i in 1.. {
        println!("attempt {i}");

        future::block_on(async {
            let listener = Async::<TcpListener>::bind(([127, 0, 0, 1], 8081)).unwrap();
            let addr = listener.get_ref().local_addr().unwrap();

            let task_client = spawn(async move { Async::<TcpStream>::connect(addr).await });
            let task_server = spawn(async move { listener.accept().await });

            let stream_client = task_client.await.unwrap();
            let stream_server = task_server.await.unwrap().0;
            let stream_server = listener.accept().await.unwrap().0;
            let stream_client = task_client.await.unwrap();

            assert_eq!(
                stream_client.get_ref().peer_addr().unwrap(),
                stream_server.get_ref().local_addr().unwrap(),
            );
            assert_eq!(
                stream_client.get_ref().peer_addr().unwrap(),
                stream_server.get_ref().local_addr().unwrap(),
            );
        })
    }
}

On my OS X laptop, it will succeeds for about approximately ~16000 loops before I get:

...
attempt 16277
thread 'tcp_connect_bug' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 60, kind: TimedOut, message: "Operation timed out" }', tests/async.rs:48:51
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
test tcp_connect_bug ... FAILED

I don't get any time out running this on a Linux machine after ~100000ish cycles.

Add cooperative task yielding to this runtime

tokio uses cooperative task yielding to improve tail latencies in the circumstance that a resource polled in a loop fails to yield, leading to starvation of other tasks. Something similar could be set up for async-io as well.

The Reactor already has a concept of "ticks", which are used to coordinate polling. My idea would be that each Source keeps track of the tick which it last returned Poll::Pending. If it compares tick and this theoretical last_pending_tick value and finds that the difference between the two is a large value (say, 10_000, but this could use some tweaking), pooling the Source automatically returns Pending.

Any thoughts on this? This could be a global counter as well, or maybe a thread-local one.

I would be willing to implement this if the functionality is desired.

Does `read_with` only take idempotent operations?

I was reading the implementation behind read_with and I was wondering - wouldn't the loop in the method potentially cause the op, such as l.accept() to be called multiple times? Are all methods fed to read_with meant to be idempotent?

    /// # Examples
    ///
    /// ```no_run
    /// use async_io::Async;
    /// use std::net::TcpListener;
    ///
    /// # futures_lite::future::block_on(async {
    /// let listener = Async::<TcpListener>::bind(([127, 0, 0, 1], 0))?;
    ///
    /// // Accept a new client asynchronously.
    /// let (stream, addr) = listener.read_with(|l| l.accept()).await?;
    /// # std::io::Result::Ok(()) });
    /// ```
    pub async fn read_with<R>(&self, op: impl FnMut(&T) -> io::Result<R>) -> io::Result<R> {
        let mut op = op;
        loop {
            match op(self.get_ref()) {
                Err(err) if err.kind() == io::ErrorKind::WouldBlock => {}
                res => return res,
            }
            optimistic(self.readable()).await?;
        }
    }

I do notice that methods like accept, when set to nonblocking won't wait for a new TCP connection to be established:

#[test]
fn test_accept() {
    let l = std::net::TcpListener::bind("127.0.0.1:8080").unwrap();
    l.set_nonblocking(true).unwrap();

    loop {
        let res = l.accept();
        match res {
            Ok(_) => break,
            Err(_) => {}
        }
    }
    println!("Finished");
}

A blank Timer that never fires

How about a default Timer::new() constructor that makes a timer that never fires? Of course, that timer would be possible to update with set_at(), set_after() and so on.

I'm inclined to just add this constructor (and a Default impl for Timer) but am wondering if anyone has objections?

Support creating `Async` without setting non-blocking mode

I have a project aims to be pluggable into different async executors through FDs/Handles. It works great, I have a PoC working with tokio. It also works on linux with smol. Example of how I'm using Async.

However, on macOS I'm getting ENOTTY, which stems from Async::new unconditionally running fcntl to set O_NONBLOCK, even on FDs that do not support it (like kqueue).

My usecase of Async is purely to have async-io poll for read-readiness of the FD, and have my task woken up. That's the whole extent of it. Maybe it would make sense to expose a different type just for polling, but the simpler way would be to add an overload for Async::new that allows making fcntl call conditional.

Explicit `socket2` support

The library internally already uses the socket2 crate. It would be cool if this functionality was also exposed in a convenient way as part of the public API. For example, I'd like to have an impl Async<socket2::Socket> that provides methods like bind etc.

Async::<UnixStream>::connect returning with io::ErrorKind::WouldBlock

I am in the process of moving swayipc from async-std to async-io. While running the unit tests I encountered tests failing with io::ErrorKind::WouldBlock. After re-running the current tests with the latest version of async-std I encountered the same behaviour (since async-std moved to async-io). The unit tests run without failing on async-std version 1.5 since this is the last version using mio-uds. I extracted an example on how to reproduce this bug:

use async_io::Async;
use futures_lite::future;
use std::os::unix::net::UnixStream;
use std::thread;

fn main() {
    let mut handles = Vec::new();
    for _ in 1..100 {
        let handle = thread::spawn(|| {
            future::block_on(async {
                Async::<UnixStream>::connect("pathtosomeunixsocket").await
            })
        });
        handles.push(handle);
    }
    for handle in handles {
        if let Err(error) = handle.join().unwrap() {
            dbg!(error);
        }
    }
}

RUSTSEC-2021-0127: serde_cbor is unmaintained

serde_cbor is unmaintained

Details
Status	unmaintained
Package	serde_cbor
Version	0.11.2
URL	https://github.com/pyfisch/cbor
Date	2021-08-15

The serde_cbor crate is unmaintained. The author has archived the github repository.

Alternatives proposed by the author:

• ciborium
• minicbor

See advisory page for additional details.

Async<TcpListener>::accept removes interest on drop, and no poll_* based alternative

The source of async-std has the following snippet:

impl<'a> Stream for Incoming<'a> {
    type Item = io::Result<TcpStream>;

    fn poll_next(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> {
        let future = self.0.accept();
        pin_utils::pin_mut!(future);

        let (socket, _) = futures_core::ready!(future.poll(cx))?;
        Poll::Ready(Some(Ok(socket)))
    }
}

link

In the latest release of async-io, the accept future was changed to release interest when the future is dropped, which breaks the above implementation. And as far as I can see, there's no poll_* based alternative for waiting for read-readiness on the Async type.

Originally reported to async-std as async-rs/async-std#888

RUSTSEC-2021-0145: Potential unaligned read

Potential unaligned read

Details
Status	unsound
Package	atty
Version	0.2.14
URL	softprops/atty#50
Date	2021-07-04

On windows, atty dereferences a potentially unaligned pointer.

In practice however, the pointer won't be unaligned unless a custom global allocator is used.

In particular, the System allocator on windows uses HeapAlloc, which guarantees a large enough alignment.

atty is Unmaintained

A Pull Request with a fix has been provided over a year ago but the maintainer seems to be unreachable.

Last release of atty was almost 3 years ago.

Possible Alternative(s)

The below list has not been vetted in any way and may or may not contain alternatives;

• is-terminal
• std::io::IsTerminal nightly-only experimental

See advisory page for additional details.

Fix panic in `Timer` with large durations

See async-rs/async-std#1037. We should probably use checked_add when calculating the instant instead of regular addition.

WouldBlock bubbling up to the calling code

I have an issue with async-native-tls together with async-std, and looking into the backtrace, it points to somewhere in this crate.

Our crate (tiberius) runs tests just fine with tokio and async-std 1.6.3, but updating async-std to 1.6.4 or later randomly throws Io(Kind(WouldBlock)). We have a few hundred tests, and 1-5 of them randomly fail into this error. The error happens when calling connect from TlsConnector (async-native-tls 0.3.3) and passing a TcpStream to it from async-std 1.6.5 (or 1.6.4).

As I said, when passing a TcpStream from async-std 1.6.3 or a Compat version from tokio 0.2.x, the errors never happen. Hopefully this is the right crate to create the issue!

Here's a full backtrace from one of the failing tests:

---- transactions_async_std stdout ----
thread 'transactions_async_std' panicked at 'called `Result::unwrap()` on an `Err` value: Ssl(Error { code: ErrorCode(5), cause: Some(Io(Kind(WouldBlock))) }, X509VerifyResult { code: 18, error: "self signed certificate" })', /home/pimeys/code/tiberius/src/client/connection.rs:387:22
stack backtrace:
   0:     0x55f6b0a016d5 - backtrace::backtrace::libunwind::trace::h14d338b30b3ea0a7
                               at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.46/src/backtrace/libunwind.rs:86
   1:     0x55f6b0a016d5 - backtrace::backtrace::trace_unsynchronized::h73ea91d74a3fd67f
                               at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.46/src/backtrace/mod.rs:66
   2:     0x55f6b0a016d5 - std::sys_common::backtrace::_print_fmt::hd42948c952866e12
                               at src/libstd/sys_common/backtrace.rs:78
   3:     0x55f6b0a016d5 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::ha8f928866ff7571e
                               at src/libstd/sys_common/backtrace.rs:59
   4:     0x55f6b0a2b75c - core::fmt::write::he0c1e5f7426d2718
                               at src/libcore/fmt/mod.rs:1076
   5:     0x55f6b073d675 - std::io::Write::write_fmt::h1152922039931516
                               at /rustc/04488afe34512aa4c33566eb16d8c912a3ae04f9/src/libstd/io/mod.rs:1537
   6:     0x55f6b09f9891 - std::io::impls::<impl std::io::Write for alloc::boxed::Box<W>>::write_fmt::ha927d7e5f294bce5
                               at src/libstd/io/impls.rs:176
   7:     0x55f6b0a03e60 - std::sys_common::backtrace::_print::hfc0110703f3696fd
                               at src/libstd/sys_common/backtrace.rs:62
   8:     0x55f6b0a03e60 - std::sys_common::backtrace::print::h3f77c6990ddfaa22
                               at src/libstd/sys_common/backtrace.rs:49
   9:     0x55f6b0a03e60 - std::panicking::default_hook::{{closure}}::heae49580a8d62d75
                               at src/libstd/panicking.rs:198
  10:     0x55f6b0a03b5a - std::panicking::default_hook::hecc34e3f729e213c
                               at src/libstd/panicking.rs:214
  11:     0x55f6b0a044a3 - std::panicking::rust_panic_with_hook::he82f5d0644692441
                               at src/libstd/panicking.rs:526
  12:     0x55f6b0a0409b - rust_begin_unwind
                               at src/libstd/panicking.rs:437
  13:     0x55f6b0a296c1 - core::panicking::panic_fmt::h09c929f06bb87c98
                               at src/libcore/panicking.rs:85
  14:     0x55f6b0a294e3 - core::option::expect_none_failed::h188f17af6c9f404b
                               at src/libcore/option.rs:1269
  15:     0x55f6b00dbbcc - core::result::Result<T,E>::unwrap::h0d451077fbe85e9f
                               at /home/pimeys/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/result.rs:1005
  16:     0x55f6b053bb63 - tiberius::client::connection::Connection<S>::tls_handshake::{{closure}}::h683d4d491de77824
                               at /home/pimeys/code/tiberius/src/client/connection.rs:384
  17:     0x55f6b02a6072 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::hf56e4ae22867706d
                               at /home/pimeys/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/future/mod.rs:78
  18:     0x55f6b0562bc9 - tiberius::client::connection::Connection<S>::connect::{{closure}}::hc5b1351ccc0c3fa2
                               at /home/pimeys/code/tiberius/src/client/connection.rs:84
  19:     0x55f6b02847d2 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h1935640bb1256dc9
                               at /home/pimeys/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/future/mod.rs:78
  20:     0x55f6b033b757 - tiberius::client::Client<S>::connect::{{closure}}::h4f4a431e44ae57ad
                               at /home/pimeys/code/tiberius/src/client.rs:63
  21:     0x55f6b028e432 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h59693abadb793728
                               at /home/pimeys/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/future/mod.rs:78
  22:     0x55f6b03bede2 - query::transactions_async_std::{{closure}}::hc7d85b6e3f838d65
                               at tests/query.rs:50
  23:     0x55f6b0285312 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h1c531d3b875327d1
                               at /home/pimeys/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/future/mod.rs:78
  24:     0x55f6b018c94e - <async_std::task::builder::SupportTaskLocals<F> as core::future::future::Future>::poll::{{closure}}::h865bd302a785556b
                               at /home/pimeys/.cargo/registry/src/github.com-1ecc6299db9ec823/async-std-1.6.5/src/task/builder.rs:199
  25:     0x55f6b01e82c7 - async_std::task::task_locals_wrapper::TaskLocalsWrapper::set_current::{{closure}}::ha6f4bab892b612bd
                               at /home/pimeys/.cargo/registry/src/github.com-1ecc6299db9ec823/async-std-1.6.5/src/task/task_locals_wrapper.rs:60
  26:     0x55f6b00c3781 - std::thread::local::LocalKey<T>::try_with::hb661b4d0c7a4f06d
                               at /home/pimeys/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/thread/local.rs:263
  27:     0x55f6b007d19e - std::thread::local::LocalKey<T>::with::h0c21588fbe2ca350
                               at /home/pimeys/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/thread/local.rs:239
  28:     0x55f6b01dffef - async_std::task::task_locals_wrapper::TaskLocalsWrapper::set_current::h16226ade7ae65d25
                               at /home/pimeys/.cargo/registry/src/github.com-1ecc6299db9ec823/async-std-1.6.5/src/task/task_locals_wrapper.rs:55
  29:     0x55f6b018b28b - <async_std::task::builder::SupportTaskLocals<F> as core::future::future::Future>::poll::h7cd9e5be02a3bc58
                               at /home/pimeys/.cargo/registry/src/github.com-1ecc6299db9ec823/async-std-1.6.5/src/task/builder.rs:197
  30:     0x55f6b03b140f - <futures_lite::future::Or<F1,F2> as core::future::future::Future>::poll::h0bd82c5f22cb39ad
                               at /home/pimeys/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-lite-1.8.0/src/future.rs:528
  31:     0x55f6b02e8ac8 - async_executor::Executor::run::{{closure}}::hf7027d9be0e1c9d8
                               at /home/pimeys/.cargo/registry/src/github.com-1ecc6299db9ec823/async-executor-1.3.0/src/lib.rs:214
  32:     0x55f6b0294bb2 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h7f4740670e789cad
                               at /home/pimeys/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/future/mod.rs:78
  33:     0x55f6b02c9e8d - async_executor::LocalExecutor::run::{{closure}}::h3a163a88dc05d212
                               at /home/pimeys/.cargo/registry/src/github.com-1ecc6299db9ec823/async-executor-1.3.0/src/lib.rs:393
  34:     0x55f6b028a0b2 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h39ca6bd8d2b323b3
                               at /home/pimeys/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/future/mod.rs:78
  35:     0x55f6b0215a95 - async_io::driver::block_on::h517fd1baac558c2d
                               at /home/pimeys/.cargo/registry/src/github.com-1ecc6299db9ec823/async-io-1.1.4/src/driver.rs:142
  36:     0x55f6b039a373 - async_global_executor::reactor::block_on::ha8fa1557f5aca4b1
                               at /home/pimeys/.cargo/registry/src/github.com-1ecc6299db9ec823/async-global-executor-1.3.0/src/lib.rs:53
  37:     0x55f6b02fa439 - async_global_executor::block_on::{{closure}}::ha076dcfa8bc56aa5
                               at /home/pimeys/.cargo/registry/src/github.com-1ecc6299db9ec823/async-global-executor-1.3.0/src/lib.rs:179
  38:     0x55f6b00c6d45 - std::thread::local::LocalKey<T>::try_with::hc575102fb44c4620
                               at /home/pimeys/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/thread/local.rs:263
  39:     0x55f6b0091845 - std::thread::local::LocalKey<T>::with::headb35e05fa41fab
                               at /home/pimeys/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/thread/local.rs:239
  40:     0x55f6b02f7ee0 - async_global_executor::block_on::h13eb69a060da3402
                               at /home/pimeys/.cargo/registry/src/github.com-1ecc6299db9ec823/async-global-executor-1.3.0/src/lib.rs:179
  41:     0x55f6b01ab8ab - async_std::task::builder::Builder::blocking::{{closure}}::{{closure}}::h8ce56e42b4a651ce
                               at /home/pimeys/.cargo/registry/src/github.com-1ecc6299db9ec823/async-std-1.6.5/src/task/builder.rs:171
  42:     0x55f6b01e6f9a - async_std::task::task_locals_wrapper::TaskLocalsWrapper::set_current::{{closure}}::h7e827a3b9f989c97
                               at /home/pimeys/.cargo/registry/src/github.com-1ecc6299db9ec823/async-std-1.6.5/src/task/task_locals_wrapper.rs:60
  43:     0x55f6b009cea1 - std::thread::local::LocalKey<T>::try_with::h2b7cd734ad2bfc51
                               at /home/pimeys/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/thread/local.rs:263
  44:     0x55f6b007e8c5 - std::thread::local::LocalKey<T>::with::h1b5335748a459c1e
                               at /home/pimeys/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/thread/local.rs:239
  45:     0x55f6b01e0cb5 - async_std::task::task_locals_wrapper::TaskLocalsWrapper::set_current::h52e3c6d593cdc45f
                               at /home/pimeys/.cargo/registry/src/github.com-1ecc6299db9ec823/async-std-1.6.5/src/task/task_locals_wrapper.rs:55
  46:     0x55f6b01a8467 - async_std::task::builder::Builder::blocking::{{closure}}::hefc8a66421fbf332
                               at /home/pimeys/.cargo/registry/src/github.com-1ecc6299db9ec823/async-std-1.6.5/src/task/builder.rs:168
  47:     0x55f6b00d69cd - std::thread::local::LocalKey<T>::try_with::hfe7cb36aa29f4083
                               at /home/pimeys/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/thread/local.rs:263
  48:     0x55f6b0086eb5 - std::thread::local::LocalKey<T>::with::h6fb1471021bc304a
                               at /home/pimeys/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/thread/local.rs:239
  49:     0x55f6b019b72c - async_std::task::builder::Builder::blocking::h6b201419829dfbb8
                               at /home/pimeys/.cargo/registry/src/github.com-1ecc6299db9ec823/async-std-1.6.5/src/task/builder.rs:161
  50:     0x55f6b0201b27 - async_std::task::block_on::block_on::h17621ea76d951fe8
                               at /home/pimeys/.cargo/registry/src/github.com-1ecc6299db9ec823/async-std-1.6.5/src/task/block_on.rs:33
  51:     0x55f6b03283c2 - query::transactions_async_std::h891bf3802642b022
                               at tests/query.rs:50
  52:     0x55f6b03be5a1 - query::transactions_async_std::{{closure}}::h17ead2905b73b6af
                               at tests/query.rs:50
  53:     0x55f6b04e0e7e - core::ops::function::FnOnce::call_once::he8d8373c6c2f303b
                               at /home/pimeys/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ops/function.rs:233
  54:     0x55f6b0764bec - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::h6633cb15d0d76942
                               at /rustc/04488afe34512aa4c33566eb16d8c912a3ae04f9/src/liballoc/boxed.rs:1081
  55:     0x55f6b0764bec - <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h1e20120def172c5c
                               at /rustc/04488afe34512aa4c33566eb16d8c912a3ae04f9/src/libstd/panic.rs:318
  56:     0x55f6b0764bec - std::panicking::try::do_call::hcc2ec3bbb75e9316
                               at /rustc/04488afe34512aa4c33566eb16d8c912a3ae04f9/src/libstd/panicking.rs:348
  57:     0x55f6b0764bec - std::panicking::try::h94eaebaaa7dd6f41
                               at /rustc/04488afe34512aa4c33566eb16d8c912a3ae04f9/src/libstd/panicking.rs:325
  58:     0x55f6b0764bec - std::panic::catch_unwind::h151c07c08497cf8b
                               at /rustc/04488afe34512aa4c33566eb16d8c912a3ae04f9/src/libstd/panic.rs:394
  59:     0x55f6b0764bec - test::run_test_in_process::hd082de93b1922c89
                               at src/libtest/lib.rs:541
  60:     0x55f6b0764bec - test::run_test::run_test_inner::{{closure}}::h22369c9424e5ab3a
                               at src/libtest/lib.rs:450
  61:     0x55f6b073cb66 - std::sys_common::backtrace::__rust_begin_short_backtrace::h0660a89f67243e05
                               at /rustc/04488afe34512aa4c33566eb16d8c912a3ae04f9/src/libstd/sys_common/backtrace.rs:130
  62:     0x55f6b0741c15 - std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}}::h8bb2049509aa1add
                               at /rustc/04488afe34512aa4c33566eb16d8c912a3ae04f9/src/libstd/thread/mod.rs:475
  63:     0x55f6b0741c15 - <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::hb49f3484cb9c3dd3
                               at /rustc/04488afe34512aa4c33566eb16d8c912a3ae04f9/src/libstd/panic.rs:318
  64:     0x55f6b0741c15 - std::panicking::try::do_call::ha910a12d1577339b
                               at /rustc/04488afe34512aa4c33566eb16d8c912a3ae04f9/src/libstd/panicking.rs:348
  65:     0x55f6b0741c15 - std::panicking::try::hd8b3d620360e55fa
                               at /rustc/04488afe34512aa4c33566eb16d8c912a3ae04f9/src/libstd/panicking.rs:325
  66:     0x55f6b0741c15 - std::panic::catch_unwind::h475454730ea43154
                               at /rustc/04488afe34512aa4c33566eb16d8c912a3ae04f9/src/libstd/panic.rs:394
  67:     0x55f6b0741c15 - std::thread::Builder::spawn_unchecked::{{closure}}::h2407d9379d805151
                               at /rustc/04488afe34512aa4c33566eb16d8c912a3ae04f9/src/libstd/thread/mod.rs:474
  68:     0x55f6b0741c15 - core::ops::function::FnOnce::call_once{{vtable.shim}}::haef9f772dfab225e
                               at /rustc/04488afe34512aa4c33566eb16d8c912a3ae04f9/src/libcore/ops/function.rs:233
  69:     0x55f6b0a0b1aa - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::hd2b3bc04af94a84f
                               at /rustc/04488afe34512aa4c33566eb16d8c912a3ae04f9/src/liballoc/boxed.rs:1081
  70:     0x55f6b0a0b1aa - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::h1044417e186e567a
                               at /rustc/04488afe34512aa4c33566eb16d8c912a3ae04f9/src/liballoc/boxed.rs:1081
  71:     0x55f6b0a0b1aa - std::sys::unix::thread::Thread::new::thread_start::h276e6ca033938925
                               at src/libstd/sys/unix/thread.rs:87
  72:     0x7fd8698f3609 - start_thread
                               at /build/glibc-ZN95T4/glibc-2.31/nptl/pthread_create.c:477
  73:     0x7fd8697fd293 - __clone
  74:                0x0 - <unknown>

Make sure we can't deregister non-registered sources

Calloop recently found a bug where it's possible to deregister a source by trying to register it twice. See Smithay/calloop#153 for more information.

I don't think async-io is affected by this bug, but we should add a test case to make sure.

async-io now fails to compile on 1.46

With the new release of once_cell 1.15, the MSRV had been increased to 1.56 and the edition has been bumped to 2021. This means async-io, which depends on once_cell, no longer builds on 1.46:

$ cargo +1.46 check         
warning: unused manifest key: package.rust-version
    Updating crates.io index
  Downloaded once_cell v1.15.0
error: failed to parse manifest at `/home/jtnunley/.cargo/registry/src/github.com-1ecc6299db9ec823/once_cell-1.15.0/Cargo.toml`

Caused by:
  failed to parse the `edition` key

Caused by:
  this version of Cargo is older than the `2021` edition, and only supports `2015` and `2018` editions.

I see two options:

• Pin our once_cell version to 1.14.x. This would keep our MSRV at 1.46, but prevent us from receiving any new once_cell updates.
• Bump our MSRV to 1.56. However, we would no longer be able to build this project on unmodified Debian.

See also: matklad/once_cell#201

Remove the waker-fn dependency

It should be replaced by an object that implements Wake.

Optimization: skip optimistic read/write if there are wakers waiting for I/O resource to become readable/writable

Right now, inside read_with()/write_with() we aways optimistically attempt a read/write operation. This can be costly in loops that select! over I/O operations.

However, we know whether the resource is ready simply by looking at wakers. If there are wakers waiting for the I/O handle to become readable/writable, we can skip the optimistic attempt.

Mach port support

I would like to watch Mach ports for new messages using smol.

There are basically two ways to go about watching a Mach port. One, which would work on any Mach system, is to have the reactor use Mach port sets in place of current epoll/kqueue.

Another way, which only works on Darwin, is to use EVFILT_MACHPORT with kqueue. This could then be exposed in Async with a cfg (e.g. Async::mach_port(...)).

Unused fastrand dependency

Same deal as smol-rs/async-net#17, looks like its usage was removed in this commit.

Optionally disable timers or Asyncs on features

WASM targets don't support Async and probably won't until WASI reaches a more stable point. However, since parking is usable on WASM thanks to atomics, it may be desirable to use the Timer implementation.

My suggestion is to add two new features: io and timers. They compose as such:

• With both features enabled (they are by default), the crate functions as normal.
• With io disabled but timers enabled, the Reactor retains its timer processing capabilities but instead of calling to polling, it calls parking (which works with WASM) with the timeout resulting from process_timer_ops.
• With both features disabled, Reactor no longer exists and block_on is just an alias to the ones in futures_lite.

This would be a breaking change.

Expose a way to get the file descriptor of the inner poller

Would solve smol-rs/smol#94

Implement async-io v1 in terms of async-io v2

Many programs will still be using async-io v1 due to difficulties involved with porting over to the new API. So to make sure any future correctness/performance improvements are ported from async-io v2 throughout the ecosystem, v1 should be implemented in terms of v2.

block_on and Timer should be trivial to port since they don't have any semver breaking changes; they can just be imported from v2. v1's Async has to be implemented as a wrapper around v2.

move `impl Async<std::net::*>` to `async-net`

I actually thought about proposing this while investigating #18 and now that async-io is at v1.0.x I am not sure if a breaking change directly after v1.0 would be appreciated.

That said I believe it would be a good idea to move all the net related impls meaning impl Async<TcpListener>, impl Async<TcpStream> and so on to async-net. That way we can remove the socket2 dependency from async-io and it would reduce the overall code base. I already looked into this a bit and one obvious downside would be that the current doc tests have to be rewritten since they mostly 12/20 use net related types. Feel free to close this issue if you don't think this is a good idea or if I have simply missed something and it's not possible to implement it that way 😄

How to wait on read or write with Async<T>?

Back before this crate was split from smol, there used to be a with method which would wait for either a readable or writable event.

This method was deprecated even then, and nowadays there is only read_with and write_with.

Is there any alternative way to do the equivalent?

My use case is that I am trying to update the async-ssh2 crate, which tries to provide an async wrapper around the libssh2 C library. My fork can be found here.

This C library provides a non-blocking interface. When using this interface, when you make calls to async functions, they may return LIBSSH2_ERROR_EAGAIN indicating that you should call back again later when the underlying socket is ready. The problem is you don't know if it is waiting for a read or a write - it could be either depending on the state of the protocol.

The original code, which was written against smol 0.1, used the above mentioned 'with' method, and appeared to work. It was doing something like this:

    let res = stream
        .read(|_s| match cb() {
            Ok(v) => Ok(Ok(v)),
            Err(e)
                if io::Error::from(ssh2::Error::from_errno(e.code())).kind()
                    == io::ErrorKind::WouldBlock =>
            {
                Err(io::Error::new(io::ErrorKind::WouldBlock, e))
            }
            Err(e) => Ok(Err(e)),
        })
        .await??;

where cb is a callback that is calling a libssh2 function that may return LIBSSH2_ERROR_EAGAIN (which is converted to an io::Error of kind WouldBlock).

I am trying to update it to depend on async-io directly. I've replaced the calls to read with calls to read_with or read_write, but it ends up hanging at various points. I suspect the reason is that the C function wants to write from within read_with or vice versa.

Sorry for the convoluted explanation ... hopefully it makes sense.