Seeming deadlock on select!

[rust-embedded]

Seeming deadlock on select!

[ryankurte]

huh, interesting failure! i have a bit of async experience but, the internals of this are unfamiliar to me... @mgottschlag no obligations but, any ideas?

absolutely grasping at straws but:

• have you tried reading from two pins on different chips to check whether it could be an interaction from sharing chips?
• i would be interested to know whether the behaviour changes if you restrict the executor to a single thread (not sure this will catch anything but, just in case it was related to #50 or similar to the classic rust-lang/rust#71072 (comment))
• have you tried to run it under gdb? it can be pretty rough where async is involved but will usually highlight the problem area (ime strace and cargo flamegraph can also be useful for getting a grasp of these things, though raspbian is missing a bunch of kernel tools so this may not be workable for you)

[datdenkikniet]

Thanks for your response :)

I've tried running it with strace, but then I get the same behaviour as with a longer-than-a-millisecond delay where deadlock seems to disappear but the ordering gets all messed up (presumably because of the overhead that strace induces). I'll be sure to try it with GDB, and the other two experiments you've mentioned.

A detail I forgot to mention: I'm running this in a Buildroot environment on kernel 4.19.97, in case anyone is trying to reproduce this issue. I've not quite figured out how to build a newer kernel this way, so I hope that this version isn't missing some important fixes to the gpio device driver that could affect this issue.

[datdenkikniet]

Additionally: I'd completely missed it while trying to look for similar issues, but it seems that I'm effectively trying to accomplish what #18 is describing. Sadly no real solution to the problem/how to do it either, so this is just what I came up with.

[datdenkikniet]

When running this with gdb, it seems like the main thread doesn't actually start/get to the futex_wait syscall that it goes to when in "working" condition. The 2nd thread does successfully go to the epoll_wait() that it does under normal execution too. Alternatively, the futex_wait is completing without actually firing an event/something that tokio catches in the select!. (I tested this both with std's select + fuse and tokio's select, but same result).

Seems like this futex_wait is actually due to the block_on that I'm using to wait for the reading of the bits to finish, so this information might not be all that relevant.

When switching to a single thread of execution (using tokio::main(max_threads = 1, core_threads = 1) or tokio::main(basic_scheduler) the program stops working all together. The program gets to aforementioned futex_wait but that's never completed.

I'll also be trying this with the gpiomon tool.

[datdenkikniet]

I've tried this with gpiomon now (i.e. gpiomon -lf gpiochip0 20 21), and with that it seems to work just fine. I get the correct output consistently, and it doesn't hang. I'll see if I can somehow figure out what's going on.

[datdenkikniet]

After some more investigation, it seems that it might be an issue in Tokio itself or the way Streams are implemented, somehow. I made my own fork and updated it to Tokio 1 (master...datdenkikniet:tokio-update), but it seems to exhibit the exact same behaviour. What happens is: the poll_next function implemented for AsyncLineEventHandle is called many times, but seemingly never completes (none of the match arms are matched, nor is an error produced by the ready! call). In the background it must complete somehow, since the memory usage of the program does not change, but somewhere something is not going right.

Removing the select! in favour of an await on a single one of the events doesn't change anything, either. The amount of events that can be read is still the same.

I'm unsure if this issue still belongs in gpio-cdev (possibly an incorrect implementation of poll_event?) or in Tokio/Futures. Any guidance would be appreciated.

[mgottschlag]

I am sorry I cannot contribute - I actually do not have much low-level tokio experience.

Anyways, so the summary right now is:

• The problem is not caused by incorrect usage of select!? I had problems with select! in the past - the macro has non-obvious failure modes, requirements to the futures which trigger those failure modes when not met, and I kept confusing the two different macris from futures and tokio, so it would be nice to rule out select! as a reason for the problems.
• The problem is not actually caused by waiting on multiple lines, as await on one line instead of select! on two does not fix the problem? This is weird, the code used to work fairly reliably for me for such use cases. I use tokio::select! with a single line without problems.

[datdenkikniet]

Whoops, pressed the wrong button.

Thanks for the feedback.

Yes, your summary seems to reflect what I've found so far. AFAIK I'm using select! correctly, and I'm definitely using Tokio's select!.

I'll re-try and remove all of the references to the 2nd line and re-try, maybe it starts working again in that case. Yeah, even when doing everything with a single line (i.e. only getting 1 line from the Chip, getting an async handle to its events, etc.) the issue still occurs.

[datdenkikniet]

What happens is: the poll_next function implemented for AsyncLineEventHandle is called many times, but seemingly never completes (none of the match arms are matched, nor is an error produced by the ready! call).

So, I wasn't entirely correct:

What happens is that the file.poll_read() in Tokio 1.2 (or poll_read_ready() in the older verison) is simply never ready. I'd missed that the ready! macro actually returns Poll::Pending if the result of the Poll that is passed as an argument is also Poll::Pending.

Very unsure why it happens.