timelydataflow/diagnostics

Diagnostic tools for timely dataflow computations

diagnostics's Issues

Diagnostics PageRank Example Stuck

Hello. I was trying to follow the example on the README.md but tdiag gets stuck.

1. One one terminal I execute: cargo run --release -- --source-peers 2 graph --out graph.html
2. One second terminal I execute: env TIMELY_WORKER_LOG_ADDR="127.0.0.1:51317" cargo run --release --example pagerank 1000 100000 -w 2
3. pagerank runs to completion.
4. tdiag acknowledges connection via:

Listening for 2 connections on 127.0.0.1:51317
Trace sources connected
Press enter to generate graph (this will crash the source computation if it hasn't terminated).

5. I press enter but tdiag hangs indefinitely.

Looking at the stack trace of tdiag there are two threads. The main thread is waiting on a thread join. Thread2 also seems stuck on await_events. Stack trace for Thread2:

futex_wait_cancelable 0x00007ffff7d9c376
__pthread_cond_wait_common 0x00007ffff7d9c376
__pthread_cond_wait 0x00007ffff7d9c376
std::sys::unix::condvar::Condvar::wait condvar.rs:73
std::sys_common::condvar::Condvar::wait condvar.rs:50
std::sync::condvar::Condvar::wait condvar.rs:200
std::thread::park mod.rs:923
<timely_communication::allocator::thread::Thread as timely_communication::allocator::Allocate>::await_events thread.rs:44
<timely_communication::allocator::generic::Generic as timely_communication::allocator::Allocate>::await_events generic.rs:99
timely::worker::Worker<A>::step_or_park worker.rs:216
timely::execute::execute::{{closure}} execute.rs:206
timely_communication::initialize::initialize_from::{{closure}} initialize.rs:269
std::sys_common::backtrace::__rust_begin_short_backtrace backtrace.rs:130
std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}} mod.rs:475
<std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once panic.rs:318
std::panicking::try::do_call panicking.rs:297
__rust_try 0x000055555661a74d
std::panicking::try panicking.rs:274
std::panic::catch_unwind panic.rs:394
std::thread::Builder::spawn_unchecked::{{closure}} mod.rs:474
core::ops::function::FnOnce::call_once{{vtable-shim}} function.rs:232
<alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once boxed.rs:1034
<alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once boxed.rs:1034
std::sys::unix::thread::Thread::new::thread_start thread.rs:87
start_thread 0x00007ffff7d95609
clone 0x00007ffff7ed1103

Please lmk if I missed something when executing the commands?

[graph] visualization to indicate where a dataflow is stuck

@frankmcsherry:

It is hard to diagnose a "stuck" timely dataflow computation, where for some reason there is a capability (or perhaps message) in the system that prevents forward progress. In the system there is fairly clear information (in the progress tracking) about which pointstamps have non-zero accumulation, and although perhaps not strictly speaking a "visualization" we could imagine extracting and presenting this information.

@antiguru recently had a similar issue, in which he wanted to "complete" a dataflow without simply exiting the worker (to take some measurements), and when he attempts this the dataflow never reports completion. The root cause was ultimately that a forgotten input was left un-closed.

One idiom that seemed helpful here was to imagine a version of the dataflow graph that reports e.g. whether operators have been tombstoned or not (closed completely, memory reclaimed). This would reveal who was keeping a dataflow open, which is a rougher version of what is holding a dataflow back. We might also look for similar idioms that allow people to ask, for a given timestamp/frontier, which operators have moved past that frontier and which have not, revealing where in the dataflow graph a time is "stuck".

This tool seems to be broken

Hi,

I have tried to use this tool with different examples and crate versions.

It always gets stuck: pressing enter has no result.

To reproduce:

FROM ubuntu:latest
RUN apt-get update
RUN apt-get install -y cargo git

RUN git clone https://github.com/TimelyDataflow/timely-dataflow
WORKDIR timely-dataflow

RUN cargo install tdiag

ENV PATH="$PATH:/root/.cargo/bin"

Run:

docker build -t foo .

And run this docker in two different shells:

docker run --name foo_container --rm -it foo tdiag --source-peers 2 graph --out graph.html

docker exec -it foo_container env TIMELY_WORKER_LOG_ADDR="127.0.0.1:51317" cargo run --example pagerank 1000 1000000 -w 2

[differential arrangements] Support batch drop events

Differential just got TimelyDataflow/differential-dataflow#201, which makes it possible to track batches created in transient trances. Seems like differential arrangements could do something useful with that.

[graph] Support filtering/collapsing of data flow regions to make huge graphs readable

This may be needed for complex graphs like Frank's epic doop graph.

Various snippets from #1:

---

@frankmcsherry

I've been thinking a bit about how to present these, and one thought was: maybe it makes sense to have two nodes for the feedback node, and to not connect them other than visually. This maybe allows the graph to dangle a bit better, and reveals the acyclic definitions.

@comnik

I think this could be a good use case for a touch of interactivity, e.g. draw the nodes somewhat differently to indicate an outgoing / incoming feedback edge, and then highlight the pair when the user hovers on either of the two nodes.

As an experiment, I built an extra script for adding DataScript into the mix. This is intended to be completely opt-in, without changing anything about the current representation.

I also added a hook to re-render the whole thing reactively.

This should give us a low-overhead (no React!) way to experiment with a few more dynamic features, such as highlighting feedback edges.

It would be helpful to have scopes be exported as well, which would allow us to do things such as collapsing / expanding scopes.

Update to timely / differential 0.10

The crates.io release currently points at timely & differential 0.9, so it can't be used together with a 0.10 codebase.

[graph] Subgraph names & operator names aren't shown

Named regions, named scopes and named operator invocations don't seem to be shown on the graph, which greatly limits how useful the data it gives is