elfo-rs/elfo

Your next actor system

elfo's Issues

Use dynstack for dump storage

We can use the dynstack crate for storing dumps instead of SmallBox<...> to avoid extra allocations.

Print error's backtraces

• Custom eyre's handler that captures backtrace at the moment of error.
• Remove all frames starting at the exec function to provide short traces.
• Print error.stack = [...] in logger.

Asynchronous responses

To do describe the problem

Support metrics outside the actor system

The simplest example is the info metric with the version of the service.

Detect hanging actors

An actor containing an infinite loop cannot be detected by elfo_busy_time_seconds for now, because the metric is updated after polling the actor's future. Thus, we need to have a way to calculate the metric also outside of the polling.

One possible implementation is to have some "global" ThreadLocal<(Addr, Timestamp)> (thread_local). Before polling the future we can store here the current actor's address and ts. Then, the special actor iterates over it periodically and updates the metric.

However, it requires a way to emit metrics with a custom ActorMeta. Also requires #8

Get rid of SmallBox in the mailbox in favor of "dynqueue"

to do describe later

Metrics for restarts and configures

Run routers in the scope of the corresponding group

Now they're called in the sender's scope.

Can degrade performance, need to run benchmarks.

Revise `UpdateConfig` and `ConfigUpdated` mechanism

• ValidateConfig should be discarded by default.
• UpdateConfig should bypass a mailbox UpdateConfig shouldn't be sent as request.

It should highly decrease the time of config updating in case of many actors.

Support `valuable` in the logger

`elfo::messages::Impossible`

#[message]
enum Impossible {}

`ctx.try_recv` should poll sources

Sources are polled only in ctx.recv().await call for now.

Allow functions in `assert_msg!` macros.

It would be nice to have an ability to define functions as arguments to assert_msg! macros.
Right now it supports only structures.

Check config hashing

Actualize `CachePadded`

crossbeam's CachePadded contains more variants than the elfo-utils' one. Need to update the code or just use crossbeam-utils directly.

Metrics for responses

ctx.request(..).await doesn't produce any metric for responses. Actually, we produce no special counters for all incoming messages, but it's ok for incoming via the mailbox metrics because there is elfo_handling_time_seconds_count.

The sump for hanging actors

There is no protection against hanging tasks (e.g. due to an infinite loop) in tokio.

One possible way is to have one dedicated CPU for threads with hanging actors. Once a hanging actor is detected, all other tasks should be moved to other threads. Then, we can set the affinity of the thread to the sump CPU. All hanging actors will share the same CPU and rely on OS's scheduler.

Unlikely it's possible to do using tokio's scheduler because of a leak of the appropriate API.

Logger: make timestamp changes bold

It can be useful to make bold changes in timestamps between lines.

Improve actors' budgets

Now we're limiting the number of recv() calls to avoid starving other actors on the same scheduler's thread.

However, there are several ways to improve it:

• Reset the counter if the mailbox is empty. Requires moving to try_recv() + separate recv().await calls.
• (Need to discuss it). If the current timestamp is calculated already (e.g. for metrics), check the elapsed time instead of just counter.

Utilization of logger cannot be calculated right way

The logger is based on tokio::select for now. The related issue: #25.

Need to use Stream to avoid using tokio::select.

Merge config sections

Works fine:

[common]
system.logging.targets.hyper.max_level = "Info"
[a]
system.logging.targets.reqwest.max_level = "Info"

Broken:

[common.system.logging.targets]
hyper.max_level = "Info"

[a]
system.logging.targets.reqwest.max_level = "Info"

Compile without warning if the "unstable" feature is disabled

Sources should have separate controls and be more dynamic

To do describe the problem

System messages must be `non_exhaustive`

Work better with `tokio::select`

Actors can use tokio::select, but the elfo_message_handling_time_seconds metric isn't calculated correctly, which breaks the calculation of utilization. Maybe we need another way to calculate utilization?

An actor's `exec` should have a way to alter `RestartPolicy`

Support exemplars in telemetry

OpenMetrics defines exemplars and prometheus has supported them already in the experimental mode.

Dump the actor lifecycle

Status changes should be dumped.

Improve API of multiple runtimes

Dumping: per class settings

We can configure the dumping only for all classes at the same time. Add a way to override settings for specific classes.

`Proxy::collect().await` to collect all available messages to `Vec`

Support logging in JSON format

Revise `elfo_busy_time_seconds`

The metric is expensive for two reasons:

• It's emitted on every polling, which should be too expensive in the case of tokio::task::yield_now() or for a lot of fast handled messages.
• It's a histogram metric, the most expensive of all types.

Instead, we can calculate separately elfo_busy_time_seconds_max, elfo_busy_time_seconds_sum and elfo_busy_time_seconds_count. It's enough for most use cases.

Configure the behavior for signal (force vs normal)

Remove metrics of finished actors

Remove metrics of finished actors if they have been rendered at least once.

A custom allocator considering actor's OOM score

`time::Interval` should have the method to produce biased ticks

Sometimes it's useful to produce ticks within some area around the configured period. For instance, when a lot of actors inside a group have the same period to do something heavy (e.g. flushing to DB). It helps to avoid spikes.

`request().all()` should provide `ActorMeta`

For now, it returns only responses.

Reuse TraceId Chunks

Metrics outside the actor system

Testing the dump storage

After rewriting the storage most tests are gone, need to rewrite them.

Support sending requests in `ctx.send`

It should be possible to call ctx.send::<SomeRequest>() to support the "fire and forget" case and have better evolution in distributed systems.

A custom panic handler

We don't need to print panics inside actors, because they are handled by the supervisor. Or we need to print them using the logger.

Get rid of the requests `all` hack

https://github.com/elfo-rs/elfo/blob/master/elfo-core/src/request_table.rs#L85

Revise the mailbox implementation

Some variants:

The behavior for missed ticks in `time::Interval` should be configurable

https://docs.rs/tokio/latest/tokio/time/enum.MissedTickBehavior.html

Label ordering matters

register_gauge!("test", "a" => "a", "b" => "b");
register_gauge!("test", "b" => "b", "a" => "a");

rendered:

test{actor_group="reporters",a="a",b="b"} 0
test{actor_group="reporters",b="b",a="a"} 0

Dumping: introduce verbosity levels

What about introducing levels (Verbose vs Normal) in dumping to have a more flexible way to filter messages out.

Metrics about the dump storage usage

• elfo_emitted_dumps_total, elfo_limited_dumps_total, elfo_lost_dumps_total, elfo_written_dumps_total
• elfo_dumps_usage_bytes, elfo_dumps_usage_items
• elfo_dumps_capacity (in elements? in parts?)

Move to resolver v2 & edition 2021