ysbaddaden/earl

Service Objects for Crystal (Agents, Artists, Supervisors, Pools, ...)

earl's Issues

Improve Scheduler design

The Scheduler agent was quickly put together and only supports CRON scheduling. We should review it's API design, and maybe introduce alternatives ways to schedule something, leveraging Time::Span or skipping Earl::Scheduler for a simpler interface.

For example:

Earl.schedule(agent, cron: "5 4 * * 6") # every Saturday at 04:05 am
Earl.schedule(agent, every: 5.minutes)

EDIT

It might also be nice to schedule one time jobs, they wouldn't survive a restart, but still interesting nonetheless:

Earl.schedule(agent, at: Time.parse("2023-05-31T15:12:00Z"))
Earl.schedule(agent, in: 5.minutes)

• Earl::Schedulable
• Earl::Every
• Earl.schedule(agent, cron:)
• Earl.schedule(agent, every:)
• Inherit from DynamicSupervisor (blocked by #7)
• Tests
• README
• SPEC

Related #10

Rework Earl::Logger to use Log (stdlib)

The current Earl::Logger never made the jump to Log from stdlib. Worst, it implements its custom Logger. Upgrading to Log is long overdue.

Ideally, Earl::Logger would supervise any fiber that Log spawns (using Earl.application), because Earl doesn't accept fibers to be started without supervision.

Erlang gen_server like behavior

The Erlang gen_server behavior allows to abstract the communication with an actor, by providing a public API that sends messages to the actor (from the outside actors) and have distinct internal handlers to execute in the actor's context, hence free from any concurrency/parallelism issues.

The behavior could avoid lots of boilerplate in Earl, and completely avoid the need to define the M type of Mailbox(M), could help to transparently support synchronous and asynchronous communication, and even have defaults to handle agent hooks such as terminate and trap as standardized info messages.

I doubted that it was possible, but playing with the concept showed that some subset of gen_server can be applied to Crystal, with the help of macros (of course): https://gist.github.com/ysbaddaden/fb41d89663e40dbd1e4d19cc2438931e

I doubt we can fully implement the behavior:

1. there is no pattern matching in Crystal for example, so the proof-of-concept above expects that the first argument is a Symbol while the rest are arguments;
2. I don't see how to support {:reply, *args), maybe a very simplified version, like passing whatever the handler returns? But we'd need a Future(T) and thus need to know or infer T... while we can't call typeof in macros... but maybe we can access a method's return type in macros? 🤔

Yet, it's already nice to avoid to repeated boilerplate (see the example Bus in the above Gist).

Rename Registry as Broadcast, PubSub or ...

The current Registry actor is poorly named.

It's purpose is to broadcast the messages it receives to all subscribed listeners. There is no selection of what listeners will receive, so it's closer to a broadcast (notify everyone) than PubSub (notify me of messages with these topics).

It should thus either be named Broadcast or PubSub if we intent to have topic subscription, where broadcasting would be the wildcard (*). I'm all ears for a better naming, especially if the concept already exists in Erlang OTP.

Registry would be re-purposed to become a named registry of actors. See #9.

Earl::Aplication on-exit behavior

On exit, Earl::Application calls terminate on all agents in the supervision tree, but does not wait for them to transition from stopping to stopped or crashed state. This leaves no time for agents to gracefully stop, should Earl::Application wait for itself to transition to stopped? Perhaps with a short timeout to terminate anyway.

Bidirectional communication (replies)

Right now Mailbox(M) merely delegates to Channel(M) which only permits unidirectional communication from an actor A to an actor B, but doesn't permit B to reply to A after it processed its message, because there is no way to know about A. In order to achieve this the developer must manually create and use a type to wrap the message (e.g. {Actor, M}) then manually which is bothersome.

Earl must introduce a solution.

Should Agent#reset raise by default

If reset on an Agent isn't explicitly overridden, it might be better to assume that the agent cannot be reset, instead of doing nothing.

CPU 100% with the pool

Hello,

First of all, thank you for this lib, it's awesome. I've been testing it for five hours, top.
I encountered a bug or a mistake on my part maybe. If the capacity of a pool is above 5, the CPU rises to 100% until I kill the process.

 class Worker
    include Earl::Agent
    include Earl::Mailbox(String)

    def call
      while message = receive?
        p message
      end
    end

    def terminate
      puts "terminate"
    end
  end

  pool = Earl::Pool(Worker, String).new(capacity: 10)

  spawn do
    8.times do |i|
      pool.send("message #{i}")
    end
    pool.stop
    puts "stop"
  end

  pool.start

Is it related to the number of CPUs in my machine?

Agent state should be an Atomic

An agent's state may be changed by whatever fiber is calling an action on the agent (e.g. call #stop). The state machine only allows state to transition from one state to another, but doesn't do it atomically.

This isn't a problem with a single thread (only one fiber is running at any given time) but with MT multiple fibers may affect the state, and there is a race condition between checking the current state and setting the new state.

Using an atomic with cmpxchg might be able to resolve this issue. This may also help with #7.

However, this may bring some other questions to light, related to the finite state machine, like silencing some state change errors (e.g. stopping a stopped agent), or resolving some concurrent situations.

Document Scheduler

Earl::Scheduler should be properly documented. A brief example in the README.md as well as an extended documentation in SPEC.md.

Related to #11

Unhandled exception in spawn: can't transition agent state from Starting to Stopping

Hello,

I have tried pool.spawn and the process crashed with this error:

[00] Unhandled exception in spawn: can't transition agent state from Starting to Stopping (Earl::TransitionError)
[00]   from Earl::Agent::State#transition<Earl::Pool(App::Worker, String), Earl::Agent::Status>:Nil
[00]   from Earl::Pool(App::Worker, String)
[00]   from Earl::Pool(App::Worker, String)
[00]   from ~procProc(Nil)
[00]   from Fiber#run:(IO::FileDescriptor | Nil)
[00]   from ~proc2Proc(Fiber, (IO::FileDescriptor | Nil))
[00]   from ???
[00] "message 0"
[00] "message 1"
[00] "message 2"
[00] "message 3"
[00] "message 4"
[00] "message 5"
[00] "message 6"
[00] "message 7"
[00] "message 8"
[00] "message 9"
[00] Unhandled exception in spawn: can't transition agent state from Running to Recycling (Earl::TransitionError)
[00]   from Earl::Agent::State#transition<App::Worker, Earl::Agent::Status>:Nil
[00]   from App::Worker
[00]   from Earl::Pool(App::Worker, String)
[00]   from App::Worker
[00]   from ~procProc(Nil)
[00]   from Fiber#run:(IO::FileDescriptor | Nil)
[00]   from ~proc2Proc(Fiber, (IO::FileDescriptor | Nil))
[00]   from ???
[00] Unhandled exception in spawn: can't transition agent state from Running to Recycling (Earl::TransitionError)
[00]   from Earl::Agent::State#transition<App::Worker, Earl::Agent::Status>:Nil
[00]   from App::Worker
[00]   from Earl::Pool(App::Worker, String)
[00]   from App::Worker
[00]   from ~procProc(Nil)
[00]   from Fiber#run:(IO::FileDescriptor | Nil)
[00]   from ~proc2Proc(Fiber, (IO::FileDescriptor | Nil))
[00]   from ???
[00] Unhandled exception in spawn: can't transition agent state from Running to Recycling (Earl::TransitionError)
[00]   from Earl::Agent::State#transition<App::Worker, Earl::Agent::Status>:Nil
[00]   from App::Worker
[00]   from Earl::Pool(App::Worker, String)
[00]   from App::Worker
[00]   from ~procProc(Nil)
[00]   from Fiber#run:(IO::FileDescriptor | Nil)
[00]   from ~proc2Proc(Fiber, (IO::FileDescriptor | Nil))
[00]   from ???
[00] Unhandled exception in spawn: can't transition agent state from Running to Recycling (Earl::TransitionError)
[00]   from Earl::Agent::State#transition<App::Worker, Earl::Agent::Status>:Nil
[00]   from App::Worker
[00]   from Earl::Pool(App::Worker, String)
[00]   from App::Worker
[00]   from ~procProc(Nil)
[00]   from Fiber#run:(IO::FileDescriptor | Nil)
[00]   from ~proc2Proc(Fiber, (IO::FileDescriptor | Nil))
[00]   from ???
[00] Unhandled exception in spawn: can't transition agent state from Running to Recycling (Earl::TransitionError)
[00]   from Earl::Agent::State#transition<App::Worker, Earl::Agent::Status>:Nil
[00]   from App::Worker
[00]   from Earl::Pool(App::Worker, String)
[00]   from App::Worker
[00]   from ~procProc(Nil)
[00]   from Fiber#run:(IO::FileDescriptor | Nil)
[00]   from ~proc2Proc(Fiber, (IO::FileDescriptor | Nil))
[00]   from ???

The code:

class Worker
    include Earl::Agent
    include Earl::Mailbox(String)

    def call
      while message = receive?
        p message
      end
    end

    def terminate
      puts "terminate"
    end
  end

  pool = Earl::Pool(Worker, String).new(capacity: 5)

  spawn do
    10.times do |i|
      pool.send("message #{i}")
    end
    pool.stop
    puts "stop"
  end

  pool.spawn
  sleep 4

An idea please?

`Earl::Artist(AbstractType)` raises `#call(AbstractType) must be defined`

This example fails to compile with Error: method Bar#call(AbstractMessage) must be defined:

require "earl/artist"

abstract struct AbstractMessage
end

record FooMessage < AbstractMessage

module Foo
  macro included
    include Earl::Artist(AbstractMessage)
  end

  def call(event : FooMessage)
    p "event: #{event}"
  end
end

class Bar
  include Foo
end

bar = Bar.new
bar.spawn
bar.send FooMessage.new

sleep

But there is call(event : FooMessage) implementation in module Foo so it should not raise the error.

Repurpose Registry to be a named registry of agents

Registry should be re-purposed to become a named registry of agents.

Agents would be able to register themselves with an unique name across the application. Supervisors might be able do it for them (especially useful for Dynamic Supervisors, see #7).

Agents would then be capable to communicate with an Agent by name, without holding an actual reference. Ideally, agents could even be (re)started on demand.

This would overcome the current limit that we must hold a reference to an agent, and that all agents must always be running all the time.

Related: #8 and #7

Dynamic Supervisor

Earl misses a dynamic supervisor that would start actors on demand, and maybe stop them at some point.

No SPEC or design yet. I need to understand them better in Erlang/Elixir more before I start drafting something.