Actor persistence and journaling

[Altair-Bueno]

Actor persistence and journaling

[thomaseizinger]

We haven't released 0.6 yet but the latest version of the code in master is extremely modular and should make it possible to implement this easily on top.

For example, you can implement your own event-loop that regularly persistents an actor's state: https://github.com/Restioson/xtra/blob/master/xtra/examples/custom_event_loop.rs

I doubt that we will ever include something like this in xtra itself. One of the design goals is to remain small and simple. It would be interesting to see this built on top and I am happy to discuss changes to any of the APIs if our building blocks don't fit your needs.

[Altair-Bueno]

Thanks for the quick answer, appreciated.

Looks like with custom event loops this could be easily archived. However, I'm a bit worried about compatibility with other libraries that would also like to instrument actors in a similar fashion.

Imagine the situation where a library X would like to gather performance metrics. It would also need their own custom event loop to archive this.

Could it be possible to augment the current implementation to support this? A series of traits that would allow anything to hook up to an existing event loop lifecycle, based on something familiar like the tower::Service trait.

[thomaseizinger]

library X would like to gather performance metrics. It would also need their own custom event loop to archive this.

Why would a library provide an event loop? I'd say providing the event loop is the job of the final application that composes all the actors.

Could it be possible to augment the current implementation to support this? A series of traits that would allow anything to hook up to an existing event loop lifecycle, based on something familiar like the tower::Service trait.

Possible yes, likely no. Mostly because:

a. We want to keep xtra small and focused
b. I don't understand the requirements well and thus whatever we'd build is likely wrong

Once 0.6 is out the door (no ETA because little time), we aim to be conservative with breaking changes so you are welcome to build your own library on top of xtra that provides instrumentation primitives.

[thomaseizinger]

Also not that Actors already have lifecycle hooks and you can do whatever you want in those. Persisting another actor's state could even be an actor wrapping another actor, similar to a supervisor.

[Altair-Bueno]

I understand, it was just a bit of brainstorming before doing anything.

I played with the idea and came up with this https://github.com/Altair-Bueno/xtra-persistence. Is still a rough prototype, but it does something.

My current idea is there are three kinds of Actors involved: The trigger (not implemented yet), the persistence provider and the targets (Actors to be persisted).

• Trigger: Has the address of other actor and sends them TakeSnapshot messages when something happens (control c, for example)
• Persistence Providers: Handle the snapshot fetch and save
• Targets: The actual user actors. They take snapshots and sends them to the persistence providers. At startup they also ask the persistence providers if any snapshots are present.

The example https://github.com/Altair-Bueno/xtra-persistence/blob/08c707cd13560828d1f090843dcfe23e18ccd298/examples/mem.rs showcases an in memory persistence provider.

I think this could work, but is still a lot of boilerplate to set up. While you don't need to create a custom event loop, you still need to pass around the PersistenceProvider. And still, I have no idea on how to do journaling of the mailboxes. I tried the supervisor idea, but it feels like too much noise. If you have some spear time to look at project, I would love to hear some feedback!

[Restioson]

Hi, I'm currently away with limited reception, but this does sound like an interesting case for investigating support through custom event loops etc. I do wonder how mailboxes could be persisted especially if they have message returns. Does it make sense to track all messages in that case? Maybe a wrapper type like PersistingMessage could help, for only messages that should be persisted and return a ().

…

On Sat, 01 Jul 2023, 22:07 Altair Bueno, ***@***.***> wrote: I understand, it was just a bit of brainstorming before doing anything. I played with the idea and came up with this https://github.com/Altair-Bueno/xtra-persistence. Is still a rough prototype, but it does something. My current idea is there are three kinds of Actors involved: The trigger (not implemented yet), the persistence provider and the targets (Actors to be persisted). - Trigger: Has the address of other actor and sends them TakeSnapshot messages when something happens (control c, for example) - Persistence Providers: Handle the snapshot fetch and save - Targets: The actual user actors. They take snapshots and sends them to the persistence providers. At startup they also ask the persistence providers if any snapshots are present. The example https://github.com/Altair-Bueno/xtra-persistence/blob/08c707cd13560828d1f090843dcfe23e18ccd298/examples/mem.rs showcases an in memory persistence provider. I think this could work, but is still a lot of boilerplate to set up. While you don't need to create a custom event loop, you still need to pass around the PersistenceProvider. And still, I have no idea on how to do journaling of the mailboxes. I tried the supervisor idea, but it feels like too much noise. If you have some spear time to look at project, I would love to hear some feedback! — Reply to this email directly, view it on GitHub <#206 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABTBBNDBVIK5BMXKYSAYDQLXOB7R5ANCNFSM6AAAAAAZY2QNK4> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[Altair-Bueno]

Does it make sense to track all messages in that case?

Not every message needs to be track down. Persistence should be always opt in.

Maybe a wrapper type like PersistingMessage could help, for only messages
that should be persisted and return a ().

I though of something similar, but again a wrapper around the message just overcomplicates things.

I think the solution would be to replace Mailbox and Address with traits. They are tightly coupled with xtra, making it imposible to use another implementation. By using traits the following could be achieved.

// Clasic Mailbox
xtra::spawn_tokio(actor, MemoryMailbox::new_unbounded());
// With a given persistence provider. Loads the mailbox if it exists
xtra::spawn_tokio(actor, PersistentMailbox::new(persistence_provider));

Also, no custom event loop would be required.

[thomaseizinger]

I think the solution would be to replace Mailbox and Address with traits. They are tightly coupled with xtra, making it imposible to use another implementation. By using traits the following could be achieved.

What is xtra then if not the Address and Mailbox types?

They are essentially an mpmc channel with typed reference-counters. If you take those away, there isn't anything left lol.

Everything around it are just helper functions that we settled on including to make life a bit more convenient. In your project, you should actually write your own spawn functions and event loop to exactly fit your needs (and reduce boilerplate).

[thomaseizinger]

Also, no custom event loop would be required.

You seem to misunderstand my/our¹ design goals. An event loop is just that; a loop. There should be nothing special about it such that you'd want to reuse it.

Instead, being able to write your own loop is the whole point. It offers so much more flexibility without having to roll your own address and mailbox. Those are the bits that are tricky to get right.

Footnotes

1. Hopefully I am speaking for you too here @Restioson :) ↩

[thomaseizinger]

I played with the idea and came up with this https://github.com/Altair-Bueno/xtra-persistence. Is still a rough prototype, but it does something.

My current idea is there are three kinds of Actors involved: The trigger (not implemented yet), the persistence provider and the targets (Actors to be persisted).

• Trigger: Has the address of other actor and sends them TakeSnapshot messages when something happens (control c, for example)
• Persistence Providers: Handle the snapshot fetch and save
• Targets: The actual user actors. They take snapshots and sends them to the persistence providers. At startup they also ask the persistence providers if any snapshots are present.

The example https://github.com/Altair-Bueno/xtra-persistence/blob/08c707cd13560828d1f090843dcfe23e18ccd298/examples/mem.rs showcases an in memory persistence provider.

I think this could work, but is still a lot of boilerplate to set up. While you don't need to create a custom event loop, you still need to pass around the PersistenceProvider. And still, I have no idea on how to do journaling of the mailboxes. I tried the supervisor idea, but it feels like too much noise. If you have some spear time to look at project, I would love to hear some feedback!

Had a quick look. Like I mentioned above, a custom event loop would make this a lot more concise. You can create a trait that needs to be implemented for each actor. That allows you to bypass the message queue and access the actors directly at the correct life-cycle point.

Sending messages won't work reliably because there is no guaranteed delivery. If an actor decides to stop, the next "Persist" message won't be delivered and you'll be missing the state updates since the last persist message.

If you write your own loop, you can directly hook into the lifecycle and persist directly after shutdown, thus capturing the entire state.