Out-of-order Events

[ameech]

Out-of-order Events

[tirumaraiselvan]

That's a good question. One thing you can do is model your application such that an event which updates a resource is only created after the resource is created.

You have the following 2 actions:

1. Create resource A
2. Update resource A

Say you create events for 1) and 2) simultaneously and send them for delivery, then there is a problem as you noticed because 2) can arrive before 1).

But you can always model your application so that an event for 2) is created only after 1) is created.

From an implementation point of view, you can create event 2) by having a "AFTER INSERT" trigger on resource A. Hope that makes sense.

[tirumaraiselvan]

Another thing you can do is leverage the idempotency and retry-ability of your functions. This way even if something arrives in a unexpected order, it will fail gracefully and be retried till success.

[ameech]

If you're relying on your application to make sure 1 always happens before 2, would you be better served by a synchronous process?

How do you handle conflict resolution for multiple updates arriving out of order, but are updating the same record? You could rely on timestamps, but you can't guarantee that clients have the same timestamps.

Obviously there are ways around all of this, but does it make it easier for the developer using this pattern? I'm not sure.

One option to fix this would be not allowing out of order events as much as possible. Using something like a FIFO queue would solve a fair bit of these problems.

[coco98]

@ameech Hm...makes sense.

I think about this problem like this:

1. Try to make sure that events are as independent as possible so that you can call serverless functions as events come without worrying about order
2. If I can't ensure 1, I usually write the serverless function in a way that refetches latest state, checks if the state is as expected and then continues (perhaps in a DB transaction, if the state is fetched from a database that allows this). If state is unexpected, abort and return error with a retry-after backoff
3. If 2 is also not possible, then put dependent events in a FIFO queue

Does that make sense? I usually manage to get away with 2 and this works because I think of each serverless function being "idempotent" and infinitely retry-able. It also keeps things simple from the point of view of writing the serverless function. I try not to worry about any external guarantees about how I expect things to be.

If this whole things seems synchronous, there could be 2 possibilities:

1. Is this synchronous only for mutating database state? If yes, I can run all my business logic in a database transaction and I would move my code to the app-facing GraphQL API (or maybe even inside the database ;)).
2. Does the logic seem synchronous, but is actually synchronous across external APIs or independent datastores where I can't do things transactionally? In this case, I would try to do things as mentioned in the 3 points above.

I'm not sure if I was able to convey my thoughts clearly. :)
LMK what you think!

[ameech]

I think we're getting closer to how this should work, however I think 1 and 2 fall apart when you have multiples updates for the same record. How do you reconcile which one comes first? Seems to only be solvable by using something like a FIFO queue or some sort of log.

Also, a lot of these are based on the assumption of events being idempotent, but I find in real life that it's not possible to have that across the board. For example sending a message, you can't keep retrying that. You can get around it by using idempotency keys, but a lot of these issues just go away when you remove the "Out-of-order" best-practice.

[tirumaraiselvan]

@ameech Yes, few cases can only be solved using a FIFO queue or some well ordered mechanism but do you think this is the common case?

Also, it is serverless best practice to not design application which are dependent on order. This is because when you hit a serverless endpoint, the delivery is not strictly once or ordered. 3factor requires you to be cognizant of this and hence take some extra effort, only when required, to make your serverless handlers as independent as possible.

For e.g. in the first case you mentioned (update before create), you should emit an update event only after create is successful. And in the second case (multiple updates), you should perhaps have something like a timestamp column which will void an update if it is later than the event timestamp.