maif/thoth

Event sourcing in java with vavr, akka stream and vertx reactive PG driver

Event sourcing

This repository provides tools to implement event sourcing in your application.

It guaranties that:

• Events will be written in the database before being published in Kafka
• Publication in Kafka will be reattempted until it succeeds

It provides capabilities of defining two types of projections:

• "Transactional" projections, that are updated in the same transaction as the events
• "Eventually consistent" projections, updated asynchronously by consuming Kafka

It also allows storing snapshots of the application state, for scenarios that implies lot of events.

These libs are based on :

• Vavr for functional stuff (immutable List, Either, Future)
• Akka stream for reactive streams
• jackson for json
• jooq to build query
• vertx for reactive postgresql database access
• Postgresql and kafka are the in production tested data stores

Modules

• commons-event: POJOs that represent the stored events. Can be used by consumers to parse events.
• thoth-core: APIs for event-sourcing
• thoth-jooq: A jooq simple implementation of the thoth-core APIs
• thoth-jooq-async: A jooq implementation of the thoth-core APIs using the jooq-async-apiinterface

Documentation

See our documentation.

Limits

• A single command can't currently modify multiple entities see this issue

Development

Compile / Test

./gradlew compileJava

docker-compose -f docker-compose.test.yml up 
./gradlew test

Generate the documentation

cd thoth-documentation
sbt generateDoc

thoth's Issues

Enhancement : Spring integration ?

Hello, have you planned to add a spring integration (like Izanami), i'm not thinking about some magic annotations, but just an auto-configure that create some beans and make the integration straight forward.

I've created a demo project for my company and the integration is quite easy, but a little auto-configuration would make it easier and add a first "good impression" 😄

Prevent command concurrency

In some scenarios, commands should not be run concurrently.

There is several possible needs :

• Command should not be run concurrently with other commands of the same type on the same id
• Command should not be run concurrently with any other command on the same id

There is also several way to handle concurrent command :

• Keep second command waiting until events generated by the first command are consumed
• Reject second command

Add a size for querying to limit the results

Few queries

Hi @larousso and @ptitFicus

This project looks valuable. I have few queries relating to it.

a. Why have you decided not to use CDC-Debezium and how are you able to achieve it without using CDC.. pretty interesting? Documentation says: "It keeps trying writing to Kafka until succeeded", won't it impact performance?

b. What is the role of akka in this project? Can we do away with it because it is BSL now.

c. While writing to Kafka, Avro/protobuf formats have not been used which are really fast. Any reason for it? Is it possible to integrate Thoth with this gRPC/REST proxy for Kafka?

d. Does it support these: Optimistic Concurrency, (multiple users writing events to the same table at the same time) Event Replay (right from beginning for Audits etc.), Snapshots, topic compaction (delete/overwrite a message record for a given key), and Event Versioning?

e. What is the difference between Non blocking JOOQ and Standard JOOQ/Kafka implementations. In which case (use cases), shall we use which implementation?

f. Is there no need for Kafka-proxy

Thanks

Vertx 4 upgrade causes transaction already close error

Step builder for wiring

At the moment, it's very complicated to wire all the components, it could be really good to have a step builder that help for this.

Doc: example to intialize projection

Current projection mecanism allow to consume events that will happen in the future, however, there is nothing in the doc explaining how to catch up with past events.

Therefore we should add a documentation section explaining how to catch up by streaming journal content.

use java 16 in sample project

replace java 15 enabled preview by java 16 in sample project

Use the same objectMapper everwhere

A different ObjectMapper is used to publish events in Kafka at first try and to write events in database.
This can cause bugs when republishing events in kafka if first publication failed, since the two are not compatible, for instance on vavr lists.

I think the best way to do it would be to instantiate and use some kind of singleton ObjectMapper, that is exposed for modification.

Kafka config for durability

Provide a default config for durability in kafka producer

Give aggregate as an input to projections

Some projections may benefit from the aggregate.

It could avoid useless journal reads in some cases

Resilient kafka consumer

Create a kafka consumer implementing resilience :

• restart with backoff on crash
• handle partition parallelism and per partition commit

Create a TCK

A Technology Compatibilty Kit would be helpfull to ensure every implementations of scribre-core are corrects.

This would be a great help to #3.

Support multiple entity modification with a single command

In its current state, scribe only allow modification of a single entity with one command.

However there is some event sourcing use case when modification of multiple entity at once is necessary (e.g. bank accounts).

Therefore it would be a good feature to have.

Some rough ideas:

• replace command entityId by a list

• allow command handler to return a list of event with differents entityIds

Handle errors in eventually consistent projections

Right now when there is an error building eventually consistent projection, offset won't be comitted and consumption will be stuck.

We should add another option for error handling.

Concurrency issue for replay on multi node architecture

The replay event mecanism should use the postgresql "select for update" to replay event in order to avoid multi publication to kafka.

Create a full JDBC implementation

Documentation for multiple command in the same transaction

Schema not supported for sequences

TableNames should have a schema field