Metadata and Journal not being written atomically

[okumin]

Metadata and Journal not being written atomically

[okumin]

@ColinScott

Transactions are committed in this line, so it is expected that metadata and journal are written atomically.

Could you check some logs to confirm that transactions are written correctly?
I'm poor at postgresql but commit logs like MySQL's binlog maybe exist.

[ColinScott]

I'll see if I can build a minimal replication case and get some logs, although I'm not a PostgreSQL expert.

[ColinScott]

I've created a project that replicates the issue here: https://github.com/ColinScott/persist-test

I'm hosting the PostgreSQL instance in a docker image based on the official PostgreSQL image. You can run this with docker run --name some-postgres -e POSTGRES_PASSWORD=qazxsw21 -p 5432:5432 -d abstractcode/akka-postgres:9.4.6.

My first test run for this resulted in 7460 entries in the persistence_metadata table but none in the persistence_journal table. I didn't find any useful logs in the docker container running the database after this test (or indeed any PostgreSQL logs at all).

[okumin]

@ColinScott

Thanks for your minimal existence and it makes me realize the cause.

About inconsistency

PersistentActor replays existent a snapshot and journals first after creating.
PersistentActor of akka-persistent-sql-async inserts into metadata table on replaying those if its metadata has not yet created.

https://github.com/okumin/akka-persistence-sql-async/blob/master/core/src/main/scala/akka/persistence/snapshot/sqlasync/SQLAsyncSnapshotStore.scala#L15

Something like inconsistency happened due to that, but it is not inconsistency.
So you can run com.abstractcode.persisttest.TestMain with few Items(e.g. 1 to 100) on records created on high load.

Tests by TCK uses internal APIs directly, so I have observed transaction boundaries on such an abnormal environments.

About errors

Being related to underlying libraries, followings are guesses……

postgresql-async seems to collapse and becomes non-available once its buffer is overflowed.
That buffer is parameterized by configuring wait-queue-capacity parameter.

https://github.com/okumin/akka-persistence-sql-async/blob/master/core/src/test/resources/postgresql-application.conf#L11

However, buffers will be overflowed sometime if too many queries which postgresql cannot handle.
So you should take care of high loads.

akka-stream looks good for this problem……

[ColinScott]

Thanks for looking at this.