Immudb instance doesn't load data from S3 storage after reboot (randomly, discarding snapshots and other data)

[MaksymVynohradovDA]

Immudb instance doesn't load data from S3 storage after reboot (randomly, discarding snapshots and other data)

[jeroiraz]

Hi! We investigated the issue. Steps to reproduce:

• Run ImmuDB on AWS Fargate, connected to S3 as storage
• Reach memory limit due back-up restore or/and a lot of simultaneous queries to DB
• Fargate task will be moved to the DRAINED status and then restarted.
• New Fargate task will be running with error described in this issue and then "Started with an empty default database"

The rootcause that from the one side ImmuDB anyway host some recent data files (like tx or ay else) on the Docker conatiner (in our case - Fargate site) File system and from another side - instead docker or docker-compose - AWS Fargate create completly new instance of the tasks . Old data (volume) is just vanished without any way to restore them.

So, looks like it impossible to use ImmuDB running it on AWS Fargate and to be sure that data will not be lost after Fargate task crushes.

Currently, immudb asynchronously upload data to S3, so as you describe data may be lost in such cases. Replication could be used to mitigate this scenarios but master election is not implemented so it may require manual intervention or an external tooling to determine the best instance. @SimoneLazzaris you may be able to extend in this aspect.

Another possibility is to implement a synchronous operation mode when using S3 but a noticeable performance degradation would be expected. Nevertheless it seems a nice capability for use cases were degraded performance is still acceptable.

[MaksymVynohradovDA]

Update: according to the logs we found next:

immudb 2023/11/03 12:15:47 INFO: Index '/var/lib/immudb/defaultdb/index' {ts=0, discarded_snapshots=1} successfully loaded
immudb 2023/11/03 12:15:47 INFO: Discarding snapshots due to invalid checksum at '/var/lib/immudb/defaultdb/index'
...
immudb 2023/11/03 12:15:48 INFO: tx data is corrupted: ALH mismatch at tx 14356323323871488: discarding pre-committed transaction: 1

And a lot of other mentions about "discarded" data
Then after some time immudb starts empty db:

immudb 2023/11/03 12:15:47 INFO: Started with an empty default database

So why it may happen and how to fix this?

[jeroiraz]

Update: according to the logs we found next:

immudb 2023/11/03 12:15:47 INFO: Index '/var/lib/immudb/defaultdb/index' {ts=0, discarded_snapshots=1} successfully loaded
immudb 2023/11/03 12:15:47 INFO: Discarding snapshots due to invalid checksum at '/var/lib/immudb/defaultdb/index'
...
immudb 2023/11/03 12:15:48 INFO: tx data is corrupted: ALH mismatch at tx 14356323323871488: discarding pre-committed transaction: 1

And a lot of other mentions about "discarded" data Then after some time immudb starts empty db:

immudb 2023/11/03 12:15:47 INFO: Started with an empty default database

So why it may happen and how to fix this?

We'll review it asap.

Some data seems to be loaded and thus the mismatch described in the logs. Non-fully committed transactions may be discarded as the client shouldn't have received any confirmation.

[MaksymVynohradovDA]

@jeroiraz

Hi! Thanks a lot!
It happens when the immudb container crushed/reload by some reason (like automatically by AWS Fargate). Just my assumption that files on S3 updates not all-in-once but one-by-one or even batches. And in this case immudb can't process it completely before reload, therefore signature become invalid... it's just my assumption =)
But, anyway reload of service it's quite common process.

[MaksymVynohradovDA]

Hi! We investigated the issue. Steps to reproduce:

• Run ImmuDB on AWS Fargate, connected to S3 as storage
• Reach memory limit due back-up restore or/and a lot of simultaneous queries to DB
• Fargate task will be moved to the DRAINED status and then restarted.
• New Fargate task will be running with error described in this issue and then "Started with an empty default database"

The rootcause that from the one side ImmuDB anyway host some recent data files (like tx or ay else) on the Docker conatiner (in our case - Fargate site) File system and from another side - instead docker or docker-compose - AWS Fargate create completly new instance of the tasks . Old data (volume) is just vanished without any way to restore them.

So, looks like it impossible to use ImmuDB running it on AWS Fargate and to be sure that data will not be lost after Fargate task crushes.