Comments (5)
I'm not the original author but have worked with Litestream for a good while so take my comments with a grain of salt.
From what I understand from the code and some late fixes I've done regarding checkpointing, if the WAL gets successfully checkpointed outside Litestream's supervision it will declare that it has lost the position due to checksum mismatch and force a new generation as you've thought.
This could be tested by disabling the persistent read lock code path and forcing checkpoints and writes to a database during replication.
However I don't quite follow your reasoning why you want your application to control checkpointing? Litestream intentionally keeps a read transaction open to prevent application checkpoints from rolling in the WAL from the outside so in practice all of your own checkpoints would fail unless they race Litestream successfully which is an error condition.
from litestream.
Just to be sure I ran some tests where the read lock was intentionally removed from Litestream so it couldn't prevent external checkpoints at all. Regardless how much I abused it it would successfully always recover with:
time=2023-12-20T12:46:52.927+02:00 level=INFO msg="sync: new generation" db=//path/to/test.db generation=b9b04512b4365e9a reason="wal overwritten by another process"
time=2023-12-20T12:46:52.931+02:00 level=INFO msg="write snapshot" db=/path/to/test.db replica=file position=b9b04512b4365e9a/00000001:4152
So at worst it would do as expected and start a new generation if it lost the WAL. The remote was never corrupted and could always restore up to latest sync.
I'll update the documentation to be less scary about that, thanks!
from litestream.
Thank you for the answer! Glad to hear it fails safe. Maybe it's worth mentioning on the tips page? As it is, it looked a bit scary. ("So wait, if I don't read this page and remember to set a PRAGMA do I risk corruption?")
However I don't quite follow your reasoning why you want your application to control checkpointing? Litestream intentionally keeps a read transaction open to prevent application checkpoints from rolling in the WAL from the outside so in practice all of your own checkpoints would fail unless they race Litestream successfully which is an error condition.
Oh I wasn't clear sorry. I am saying that I don't want my application to have to be run with Litestream. Some users might use Litestream, some users might use EBS atomic snapshots, some users might choose to have no replication. If I turn off autocheckpointing, users that don't run Litestream will end up with an endlessly growing WAL, so I'd have to add a config option, which is annoying and error prone.
from litestream.
Ah, right, that makes sense.
I'd suggest keeping sane defaults and allowing overriding the PRAGMAs in config if someone wants to improve compatibility with Litestream so checkpointing on by default. That's what we've been doing: https://github.com/mautrix/go-util/blob/main/dbutil/litestream/register.go
But it indeed should be safe to have checkpointing on and it would at most force a new generation/snapshot if the app wins the unlikely race.
from litestream.
I added a new sentence to the paragraph:
When Litestream notices this it will force a new generation and take a full snapshot to ensure consistency.
That should clear up the fear of it breaking. Closing this issue.
from litestream.
Related Issues (20)
- Verbose parameter removed in version 0.3.13 HOT 3
- Extracting the executable file from the .deb release package HOT 3
- Storj S3 restore Error: "cannot find max wal index for restore: missing initial wal segment:" HOT 13
- snapshot successfully sent to S3, but not WAL segments HOT 2
- Question re: running litestream in docker warning at top of doc page HOT 1
- Website menu truncated on mobile HOT 2
- Feature request: Reverse replication HOT 2
- Crash with invalid memory address or nil pointer dereference when restoring HOT 6
- Crash on writing (deleting?) snapshot HOT 1
- Memory leak when replication targets are unhealthy HOT 4
- Is -timestamp broken when restoring? HOT 1
- Continous restore / Restartable restore HOT 2
- `plndr-cp-lock` failure with kube-vip pod when backing up K3s sqlite database
- Switch to HEAD instead of get-bucket-location in S3 init routine HOT 2
- Browser support HOT 1
- Update or remove the "blog" at https://litestream.io/blog/ HOT 1
- Restored database fails integrity checks HOT 2
- Timestamp issue while restoring HOT 1
- Azure Blob Storage Retention Error When Deleting Generation
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from litestream.