CubDB crashing when under heavy load

[sezaru]

CubDB crashing when under heavy load

[lucaong]

Hi @sezaru ,
Thanks for reporting this. Compaction seems indeed to be related to your issue. One question: what version of CubDB are you using? I added a couple of improvements to compaction recently, in particular to avoid crashing the main process if compaction crashes.

It does make sense to stop automatic compaction when backfilling data, and trigger it manually when done (or re-enabling auto compaction), but that said if compaction crashes the main process it should definitely be considered a bug.

[sezaru]

I'm using the latest one, 1.0.0-rc.3.

[lucaong]

Ok thanks, I’ll have a look into it. At a glance it looks like the problem is due to two compaction operations running concurrently, but I need to investigate a bit.

[lucaong]

Hi @sezaru ,
Thanks again for reporting this. I found the issue (a subtle race condition during compaction), and #22 should have fixed it, I just merged it and created a new release (1.0.0-rc.4). It would be awesome if you could upgrade and let me know if you still experience the issue after this fix.

After the fix, compaction should not crash in the ways you observed, even during heavy write load. That said, if you can predict when you will have heavy write load (for example during a manual data fill), it's more efficient to temporarily disable auto-compaction, and run a manual compaction after the writes instead.

[lucaong]

I am closing this issue, as I am very confident that #22 fixed it, but feel free to comment if you find that your issue still occurs with the new release, and I will reopen this if necessary.