jeffail/checkpoint

Checkpoint

This is a Go package providing solutions that can be used in order to convert an offset-based log of discrete data payloads into a queue, where a series of log offsets can be processed and committed asynchronously without the risk of a given offset being prematurely committed.

For more information about how this package functions and the problem it solves check out this crazy video.

This package is "complete" in the sense that no further development work is planned and any PRs proposing to expand its scope will be rejected. Please report bug fixes and feel free to raise PRs to address them as these will be .

checkpoint's Issues

Use an interval tree to reduce memory allocations?

When I implemented something similar, I remember we used an interval tree instead of a linked list. It allowed us to have a smaller memory pressure at the expense of worst case log(n) performance instead of log(1) for tasks handled in random order. For simple cases like FIFO, I believe it will be faster you simply adjust an integer. Tracking a new task is always log(1).

Just creating this issue as an alternative implementation. Feel free to close this issue if not relevant!

CI/CD using Github Actions?

I noticed there are tests. Add CI/CD checks that runs the tests?

Reduce memory allocations using a sync.Pool?

I stumbled across this library by chance. I happened to have implemented something similar as a Kafka consumer. After reading the source code I noticed that Uncapped.Track(...) allocated a new node on every call, and garbage collects a node when done with a task. Considered using a sync.Pool to reduce memory pressure?

Note, I have not hit any performance issues here. Mostly submitting this as a potential improvement. I notice that the benchmarks keep allocation stats, so testing whether this positively impacts garbage collection should be fairly straightforward.