flyeralarm/kafka-merge-purge

Pipe data between Kafka topics with ease and choose what to do for individual records

kafka-merge-purge's Issues

Retrying skipped events of a dead letter queue in a safe manner

For me it looks like it's not possible to retry skipped events of a dlq without also be "re asked" about the already merged and purged events (only if log compaction has not cleaned it up in the meantime).

When I first run the CLI tool, let's say for a dead letter queue "event-dlq", a retry topic "event-retry" and a given group id "dlq-handling-group".

I could decide for each event if i want to skip, merge or purge.

AFAIK skipping do nothing (but in commit mode it's commit the offset), merging put it to a given retry queue and purging creates a tombstone event...

Let's imagine i skipped message #1, merged #2 and purged #3.

Now some new messages appear: #4 and #5

Let's say i want now to process all not handled events of the "event-dlq" topic.

If I use same group id asking would start only with new messages, so it would start for message #4.

If I use a different group id, the tool will ask me for all messages which includes the merged and depending on log compaction maybe also the purged ones.

So, I don't see a way to reprocess a dlq topic in a way where i would:

• re asked for skipped events
• leaving out the purged events in a safe way (i think depending on log compaction is quite error prone)
• being asked about new events