Comments (7)
The primary concern I have with dead-letter queue is how to avoid that some misconfiguration can lead to send everything into that topic.
In some cases the application can have a pretty good idea that a message is bad/corrupted and thus triggering the rerouting of the message, but if for example a new field is introduced and the consumer is not updated, it might start sending everything to DLQ.
About doing the work in client vs server. I don't think there is a big functional difference. The 2 operations cannot be atomically executed anyway, we have to first publish and then skip the bad message.
My preference would go to doing that in client lib, to avoid scalability concern in broker.
Adding on what proposed by @agarman, what about configuring the DLQ on consumer config?
consumerConf.setDeadLetterQueue("persistent://.....");
consumerConf.sendToDeadLetterQueueAfterRetries(10);
Then in consumer code:
while (true) {
Message msg = consumer.receive();
try {
// processMessage()
} catch (Exception e) {
consumer.reject(msg);
continue;
}
consumer.acknowledge(msg);
}
The difficult part would be how to support transient vs persistent errors and rejections.
from pulsar.
I am writing rejected messages to "TopicName{{RejectedMessage}}" then acknowledging the message. It'd be nice to have the broker handle this atomically.
An API like:
reject(msgId); // sends message into dead letter box for subscription; acknowledged in process
reroute(msgId, specificTopic); // same but into a specific topic
from pulsar.
PIP-22 is the proposal for implementing dead letter queue.
from pulsar.
@codelipenghui @jiazhai : you guys can probably use this issue as the main issue for dead letter topic implementation.
from pulsar.
#2400 is implementing dead letter topic.
from pulsar.
Closing as a dupe of #2400
from pulsar.
Hey guys, are there any plans to add DLQ support in the Golang CGo based library? I see that it's supported in the pure Golang one which we are not using though due to your disclaimer saying that feature parity and stability has not been reached.
from pulsar.
Related Issues (20)
- [Bug][broker] cursor will read in dead loop when do tailing-read with enableTransaction
- [Bug] Pulsar Functions ignores compressionType and crypto config for producers created with Context produce/newOutputMessage methods
- [Bug] Update partitions call is failing when topic level replication is disabled HOT 1
- As a websocket consumer I need to set InitialSubscriptionPosition to earliest HOT 1
- [Bug][broker] BrokerId npe when broker restart HOT 1
- [Bug] Unexpected Package Manager Behavior in Pulsar 3.3.0 Standalone Mode HOT 1
- [Bug] Dead lock error in Pulsar 3.0.
- [Bug] PulsarStandalone started with error if `--stream-storage-port` is not 4181
- [master branch] failed-test: OpenTelemetrySanityTest.testOpenTelemetryMetricsOtlpExport
- [Bug][broker] MessageDeduplication replay timeout would cause topic loading stuck and become unavailable
- [Bug] bin/pulsar-perf will cause the pulsar service to freeze
- [Bug][meta] Broker enter the loop to recreate zkSession failed, cause broker stuck
- [Doc] homepage 404
- [Bug][broker]PulsarRegistrationClient writableBookieInfo cache and readOnlyBookieInfo cache update fail causing broker to misjudge that the bookie is unavailable.
- [Bug][broker] Occur so many ERROR log in broker, which is confusing HOT 2
- [improve][broker] cursor read entry would trigger readMoreEntry() one more time when addWaitingCursor and notify
- [Bug] [broker] Non persistent topic will be lost when it is unloaded.
- [Bug] [broker]Sometime after reset cursor, unable to reconsume all messages in topic when ttl is less than retention time. HOT 2
- [Transaction] One single TC unavailable cause transactional throughput down to 0. HOT 1
- [Bug] bookie-2 is not able to recover after lossing the filesystem
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pulsar.