Multiple Broker/Indexers ingesting

[cdenneen]

Multiple Broker/Indexers ingesting

[cdenneen]

@codekitchen @robbavey can anyone tackle adding the dynamodb alternative to sincedb to this input?

[jpuskar]

The kinesis plugin uses a dynamodb row that just assigns different data streams to different logstash instances.

https://github.com/logstash-plugins/logstash-input-kinesis

This isn't the most efficient method, but for deployments with logstash-as-cattle, this type of implementation at least moves the plugin from: "we can't use it", to, "okay this will work".

[alangalvino]

Any update on this?

[joshuaspence]

+1

[webmstr]

Happy one-year anniversary.

[jordansissel]

Logstash currently has no mechanism for internode coordination (two logstash nodes coordinating work efforts), and my best guess is that we would need a coordination mechanism in order to achieve what is proposed in this issue. The one external system that this input knows about is S3, and as far as I can tell, S3 can't be used for coordination because it lacks atomic operations that could make coordination possible.

At this time, I don't' have a solution, so this issue will wait until someone can come up with a solution that other S3 input users find agreeable.

[cdenneen]

If the sincedb path was added to a shared source would logstash honor that it would there be collisions?
I think if we can't do proper locking the only other option would be to use some other external source as sincedb to enable coordination mechanism. So whether for file or s3 inputs the external resource would handle managing which broker grabs which file.
Extending the sincedb logic would only suggestion I could think of to enable making inputs like this redundant across multiple brokers.
Seems like a large under taking if someone wants to suggest some plugable options (mongo, dynamo, db, etc)

[nunofernandes]

Yeah.. that would be awesome to change the sincedb to an "outside" system so that we can treat s3-input as cattle.

[evanvin]

we've thought about utilizing EFS to achieve this, but for the time being, we settled on just boosting our logstash workers/mem/cpu/batch.size. But we need more.

UPDATE:
we will be trying out the logstash-input-s3-sns-sqs plugin for (if we can get it to work with out system) this should allow us to horizontally without worrying about the same files being processed per new container