Send to multiple Logstash servers in parallel

[driskell]

Send to multiple Logstash servers in parallel

[driskell]

It's not possible at the moment. Though there may be ways I haven't thought of.

It's not gonna be simple though. What if log stash gets busy on the slower dev environment? Do we slow down? But we need to keep sending logs to production - at what point do we skip log sending to dev and stop waiting for it?

For the above reason I think it may be better to just give the dev logs hourly or something with a script for testing? Or similar. It's always "hmmmm" when crossing production/dev boundary like this as you just can't allow dev to impact production :(

[cjs226span]

I agree with your points so I wouldn't push for affecting prod.

I'm testing running 2 instances of log-courier on the systems (it's a small subset), each with their own "log file", "persist directory" and yeah, "servers". I was thinking of copying them over periodically but I set a fair amount of metadata via log-courier before sending to Logstash. Because of that, and that the data and files will change over time, I didn't want to have to maintain a hack of copying, setting metadata, etc. - I really want the logs to be sent to the dev Logstash servers in the exact manner as they're sent to prod without too much magic. Hopefully this will work out. Thanks for responding and most importantly, thank you for log-courier!

[driskell]

Hi @cjs226span

I've thought about it a bit and I can't really think of a good way to separate this out. If shipping to production slows down, shipping to dev would too. If shipping to dev slowed down, shipping to production would too. Only way around it would be to maintain two separate event queues, populated separately, which flush independently. So that means reading from a file twice with two separate threads, one for each queue. So I think at that points its probably better to just run two instances.

If you have any thoughts etc do let me know.

Jason