Comments (3)
It's not possible at the moment. Though there may be ways I haven't thought of.
It's not gonna be simple though. What if log stash gets busy on the slower dev environment? Do we slow down? But we need to keep sending logs to production - at what point do we skip log sending to dev and stop waiting for it?
For the above reason I think it may be better to just give the dev logs hourly or something with a script for testing? Or similar. It's always "hmmmm" when crossing production/dev boundary like this as you just can't allow dev to impact production :(
from log-courier.
I agree with your points so I wouldn't push for affecting prod.
I'm testing running 2 instances of log-courier on the systems (it's a small subset), each with their own "log file", "persist directory" and yeah, "servers". I was thinking of copying them over periodically but I set a fair amount of metadata via log-courier before sending to Logstash. Because of that, and that the data and files will change over time, I didn't want to have to maintain a hack of copying, setting metadata, etc. - I really want the logs to be sent to the dev Logstash servers in the exact manner as they're sent to prod without too much magic. Hopefully this will work out. Thanks for responding and most importantly, thank you for log-courier!
from log-courier.
Hi @cjs226span
I've thought about it a bit and I can't really think of a good way to separate this out. If shipping to production slows down, shipping to dev would too. If shipping to dev slowed down, shipping to production would too. Only way around it would be to maintain two separate event queues, populated separately, which flush independently. So that means reading from a file twice with two separate threads, one for each queue. So I think at that points its probably better to just run two instances.
If you have any thoughts etc do let me know.
Jason
from log-courier.
Related Issues (20)
- Duplicates observed when log-courier configuration file is overwritten HOT 3
- Undocumented behavior for paths matched by multiple fileglobs HOT 1
- log-courier can't seem to handle `~` for home in certain contexts. HOT 2
- error in logstash 7 HOT 4
- log-courier and logstash > 7.4.0? HOT 10
- Throughput question HOT 15
- does the includes configuration still work correctly? HOT 3
- Hold time setting not closing files properly
- syslog "progname" uses a full path rather than a basename HOT 1
- Information about payloads / different debug level ? HOT 3
- error while compiling code in the command line using MK in windows for Z wave HOT 1
- version.rb missing from the plugin package
- log-courier admin socket stuck HOT 2
- lc-admin does not default to tcp transport HOT 2
- order of files in lc-admin UI
- Logstash configuration auto reload is blocked by log-courier input HOT 2
- PPA packaging is using deprecated compatibility level 9 HOT 1
- 2.10.0 packages missing from PPA (Ubuntu)
- Receiver reload can cause panic
- TCP streaming receiver aborts if too many events received
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from log-courier.