Comments (19)
Hi again
I've not tested with symlink but will see if I can reproduce this.
When you say "dead file" do you mean "dead time"?
from log-courier.
Yes, "dead time". Lack of caffeine :)
2014-06-29 11:18 GMT+02:00 driskell [email protected]:
Hi again
I've not tested with symlink but will see if I can reproduce this.
When you say "dead file" do you mean "dead time"?
—
Reply to this email directly or view it on GitHub
#7 (comment).
Fabio
from log-courier.
In the log-courier output, do you see "Registrar received X events"? This confirms that events are getting sent to the server side. If events aren't getting sent/received it will keep the file open as it's not sent the contents yet.
from log-courier.
The message you refer to is present in logs and I can confirm that events are sent. I've yet to look at the code (not knowing nothing about go it will take some time :) ) but could be possible that the file gets opened with the real name, as can be seen by lsof, and the age check is performed against the symlink? this can explain the behaviour...
from log-courier.
The age check isn't performed against any property of the file. It actually just watches the file and if it doesn't change in X hours, it stops harvesting. If it then subsequently changes it will start again.
So unless those old log files are still receiving new entries, it should stop on those. And you'll see a "Stopping harvest of %s; last change was %v ago" message in the logs.
Let me have a think on this to see if I can think of a reason it would do it. If you can find out when it does close the files (you say its after 20 hours) that would be useful... if it's 24hours then it may be the config didn't apply correctly (24h is the default) - in which case can you post your config?
Thanks for your patience.
from log-courier.
Sure... I made some works yesterday, so I'm waiting for 24h to pass. I'll send updates this evening.
Thanks for your cooperation!
from log-courier.
Ok, I have some more data.
The oldest file still open on log-courier ls has been updated more than 24 hours ago:
Jun 28 18:00 (mtime), but ctime is a bit different: Jun 29 00:10 (basically, post-midnight archiving). Time now is 20:34 Jun 29, so if the file is closed by log-courier according to mtime, something is not ok. If it rotates the file according to ctime, I have to wait a little bit to see the result.
However, in my conf file I have the following data (files section):
"files": [
{
"paths": [ "/var/log/log_current" ],
"fields": { "type": "app", "set": "system" },
"dead time": "1h"
}
]
so something does not sounds ok anyway :)
from log-courier.
When it says registrar received events does it say 1024?
Dead time works on the time of the last read line (so idle time). I wonder if it's been busy and not read to the end of the file yet?
Could you get the size of the files in bytes and look inside the .log-courier file? (It gets saved in the current directory and contains latest offsets if files - it will confirm if we've read to the end or not.)
Thanks.
from log-courier.
Yes, it says 1024 (mostly, sometimes it says 1 or 2, but immediately afterwards says again 1024).
Interesting enough, in .log-courier the filename used is log_current, the name of the link, but lsof shows that log-courier keeps open the real files (of course).
Looking at the inode, I got a bit puzzled. It seems that the inode bounces back and forth between two different inodes: one is the inode of the file pointed by the log_current symlink, where the other inode is from an older file. In either cases the offset is not corresponding to the file size... but the logs are flowing to logstash in a timely fashion, at least looking to elasticsearch latest entries.
/me scratching head.
Definitely not what I expected :)
from log-courier.
Ohhh that's a bug then. It's saving two different files under the same name. I fixed that with normal rotations but possibly symlink introduces an interesting difference that isn't handled properly.
It should stop saving the offset of the old file once it goes "out of scope" and only save that for the new file. Then to prevent loss you would need to keep all files "in scope" of course.
It explains the problem though - it's not read to end of those files yet so can't close them. Possibly logstash cannot keep up with the flow. This explains the spurts of 1024 then much less as logstash buffer fills. Would you agree?
I'll look into the overlapping save though that's not supposed to happen!
from log-courier.
I just checked and all files are still open, even the ones older than 24h so this seems to confirm your diagnosis.
I'll try to work on logstash to improve the situation, do you think this can improve the situation wrt not rotated files? It's a bit of showstopper at this moment...
I'm trying to look at code to be of some help, but as stated before not being knowledgeable on golang I'm a bit stuck on basics :)
from log-courier.
I've worked out the overlapping save where offsets flicks. I will push a fix for that later tonight hopefully so you can test.
However, files will still remain open - so it won't fix the actual problem for you. What you ideally need is a broker / cache for logs between the shippers and LogStash. This way instead of the shipper holding logs because it needs them - the broker takes all the logs really quick and it keeps them instead. Logstash then feeds from the broker.
If LogStash still can't keep up with the number of logs coming in though - this will just mean you fill the broker cache and again, logs stay open on the shipper. But with a broker you can easily add more Logstash instances to boost processing. Just set it up on another machine to feed from the same broker.
I use redis and have a logstash receiver instance that simply inputs, then outputs to redis. Then I hook up three or four other logstash indexer instances that input from redis and do filtering then output to elastic search.
To make sure you get max throughput on your logstash - make sure you use the "-w" flag. This sets the number of filter workers. With a single filter worker you get a single filter thread which uses only one CPU. I usually set it to twice CPU count. So I have quad core instances and use "-w 8". See http://logstash.net/docs/1.4.2/flags
Hope this helps
from log-courier.
Thanks for the hint, I will look at redis "cache" installation. (Already worked on -w parameter :) )
Le me know about the fix, I will be more than happy to test it.
from log-courier.
I pushed the fix just now to develop branch. If you can give it a try that will be great.
from log-courier.
Ok, online now. Tomorrow I'll check what's happened. Thanks for the patch, will report later.
from log-courier.
Now the oldest file left open by log-courier is 1h old, so it seems that your fix works as expected, many thanks!
from log-courier.
@cova-fe Do you need to run a cron job to create the symlink pointing to the current file? I'm doing this for rotated log files, but my problem is the file is not always rotated at exactly the same time every cycle. Do you have such issue?
from log-courier.
@angel-smile Unfortunately, I no longer have access to this specific setup, so I can't check if the issue you report appears also here...
from log-courier.
@angel-smile My guess would be the logging program itself did the symlinking, or it was done using the logrotated scripts. Basically whatever did the rotation. We can work out something that works for you in #225
from log-courier.
Related Issues (20)
- Duplicates observed when log-courier configuration file is overwritten HOT 3
- Undocumented behavior for paths matched by multiple fileglobs HOT 1
- log-courier can't seem to handle `~` for home in certain contexts. HOT 2
- error in logstash 7 HOT 4
- log-courier and logstash > 7.4.0? HOT 10
- Throughput question HOT 15
- does the includes configuration still work correctly? HOT 3
- Hold time setting not closing files properly
- syslog "progname" uses a full path rather than a basename HOT 1
- Information about payloads / different debug level ? HOT 3
- error while compiling code in the command line using MK in windows for Z wave HOT 1
- version.rb missing from the plugin package
- log-courier admin socket stuck HOT 2
- lc-admin does not default to tcp transport HOT 2
- order of files in lc-admin UI
- Logstash configuration auto reload is blocked by log-courier input HOT 2
- PPA packaging is using deprecated compatibility level 9 HOT 1
- 2.10.0 packages missing from PPA (Ubuntu)
- Receiver reload can cause panic
- TCP streaming receiver aborts if too many events received
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from log-courier.