Prospector taking long time to scan 300k files

[driskell]

Prospector taking long time to scan 300k files

[driskell]

Hi @java2kus

The problem here is the glob match will still return 300k files and for it to know whether a file is old or new etc it will still need to check it via stat()

If we tell it to stop remembering it will just see them again.

It's a complex issue and I can't think of any solution within courier. You may be best running your own archived script to move old files to another directory.

Having so many files in one folder will cause other issues too and wouldn't be isolated to courier.

Jason

[java2kus]

Hi Jason,

Thanks for the response. We have these files in multiple folders. The folder structure goes something like this:

/<businesseventname>/YYMMDDHH/*.xml

We actually move these files every week. I guess, we will have to change the job to move it every 1 hour. Thanks for the suggestion.

[driskell]

I was thinking about it and I can see a use for a dead time action option. It would work by the user setting dead time to about an hour, then the action to delete or, say, move:/archive/ or something. This would be the best way to reliably ensure files are only archived after they are completed, as using a job to do it might cause skipped files if Logstash was really busy or down. (dead time only triggers if the file is fully processed and received by Logstash, and last modification is older)

Just for a bit more background - so you end up with 300k files each day? That's quite significant!

I'll log down dead time action as an idea. I can't commit to adding it but I will definitely explore it at some point if it sounds feasible to you. What do you think?

[driskell]

Sorry just saw you already mentioned it's 300k a day! Disregard that question.

[java2kus]

300k files a day is a bit unusual and this is because logging of the events for auditing and troubleshooting purposes came as an after thought. Typically, we would have designed the logging architecture using a high performance message broker but then it started with just logging around 10k events a day (where this was not felt necessary) to the current volume.

On the idea, I like the dead time action concept. This way we wont be skipping events (due to a hourly job running to move events into archive) even if logstash/elasticsearch is overwhelmed with events. This dead time action should only get triggered on after receiving a confirmation from logstash that the event has been processed (I know there is a bug in logstash because of the internal buffer pool which may result in some events getting skipped in case of crash). I am just learning programming in go and hence would maybe modify the source to implement a quick hack of the idea until you decide on the feasibility. Thanks!

[driskell]

Closing as it’s a fairly big task and no bandwidth. And would mean a large rewrite of prospector.