Comments (6)
When looking at fast_match
I don't see any point to do the complicated self.get_seq_distance
calls for all clusters if len(tokens)==1
.
Would something like this not be enough?:
if cluster.log_template_tokens[0]==tokens[0]:
return cluster
It seems to considerable speed up the whole thing - the effect is less visible with max_clusters activated but that seems to have an very strong negative effect on performance in any case
from drain3.
Hello,
Seems like the change you suggest will keep the correctness of the algorithm, but you should prove it in a regression test.
Also please demonstrate that it improves speed.
A PR is welcome!
BTW, please check your masking - if you get many single token templates you might need to improve that.
from drain3.
Hi,
thanks for your reply. Perhaps my log is not very typical. I have general 50% rubbish (the whole line) I want to mask. Mostly its a recursive "LS" call that basically just lists thousands of filepathes (each is one token) - and each would be a single cluster. I could remove all these rubbish cases before feeding it to drain but I thought its better to have everything in one place. Additionally, there will be always new weird one token lines. Mylogs are around 500.000 lines long as users can basically do whatever they want and it will not be feasible to always adapt masks.
As for PR, I will try my best but at the moment I added more cache related workarounds that significantly improve the whole processing (e.g. I made another "cache" for the last few tokens/masks as we have a lot of repetition - assumption is that same loglines might repeat).
Such an improvement makes the change above obsolete - at least for my logs.
from drain3.
Typical usage of Drain is for extracting up to few thousands of templates. Perhaps with the new max_clusters
feature + some optimizations, you can avoid some masking and just "forget" rare clusters. If you end up with generic improvements, please do a PR.
from drain3.
I had to do some other topics but want to get back on drain - I plan to test it with billions of streamed loglines and it seems e.g. the patch shown here still adds some huge improvements. When I use e.g. drain_bigfile_demo.py
it doesnt really make a difference though - probably not too surprising, as it contains only like 50 clusters and most of the processing is regex and not drain. So at the moment, I cannot proof that it helps here.
Perhaps it makes sense to add some drain_hugefile_demo.py
for benchmarking and such. Would be interesting too, for improving the max_clusters
feature as this is as you said fundamental for this use case. Does anyone have an idea where to get a huge (~2GB) and very diverse log?
from drain3.
Another demo with a significantly bigger log file can be a great addition.
Perhaps you can use one of the datasets here: https://github.com/logpai/loghub ?
from drain3.
Related Issues (20)
- What is the difference between drain3 with logstash in elasticsearch ?
- visualize drain parse tree (feature) HOT 1
- Hi, I've been trying to use drain for running log anomaly detection on some logs.
- Log Matching on new data HOT 2
- Chinese and English hybrid log template mining HOT 5
- Some DRAIN templates with <*> do not have parameters extracted HOT 7
- PermissionError when running with Persistance
- Is it possible to freeze templates when trainning? HOT 2
- Add a py.typed marker file
- `extra_delimiters` does not account for prefixed/suffixed delimiters
- Drain3 in golang HOT 2
- Masking Prefix and Suffix should not be escaped HOT 1
- A interesting issues. HOT 1
- big_file demo result's first cluster content is empty
- masking question,if i want to output the masking such as real info of the date,how can i putput
- Avoid creating many redis connections when you want to have seperate buckets of templates. HOT 2
- [Question] HOT 2
- one question, how to do Incremental learning in drain3 training?
- one question, how to do Incremental learning in drain3 training? HOT 6
- Release a new https://pypi.org/ version to update dependencies HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from drain3.