Comments (4)
Hello,
What do you mean by "previous results"?
Can you provide an example of your need?
from drain3.
Hi @davidohana, and thanks for replying!
I'm using Drain for feature generation for a machine learning case. I have a dataframe of log messages that I run through Drain to extract the patterns and categorize log messages. I then add these features to the dataframe.
Let's say I have one row with the log message 'aa aa aa' and then later on 'aa aa ab' as in your example in the readme.
When Drain sees 'aa aa ab' after having seen 'aa aa aa', it learns that the template is actually 'aa aa <*>'.
My preprocessing of 'aa aa aa' will still hold the template 'aa aa aa', but what I want to do now is to update this with the new template.
I could of course keep track of the cluster_id's and just update them manually after every 'cluster_template_changed', but I'm just wondering if there's some built-in logic in this library that can enable me to take care of this more efficiently.
Does that make sense?
from drain3.
It is recommended to track clusters by the cluster ID and not the template, exactly because of this issue.
Look at the following sample output:
Starting Drain3 template miner
Checking for saved state
Saved state not found
Drain3 started with 'FILE' persistence
reading from std-in (input 'q' to finish)
aa aa aa
Saving state of 1 clusters with 1 messages, 952 bytes, reason: cluster_created (1)
{"change_type": "cluster_created", "cluster_id": 1, "cluster_size": 1, "template_mined": "aa aa aa", "cluster_count": 1}
parameters: []
aa aa ab
Saving state of 1 clusters with 2 messages, 1052 bytes, reason: cluster_template_changed (1)
{"change_type": "cluster_template_changed", "cluster_id": 1, "cluster_size": 2, "template_mined": "aa aa <:*:>", "cluster_count": 1}
parameters: ['ab']
aa aa cc
{"change_type": "none", "cluster_id": 1, "cluster_size": 3, "template_mined": "aa aa <:*:>", "cluster_count": 1}
parameters: ['cc']
The cluster_id
stays 1 but template_mined
changes.
Indeed if you save some repository of cluster ID to template mapping, you can update it after each cluster_template_changed
event.
BTW this is one of the differences from the original Drain. which identifies clusters by the hash code of the template ID.
from drain3.
Thanks a lot @davidohana, that perfectly explains it! If I would need the 'template_mined' further, then I'll just update the previous entries based on their 'cluster_id'.
Closing the question, thanks a lot!
from drain3.
Related Issues (20)
- Drain3 deprecation warning with pip install command. HOT 2
- visualize drain parse tree (feature) HOT 1
- Hi, I've been trying to use drain for running log anomaly detection on some logs.
- Log Matching on new data HOT 2
- Chinese and English hybrid log template mining HOT 5
- Some DRAIN templates with <*> do not have parameters extracted HOT 7
- PermissionError when running with Persistance
- Is it possible to freeze templates when trainning? HOT 2
- Add a py.typed marker file
- `extra_delimiters` does not account for prefixed/suffixed delimiters
- Drain3 in golang HOT 2
- Masking Prefix and Suffix should not be escaped HOT 1
- A interesting issues. HOT 1
- big_file demo result's first cluster content is empty
- masking question,if i want to output the masking such as real info of the date,how can i putput
- Avoid creating many redis connections when you want to have seperate buckets of templates. HOT 2
- [Question] HOT 2
- one question, how to do Incremental learning in drain3 training?
- one question, how to do Incremental learning in drain3 training? HOT 6
- Release a new https://pypi.org/ version to update dependencies HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from drain3.