Comments (4)
Just call TemplateMiner.add_log_message()
on each log line to get the template.
https://github.com/IBM/Drain3/blob/ac71a48dc08ee6eef08961aedd2e9db7ee09aa8a/drain3/template_miner.py#L114
The masking capability is still there, you should define the regex masks in the config object/file for TemplateMiner.
from drain3.
How can I get the masked content from each log?
result = template_miner.add_log_message('Invalid user test9 from 52.80.34.196')
result
{'change_type': 'none', 'cluster_id': 2, 'cluster_size': 14552, 'template_mined': 'Invalid user <*> from ', 'cluster_count': 49}
add_log_message only return template, how can I get <*>: test9 and : 52.80.34.196 ? Is there any builtin function for that?
from drain3.
How can I get the masked content from each log?
result = template_miner.add_log_message('Invalid user test9 from 52.80.34.196')
result
{'change_type': 'none', 'cluster_id': 2, 'cluster_size': 14552, 'template_mined': 'Invalid user <*> from ', 'cluster_count': 49}add_log_message only return template, how can I get <*>: test9 and : 52.80.34.196 ? Is there any builtin function for that?
I modified the get_parameter_list in logpai/logparser, and it works..
extra_delimiters = ["_"] # In drain3.ini
def get_parameter_list(template, content):
for deli in extra_delimiters:
content = re.sub(deli,' ',content)
template_regex = re.sub(r"<.{1,}?>", "<*>", template)
if "<*>" not in template_regex: return []
template_regex = re.sub(r'([^A-Za-z0-9])', r'\\\1', template_regex)
template_regex = re.sub(r'\\ +', r'\\s+', template_regex)
template_regex = "^" + template_regex.replace("\<\*\>", "(.*?)") + "$"
parameter_list = re.findall(template_regex, content)
parameter_list = parameter_list[0] if parameter_list else ()
parameter_list = list(parameter_list) if isinstance(parameter_list, tuple) else [parameter_list]
return parameter_list
from drain3.
@cwyalpha thanks for the code snippet. I will integrate this function soon.
One problem is when the log message contains <xxx>
but that's not a mask.
I am going to add a configurable mask prefix and suffix instead of using <> always. This can somehow help to deal with such a case, by overriding the mask wrapper with something more unique.
from drain3.
Related Issues (20)
- Delete cluster from drain dict id_to_cluster | Impact | procedure HOT 1
- Skip Masking/cluster particular tokens HOT 4
- I cannot get value vector from match HOT 1
- Previously trained messages cannot always be matched HOT 7
- `get_parameters_list` can return incorrect parameters HOT 4
- Some questions about drain_bigfile_demo HOT 3
- Can i change the state saving name and path HOT 1
- Error when running the example. HOT 1
- parallel log ingestions HOT 4
- Extra delimiters in config HOT 1
- Only mask_name * is used HOT 2
- Issue with match method in Drain class HOT 1
- specify a log file HOT 1
- Saving log template/cluster and ID for each log HOT 2
- Error parsing logs: "ZeroDivisionError: float division by zero" HOT 4
- Restrictions on matching mode HOT 2
- About parameter `full_search_strategy` in drain match method HOT 12
- Windows regular expression HOT 1
- Drain3 deprecation warning with pip install command. HOT 2
- visualize drain parse tree (feature) HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from drain3.