Comments (8)
The problem in my case is i have different log sources where most of them are space separated while some of them have these special characters like pipe or comma separated. Is there any way i can handle this at runtime which works for all kind of patterns?
Also for my second query, Same pattern log, can you comment on how to make it same template group?
from logparser.
You could define different delimiters for different log sources. Another option is to define multiple delimiters:
# delimiters for white space, "=", and ":"
delimiters = '=|:|\s+'
wordL = re.split(delimiters, logmessage.strip())
For the same pattern log with extra tokens, you could merge different templates/log groups by comparing the template strings. If two templates are similar, their corresponding groups could be merged into one group.
from logparser.
Hi @PinjiaHe ,
For the same pattern log with extra tokens case.... Can you please help me on how to merge different templates/log groups by comparing the template string
For instance:
- A=a B=b C=c D=d(length 4)
- A=a' a B=b' b C=c' c D=d' d(length 8)
The above 2 log lines belong to different first layer node length 4 and length 8 respectively. What part of code should we modify to handle these cases? As per my understanding we create the first layer based on token and later look for templates matching the log line and create if it isn't existing. But we search for template in same bucket of first layer node and not all the buckets to improve performance.
Appreciate your help in this :)
from logparser.
But we search for template in same bucket of first layer node and not all the buckets to improve performance.
Right, but if you want to merge two groups, you need to add some code to link two groups to a single group (i.e., add another layer). In this case, they will output the same template. This may need some coding effort. We have implemented similar methods in
logparser/logparser/Drainjournal.py
Lines 342 to 372 in 68b638d
If all your logs are in the format of "A=a B=b ...", we would suggest finding "A, B, ..." first by inspecting word frequency.
from logparser.
Hi @PinjiaHe
Thanks for the reply. Is there any doc which explains what exactly is done in DrainJournal? Also why was this approach of getting a similar cluster not done in Drain as compare to DrainJournal... Was there any performance cost or some logical issue?
from logparser.
A draft could be found here.
We did not implement it in Drain (ICWS17) because most of the logs we encountered can be handled without merging the groups. Additionally, it's possible for the merging mechanism to wrongly merge two groups, leading to other parsing errors. Thus, a suitable parameter is important.
from logparser.
Thanks @PinjiaHe . I shall try out the suggestions you have given against my log files and see how it goes. One last query i have is that currently creates a new Root Node and new logCluL for each log file. Is there any way where we can make it global and have one root node and maintain 1 logCluL for all the logs and log files we parse through Drain?
from logparser.
The easiest way is to merge the log files first.
You could also modifier the log_to_dataframe method to let it accept a list of file names as parameters, and read them all.
logparser/logparser/Drain/Drain.py
Lines 298 to 315 in 8f1f1fa
from logparser.
Related Issues (20)
- The difference in number of records between raw data and parsed data HOT 1
- logrevised和log_corrected区别
- Support for multiline HOT 1
- How does the logpai/Drain3 repository compare to the implementation in this repository? HOT 1
- Controlling verbosity of output HOT 1
- Known log formats in benchmarks HOT 2
- Use Spell to parser HDFS have some question HOT 1
- Inconsistency with Templates and the EventTemplates in the Spark2k_corrected version file HOT 4
- raise KeyError in Drain HOT 2
- How to define log_format and regex[] for each dataset? HOT 1
- df_log always empty HOT 1
- bad escape \s in Drain.py HOT 7
- The regex is different HOT 3
- EventId in HDFS_template.csv
- Spell optimization in python using multiprocessing HOT 1
- What do precision and recall really mean in log parsing? HOT 2
- Configure logparser with python2.7
- I think there is a misspelling in '/logs/Andriod' HOT 1
- Unable to figure out how to use the logparser HOT 2
- running python Drain_demo.py , An error occurred
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from logparser.