Comments (2)
Drain3 is currently not designed for multiprocessing or even multithreading. For that to be supported, the parse tree that is created while ingesting data shall be saved on shared storage (multiprocessing) or have synchronized memory access (multithreading) - the later should be easier to achieve.
Drain3 can usually reach a processing rate of thousands of messages / second on modern hardware (single CPU). This date should be enough for log producing rates in many real-world use cases.
However, sometimes this is not enough, or you want to process historic data fast.
What can still be done:
-
You may separate the input logs by independent sources (e.g. process each data-center independently in a dedicated Drain3 instance).
-
You may shard your input logs by any other fashion you want and still process each shard with a Drain3 instance. However, keep in mind that each Drain3 instance will maintain its own list of mined templates and parse-tree, and the sequential numbers of mined templates will not match between instances. You may ignore the template ID and identify templates by the actual content e.g.
Request processed: <*>
or a hash of it, merging templates from all Drain3 instances together. The mining results will not be the same vs running a single instance Drain3, but should still be acceptable for many purposes. -
You may have multiple3 Drain3 instances that only process already-known templates, and a single instance that will process logs that do not match any known template. Whenever you encounter such a template, you will have to send it to the single instance, and then have other instances synchronize with the new state.
from drain3.
Thank you very much!!(∩_∩)
from drain3.
Related Issues (20)
- What is the difference between drain3 with logstash in elasticsearch ?
- visualize drain parse tree (feature) HOT 1
- Hi, I've been trying to use drain for running log anomaly detection on some logs.
- Log Matching on new data HOT 2
- Chinese and English hybrid log template mining HOT 5
- Some DRAIN templates with <*> do not have parameters extracted HOT 7
- PermissionError when running with Persistance
- Is it possible to freeze templates when trainning? HOT 2
- Add a py.typed marker file
- `extra_delimiters` does not account for prefixed/suffixed delimiters
- Drain3 in golang HOT 2
- Masking Prefix and Suffix should not be escaped HOT 1
- A interesting issues. HOT 1
- big_file demo result's first cluster content is empty
- masking question,if i want to output the masking such as real info of the date,how can i putput
- Avoid creating many redis connections when you want to have seperate buckets of templates. HOT 2
- [Question] HOT 2
- one question, how to do Incremental learning in drain3 training?
- one question, how to do Incremental learning in drain3 training? HOT 6
- Release a new https://pypi.org/ version to update dependencies HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from drain3.