Is there a way to add a wildcard template?

[logpai]

Is there a way to add a wildcard template?

[davidohana]

Can you give an example for what you try to achieve?

[V0id01]

I am working on a log anomaly detection project based on Deep Learning.
The training set is created by parsing corpus of log sequences. Based on that training set, a model creates vocabulary - a set of known log templates.
After training phase, model moves on to anomaly detection phase. Logs are provided in an online, streaming fashion. Based on previous logs, model starts to predict which log templates are most probable to occur, which makes it possible to evaluate if incoming log (after going through Drain) should be treated as anomaly.

The point is that the vocabulary, after training phase, should have constant size. By executing add_log_template on incoming logs, the number of clusters might increase, which could spoil anomaly detection.

A possible solution is to create a new cluster containing wildcard template (template = '<*>'), so that every log that will not be assigned to any other cluster will be assigned to the wildcard one.

[davidohana]

Few options:

1. Just add logs to Drain as usual, but after reaching n clusters, ignore the returned cluster-ID and use some constant ID
2. After reaching n clusters, stop ingesting to Drain and build some regex-rules based on Drain templates, and use those instead.
3. Modify Drain (send a PR) - add a configuration of max_clusters and in add_log_message() if no match for an existing cluster + cluster count reached limit, return the fallback cluster.