kitabisa/teler

Real-time HTTP Intrusion Detection

Important

Dear users,

We wanted to share some important updates regarding the development of teler IDS.

🛑 The development of teler IDS is currently on hold. However, we're thrilled to inform you that we've decided to take a bold step by embarking on a complete refactor, starting from scratch to enhance the overall development process. 🚀

In parallel, we're actively laying down the roadmap for teler IDS v3. 📝
What we can reveal at this point is that version 3 will leverage eBPF & teler-waf as its support, promising advanced features and capabilities. 🌟

If you have feedback or suggestions regarding the features you'd like to see in teler IDS v3, kindly participate in our anonymous survey 👉 https://s.surveyplanet.com/8he6p7rq

Stay tuned for more updates as we continue to innovate and enhance our intrusion detection system! Tipis-tipis asal menyala abangkuuhh! 🔥🔝🙌🏼

Best!
dw1

---

_{As announced in teler-announce#Fm8Yjh_4of0.}

Note

In the meantime, you can install and download the latest v2 version to continue using it.

teler

teler

Real-time HTTP Intrusion Detection
Contribute · What's new · Report Bug · Request Feature

---

teler is an real-time intrusion detection and threat alert based on web log that runs in a terminal with resources that we collect and provide by the community. ❤️

CLI	Dashboard

Note

If you upgrade from prior to v2 frontwards there will be some break changes that affect configuration files. Appropriate adaptations can refer to teler.example.yaml file.

See also:

• kitabisa/teler-waf: teler-waf is a Go HTTP middleware that provide teler IDS functionality to protect against web-based attacks and improve the security of Go-based web applications. It is highly configurable and easy to integrate into existing Go applications.

Table of Contents

• Features
• Why teler?
• Demo
• Documentation
• Supporting Materials
• Contributors
  • Resources
• Pronunciation
• Changes
• License

Features

• Real-time: Analyze logs and identify suspicious activity in real-time.

• Alerting: teler provides alerting when a threat is detected, push notifications include Slack, Mattermost, Telegram and Discord.

• Monitoring: We've our own metrics if you want to monitor threats easily, and we use Prometheus for that.

• Logging: is also provided in file form or sends detected threats to the Zinc logs search engine.

• Latest resources: Collections is continuously up-to-date.

• Minimal configuration: You can just run it against your log file, write the log format and let teler analyze the log and show you alerts!

• Flexible log formats: teler allows any custom log format string! It all depends on how you write the log format in configuration file.

• Custom threat rules: Want to reach a wider range of threats instead of engine-based (default) rules? You can customize threat rules!

• Incremental log processing: Need data persistence rather than buffer stream? teler has the ability to process logs incrementally through the on-disk persistence options.

Why teler?

teler was designed to be a fast, terminal-based threat analyzer. Its core idea is to quickly analyze and hunt threats in real time!

Demo

Here is a preview of teler with conditions of use as:

Buffer-streams	Incremental

Documentation

All related documentation about installation, usage & configuration is on teler.app.

Supporting Materials

• teler - Protect Your WebApp! Talks were brought to the OWASP Jakarta: Virtual AppSec Indonesia 2020 event.
• Tutorial: Cyber Threat Hunting - Useful Threat Hunting Tools (Part One), Semi Yulianto gave a brief explanation and how to use teler in the video.
• Empowering Teler HTTP Intrusion Detection as WAF with Fail2ban.
• Detecting web attacks using Wazuh and teler, integrate teler with a unified XDR and SIEM platform, Wazuh.

Contributors

This project exists thanks to all the people who contribute. To learn how to setup a development environment and for contribution guidelines, see CONTRIBUTING.md.

Dwi Siswanto 💻 📖 ⚠️ 🤔	ProjectDiscovery 🔧	Anton Egorov 🔧	0ktavandi 🤔	Fik 🎨	fairyhunter13 ⚠️	Zufar Dhiyaulhaq 💻
Aldin Setiawan 💻 ️️️️♿️	Noah Petherbridge 🔧	Zackky Muhammad 🚇	Andy 🔧	Aliaksandr Valialkin 🔧	Markus Tenghamn 🐛	Rick 🚧 💻
Michael BOUVY 📖	oelnaggar 📖 🐛

Resources

All external resources used in this teler are NOT provided by us. See all peoples who involved in this resources at teler Resource Collections.

Pronunciation

jv_id • /télér/ — bagaimana bisa seorang pemuda itu teler hanya dengan meminum sloki ciu (?)

Changes

For changes, see the CHANGELOG.md.

Community

We use the Google Groups as our dedicated mailing list. Subscribe to teler-announce via [email protected] for important announcements, such as the availability of new releases. This subscription will keep you informed about significant developments related to teler IDS, teler WAF, teler Proxy, and teler Resources.

For any inquiries, discussions, or issues are being tracked here on GitHub. This is where we actively manage and address these aspects of our community engagement.

License

This program is developed and maintained by members of Kitabisa Security Team, and this is not an officially supported Kitabisa product. This program is free software: you can redistribute it and/or modify it under the terms of the Apache license. Kitabisa teler and any contributions are copyright © by Dwi Siswanto 2020-2022.

teler's Issues

Log Format issues

Hello, every time I run the tool I get the same error:

[WRN] No logs analyzed, did you write log format correctly?

I have the tool configured but the problem persists.

[FEATURE] Replace Alerting System with FalcoSideKick

Is your feature request related to a problem? Please describe.
Instead of developing your own alerting sub-system, I think it might be preferable to use something that already supports dozens of backends. FalcoSideKick might be a good option

Describe the solution you'd like
See above

[FEATURE] mattermost integration

Please could you add mattermost integration, configuration would be similar to slack but a server name would be needed as well as a token (or just the entire webhook which is a URL containing server and token)

I believe the format would be the same as for slack so it is just a matter of changing the post address accordingly?

This link may be useful to see what needs doing:-

https://docs.mattermost.com/developer/webhooks-incoming.html?highlight=webhook%20slack

[DOCS] Apache Log Example Incorrect

Describe the bug

The Apache log format example in the official documentation looks like this:

log_format: |
  $remote_addr - $remote_user [$time_local] "$request_method $request_uri $request_protocol" $status $body_bytes_sent

but it will not match because it misses the last two fields. Instead, it should look like this:

log_format: |
  $remote_addr - $remote_user [$time_local] "$request_method $request_uri $request_protocol" $status $body_bytes_sent "$http_referer" "$http_user_agent"

[FEATURE] Support RAW HTTP CVE templates to detect

Is your feature request related to a problem? Please describe.
So that resources are used properly.

Describe the solution you'd like
Parse RAW HTTP request.

Describe alternatives you've considered
Using net/http.ReadRequest.

Additional context
N/A.

[FEATURE] Add Exporters to Prometheus

Is your feature request related to a problem? Please describe.
add exporters in teler, so we can get a statistical data

Describe the solution you'd like
N/A

Describe alternatives you've considered
N/A

Additional context
N/A

[FEATURE] Add CVEs resource

Additional context
Scrapping from nuclei-templates/cves/.

[DOCS] Use fully qualified Docker image name

Summary

In the REAMDE, Docker image name is reference as teler, which is not recognized by Docker.

Fully qualified image name is kitabisa/teler and should be used in docker run ... commands.

Motivation

Have ready-to-use examples in README for Docker users.

[FEATURE] Importing rules from path

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[FEATURE] Let all 3xx status code in Directory Bruteforces

[BUG] test label commenter

foo bar bar

[FEATURE] Add fail2ban

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[BUG] make build

command line option -v, the output is always 1.0.1

run bin/teler -v the output is always

$ bin/teler -v
teler 1.0.1

cause the version varibale defined in constants.go is not overwritten at build time in Makefile

To Reproduce
just clone the repo, and run make build to compile the binary, then run bin/teler -v
the output is always

$ bin/teler -v
teler 1.0.1

Expected behavior
at this time, the output version should be

 bin/teler -v
teler v1.0.1-29-g62623bc

if we tag a new commit, the -v option should produce a new version

Screenshots
sorry for my company network environment, I can just describe in text

Environment:
I think this bug should reproduce in all OS
I am using windows10 and git-bash

• OS: [windows10]
• OS version: [windows10]
• teler Version the problem is about version, so ........

[MAINTENANCE] Write tests

Pass	Packages
⭕	ktbs.dev/teler/cmd/teler
⭕	ktbs.dev/teler/common
⭕	ktbs.dev/teler/internal/alert
⭕	ktbs.dev/teler/internal/runner
⭕	ktbs.dev/teler/pkg/errors
⭕	ktbs.dev/teler/pkg/matchers
⭕	ktbs.dev/teler/pkg/parsers
⭕	ktbs.dev/teler/pkg/requests
⭕	ktbs.dev/teler/pkg/teler
⭕	ktbs.dev/teler/resource

[FEATURE] Add custom threat rules

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
e.g.

    customs:
      - name: Large File Upload
        condition: AND
        rules:
          - element: body_bytes_sent
            pattern: \d{6,}

          - element: request_method
            pattern: P(OST|UT)

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[BUG] Whitelists ignored on query parameters

Describe the bug

Parameters & queries ignore whitelists.

[BUG] Issue with nginx log format

Describe the bug

I've trouble getting the below logformat to work. Am I missing something? What

    log_format  main_timed  '$remote_addr - $remote_user [$time_local] "$request" '
                            '$status $body_bytes_sent "$http_referer" '
                            '"$http_user_agent" "$http_x_forwarded_for" '
                            '$request_time $upstream_response_time $pipe $upstream_cache_status';

Example row:

2.55.123.55 - - [03/Nov/2021:14:12:54 +0100] "GET /api/v1/settings HTTP/1.1" 200 208 "-" "iOS" "-" 0.605 0.605 . -

config:

log_format: |
  $remote_addr - $remote_user - [$time_local]
  "$request_method $request_uri $request_protocol" $status $body_bytes_sent
  "$http_referer" "$http_user_agent" "$http_x_forwarded_for" $request_time $upstream_response_time $pipe $upstream_cache_status

Output

[INF] Analyzing...
[WRN] No logs analyzed, did you write log format correctly?
[INF] Done!

Environment (please complete the following information):

• OS: Mac
• teler Version 1.2.2

[FEATURE] metrics for custom threat rules

is this feature already in progress?

I already create it on my repo, free to feel for testing it and maybe add some suggestions.

if everything is good let me create the PR.here some PoC

Where are the config files located?

Where can I find those files?

common-web-attacks.json
cves.json
bad-ip-addresses.txt
bad-referrers.txt
bad-crawlers.txt
directory-bruteforces.txt

[ASK] Teler won't build

Describe the bug

Cannot run make build. Exits with following errors:

To Reproduce

Steps to reproduce the behavior:

On a fresh Ubuntu install, install golang, clone the repo and run make build.

Your teler config file...
The default one.

// Please redacted your token and/ other sensitive informations

Expected behavior

It builds without errors

Screenshots

If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• OS: Ubuntu Bionic (18.04)
• OS version: Linux vpn-machine 4.15.0-151-generic #157-Ubuntu SMP Fri Jul 9 23:07:57 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
• teler Version N/A

Additional context
Add any other context about the problem here. Full output log is probably a helpful thing to add here.

• See screenshot

[DOCS] Set up Wiki pages

Move the documentation on Wiki

[FEATURE] Improve directory bruteforce detection

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[BUG] Improves whitelist for CWA & CVE

[BUG] empty CVEs rule resources

Please refer to https://github.com/kitabisa/teler-resources/issues/2.

[BUG]

Describe the bug

maybe this is not a bug, but I can't find a suitable classification of this issue.

the Dockerfile add a useless instruction RUN mkdir -p /app
just as the docker documentation says: If the WORKDIR doesn’t exist, it will be created, so there is no need to add this insctruction before WORKDIR /app, but if you indeed want to add it, it works fine!
so leave it or not? it's up to you.

And for linux platform, the go build produced binary wasn't added to .gitignore

To Reproduce

nothing

Your teler config file...
// Please redacted your token and/ other sensitive informations

Expected behavior

A clear and concise description of what you expected to happen.

Screenshots

If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• OS: [e.g. mac, linux]
• OS version: [uname -a]
• teler Version [teler --version]

Additional context
Add any other context about the problem here. Full output log is probably a helpful thing to add here.

[BUG] Panic: url.Query()

Describe the bug

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x60 pc=0x57c702]

goroutine 72 [running]:
net/url.(*URL).Query(0x0, 0x2f)
        /snap/go/6274/src/net/url/url.go:1032 +0x22
ktbs.dev/teler/pkg/teler.Analyze(0xc00006a370, 0xc00039a038, 0xc000667f01, 0x1)
        /home/dw1/Tools/teler/pkg/teler/teler.go:46 +0x112b
ktbs.dev/teler/internal/runner.New.func1(0xc00001c6c0, 0xc00006a370, 0xc000018460)
        /home/dw1/Tools/teler/internal/runner/runner.go:35 +0x91
created by ktbs.dev/teler/internal/runner.New
        /home/dw1/Tools/teler/internal/runner/runner.go:33 +0xfd
tail: error writing 'standard output': Broken pipe

[FEATURE] Caching resources to local user-level cache directory

Additional context
It supports cross-platforms.

test: ignore this

[BUG] Custom whitelists do not match

Describe the bug

I can't use custom excludes to reduce false positives.

To Reproduce

# Lighttpd default log format
log_format: |
  $remote_addr $host $remote_user [$time_local] "$request_method $request_uri $request_protocol" $status $body_bytes_sent "$http_referer" "$http_user_agent"

# Rules
rules:
  cache: true
  threat:
    excludes:
      - "79\\.0\\.10\\.100"
      - "^/favicon\\.ico"

But neither of them prevents this alert:

$ tail -n1000 lighttpd-access.log | teler -c ~/teler.yaml -o foobar.log
[04/Mar/2021:00:46:10 +0100] [79.0.10.100] [Directory Bruteforce] /favicon.ico

Expected behavior

A clear description of what you expected to happen: No output is expected.

Environment (please complete the following information):

• OS: GNU/Linux
• OS version: x86_64
• teler 1.1.0

[FEATURE] Using Webhook for Alert (if any)

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[BUG] Regex not matching Slack token

Describe the bug

The regex which validates Slack tokens expects the third part of the token to be exactly 12 digits. However, I generated a token via Slack which has 13 digits. I run my command like this:

tail -f -n300 /var/log/caddy.log | teler -c ~/teler.yml

I receive the following error.

[ERR] Error! Only validates token; please check your config file
[INF] Use "-h" flag for more info about command.
Terminated

To Reproduce

Steps to reproduce the behavior:
Go to Slack and create a new bot at (yourworkspace).slack.com/apps/manage/custom-integrations
Then try to use the new token in your yml configuration.

log_format: |
  $remote_addr - $remote_user [$time_local] "$request_method $request_uri $request_protocol" $status $body_bytes_sent

alert:
  active: true
  provider: "slack"

notifications:
  slack:
    token: "xoxb-nnnnnnnnnnn-nnnnnnnnnnnnn-XXXXXXXXXXXXXXXXXXXXXXXX"
    color: "#ffd21a"
    channel: "XXXXXXXXXXX"

Expected behavior

I expected the Slack token to be accepted as a valid token.

Environment (please complete the following information):

• OS: linux
• OS version: Linux Ubuntu-1604-xenial-64-minimal 4.15.0-96-generic #97~16.04.1-Ubuntu SMP Wed Apr 1 03:03:31 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
• teler Version: teler 0.0.4-dev

Additional context

I believe the issue is the regex found here

teler/pkg/matchers/patterns.go

Line 4 in f0f8ed5

PatternToken = `^(xox[p|b|o|a]-\d{10,12}-\d{12}-\w+)|(\d{9}:[a-zA-Z0-9_-]{35})|([MN][A-Za-z\d]{23}\.[\w-]{6}\.[\w-]{27})$`

I think the d{12} should be changed to d{12,13}.

[FEATURE] Add Traefik log parsing

All in the title.

Currently, this handles text logs like a charm, but it could get a bit complicated to parse JSON output like traefik does.

[FEATURE] Zinc search engine

Is your feature request related to a problem? Please describe.
Supporting https://github.com/prabhatsharma/zinc

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[DOCS] Support Envoy access log

Summary

I want to integrate the Envoy access log with Teler, a lot of service mesh tooling use Envoy as the main proxy. Do you know any variable that teler support from the access log itself? So I can make a PR with the correct Envoy format.

Some recommended topics to cover:
N/A

Motivation

• Envoy used in service mesh like Istio

[FEATURE] Add integration with abuseipdb

Is your feature request related to a problem? Please describe.
I know teler was a IDS,but i think it's will be great if teler can fight back against the threat
like reporting the source of ip address threat to abuseipdb

Describe the solution you'd like
N/A

Describe alternatives you've considered
N/A

Additional context
N/A

[FEATURE] Add credits

[FEATURE] Resource options

Is your feature request related to a problem? Please describe.
In its current state, teler always consumes internet connection if it wants to use resources.

Describe the solution you'd like
Give options to downloading resources and storing them locally: in other words the user DOES NOT need to use an internet connection if he wants to analyze logs with persistent data.

Describe alternatives you've considered
N/A

Additional context
Like adding an -dl or --download-resources flag; stores all resources at $HOME, checks if all resources are in local storage; otherwise it will use the internet connection to download resources.

[FEATURE] Input data source supports Kafka

For companies with a certain scale, Nginx is deployed in clusters, and the access.log file is distributed in each node machine, and the production environment server is not allowed to run programs that occupy unstable resources

Therefore, I hope to support Kafka as an input data source

// Thanks for open source this project, this is great work :)

[FEATURE] Validates custom threat rules element

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[FEATURE] Discord notification alerts

[BUG] panic: fatal error: concurrent map iteration and map write

Describe the bug

fatal error: concurrent map iteration and map write

To Reproduce

Steps to reproduce the behavior:

run this a couple of times on a logfile with no issues:

tail /var/log/httpd/domains/example.com.log | ./teler -c teler.yml

result

fatal error: concurrent map iteration and map write

goroutine 27 [running]:
runtime.throw(0x86a1c18, 0x26)
	/opt/hostedtoolcache/go/1.14.15/x64/src/runtime/panic.go:1116 +0x6a fp=0xa4b8c74 sp=0xa4b8c60 pc=0x807821a
runtime.mapiternext(0xa4b8edc)
	/opt/hostedtoolcache/go/1.14.15/x64/src/runtime/map.go:853 +0x481 fp=0xa4b8cbc sp=0xa4b8c74 pc=0x80550b1
runtime.mapiterinit(0x85dc6a0, 0xa5084a0, 0xa4b8edc)
	/opt/hostedtoolcache/go/1.14.15/x64/src/runtime/map.go:843 +0x189 fp=0xa4b8cc8 sp=0xa4b8cbc pc=0x8054b49
ktbs.dev/teler/pkg/teler.Analyze(0xa50f440, 0xa40e120, 0x0, 0x8050ebf)
	/home/runner/work/teler/teler/pkg/teler/teler.go:32 +0x2c0 fp=0xa4b8f10 sp=0xa4b8cc8 pc=0x8547fd0
ktbs.dev/teler/internal/runner.New.func3.1(0xa5800a0, 0xa50f440, 0x0, 0xa502158, 0xa40e120)
	/home/runner/work/teler/teler/internal/runner/runner.go:72 +0x72 fp=0xa4b8fd8 sp=0xa4b8f10 pc=0x8551652
runtime.goexit()
	/opt/hostedtoolcache/go/1.14.15/x64/src/runtime/asm_386.s:1337 +0x1 fp=0xa4b8fdc sp=0xa4b8fd8 pc=0x80a40b1
created by ktbs.dev/teler/internal/runner.New.func3
	/home/runner/work/teler/teler/internal/runner/runner.go:69 +0x93

goroutine 1 [chan receive]:
github.com/satyrius/gonx.(*Reader).Read(0xa48df84, 0xa48df40, 0x0, 0x0)
	/home/runner/go/pkg/mod/github.com/satyrius/[email protected]/reader.go:41 +0x3f
ktbs.dev/teler/internal/runner.New(0xa50f440)
	/home/runner/work/teler/teler/internal/runner/runner.go:126 +0x3fc
main.main()
	/home/runner/work/teler/teler/cmd/teler/main.go:20 +0x1f

goroutine 35 [syscall]:
os/signal.signal_recv(0x0)
	/opt/hostedtoolcache/go/1.14.15/x64/src/runtime/sigqueue.go:147 +0x12f
os/signal.loop()
	/opt/hostedtoolcache/go/1.14.15/x64/src/os/signal/signal_unix.go:23 +0x1a
created by os/signal.Notify.func1
	/opt/hostedtoolcache/go/1.14.15/x64/src/os/signal/signal.go:127 +0x33

goroutine 22 [chan receive]:
ktbs.dev/teler/internal/runner.New.func2(0xa5ae080, 0xa50c200, 0xa49c01c)
	/home/runner/work/teler/teler/internal/runner/runner.go:57 +0x2d
created by ktbs.dev/teler/internal/runner.New
	/home/runner/work/teler/teler/internal/runner/runner.go:56 +0x246

goroutine 23 [chan receive]:
ktbs.dev/teler/internal/runner.New.func3(0xa50c200, 0xa5800a0, 0xa50f440, 0x0, 0xa502158)
	/home/runner/work/teler/teler/internal/runner/runner.go:67 +0xa7
created by ktbs.dev/teler/internal/runner.New
	/home/runner/work/teler/teler/internal/runner/runner.go:66 +0x312

goroutine 24 [semacquire]:
sync.runtime_Semacquire(0xa42a0a8)
	/opt/hostedtoolcache/go/1.14.15/x64/src/runtime/sema.go:56 +0x36
sync.(*WaitGroup).Wait(0xa42a0a0)
	/opt/hostedtoolcache/go/1.14.15/x64/src/sync/waitgroup.go:130 +0x7c
github.com/satyrius/gonx.MapReduce.func1(0xa432180, 0x8775d10, 0xa40c090, 0xa5055c0, 0xa)
	/home/runner/go/pkg/mod/github.com/satyrius/[email protected]/mapreduce.go:67 +0x104
created by github.com/satyrius/gonx.MapReduce
	/home/runner/go/pkg/mod/github.com/satyrius/[email protected]/mapreduce.go:26 +0x96

goroutine 25 [chan receive]:
github.com/satyrius/gonx.(*ReadAll).Reduce(0x8bc038c, 0xa5055c0, 0xa4321c0)
	/home/runner/go/pkg/mod/github.com/satyrius/[email protected]/reducer.go:23 +0x57
created by github.com/satyrius/gonx.MapReduce
	/home/runner/go/pkg/mod/github.com/satyrius/[email protected]/mapreduce.go:73 +0xe1

goroutine 38 [runnable]:
regexp/syntax.(*compiler).inst(...)
	/opt/hostedtoolcache/go/1.14.15/x64/src/regexp/syntax/compile.go:173
regexp/syntax.Compile(0xa5ae400, 0xa5ae400, 0x2, 0x2)
	/opt/hostedtoolcache/go/1.14.15/x64/src/regexp/syntax/compile.go:84 +0x186
regexp.compile(0x868de83, 0x9, 0x80000d4, 0x85dc7e0, 0xa5086e0, 0x868d034)
	/opt/hostedtoolcache/go/1.14.15/x64/src/regexp/regexp.go:178 +0x9f
regexp.Compile(...)
	/opt/hostedtoolcache/go/1.14.15/x64/src/regexp/regexp.go:133
regexp.MustCompile(0x868de83, 0x9, 0xa5120e0)
	/opt/hostedtoolcache/go/1.14.15/x64/src/regexp/regexp.go:309 +0x39
ktbs.dev/teler/pkg/matchers.IsMatch(0x868de83, 0x9, 0xa5fc2ae, 0x3, 0xa550200)
	/home/runner/work/teler/teler/pkg/matchers/regex.go:10 +0x4b
ktbs.dev/teler/pkg/teler.Analyze(0xa50f440, 0xa502180, 0x1, 0x8050ebf)
	/home/runner/work/teler/teler/pkg/teler/teler.go:212 +0x1ae1
ktbs.dev/teler/internal/runner.New.func3.1(0xa5800a0, 0xa50f440, 0x0, 0xa502158, 0xa502180)
	/home/runner/work/teler/teler/internal/runner/runner.go:72 +0x72
created by ktbs.dev/teler/internal/runner.New.func3
	/home/runner/work/teler/teler/internal/runner/runner.go:69 +0x93

goroutine 7 [runnable]:
github.com/satyrius/gonx.MapReduce.func1.1(0xa42a0a0, 0xa432180, 0xa546000, 0x8775d10, 0xa40c090, 0xa5055c0)
	/home/runner/go/pkg/mod/github.com/satyrius/[email protected]/mapreduce.go:46 +0x66
created by github.com/satyrius/gonx.MapReduce.func1
	/home/runner/go/pkg/mod/github.com/satyrius/[email protected]/mapreduce.go:43 +0xd6

goroutine 8 [runnable]:
github.com/satyrius/gonx.MapReduce.func1.1(0xa42a0a0, 0xa432180, 0xa546000, 0x8775d10, 0xa40c090, 0xa5055c0)
	/home/runner/go/pkg/mod/github.com/satyrius/[email protected]/mapreduce.go:46 +0x66
created by github.com/satyrius/gonx.MapReduce.func1
	/home/runner/go/pkg/mod/github.com/satyrius/[email protected]/mapreduce.go:43 +0xd6

goroutine 9 [runnable]:
github.com/satyrius/gonx.MapReduce.func1.1(0xa42a0a0, 0xa432180, 0xa546000, 0x8775d10, 0xa40c090, 0xa5055c0)
	/home/runner/go/pkg/mod/github.com/satyrius/[email protected]/mapreduce.go:46 +0x66
created by github.com/satyrius/gonx.MapReduce.func1
	/home/runner/go/pkg/mod/github.com/satyrius/[email protected]/mapreduce.go:43 +0xd6

goroutine 10 [runnable]:
github.com/satyrius/gonx.(*Entry).SetField(...)
	/home/runner/go/pkg/mod/github.com/satyrius/[email protected]/entry.go:50
github.com/satyrius/gonx.(*Parser).ParseString(0xa40c090, 0xa5fc6c0, 0x115, 0x0, 0x0, 0x0)
	/home/runner/go/pkg/mod/github.com/satyrius/[email protected]/parser.go:46 +0xf7
github.com/satyrius/gonx.MapReduce.func1.1(0xa42a0a0, 0xa432180, 0xa546000, 0x8775d10, 0xa40c090, 0xa5055c0)
	/home/runner/go/pkg/mod/github.com/satyrius/[email protected]/mapreduce.go:53 +0x97
created by github.com/satyrius/gonx.MapReduce.func1
	/home/runner/go/pkg/mod/github.com/satyrius/[email protected]/mapreduce.go:43 +0xd6

goroutine 11 [runnable]:
github.com/satyrius/gonx.MapReduce.func1.1(0xa42a0a0, 0xa432180, 0xa546000, 0x8775d10, 0xa40c090, 0xa5055c0)
	/home/runner/go/pkg/mod/github.com/satyrius/[email protected]/mapreduce.go:46 +0x66
created by github.com/satyrius/gonx.MapReduce.func1
	/home/runner/go/pkg/mod/github.com/satyrius/[email protected]/mapreduce.go:43 +0xd6

goroutine 12 [runnable]:
github.com/satyrius/gonx.MapReduce.func1.1(0xa42a0a0, 0xa432180, 0xa546000, 0x8775d10, 0xa40c090, 0xa5055c0)
	/home/runner/go/pkg/mod/github.com/satyrius/[email protected]/mapreduce.go:46 +0x66
created by github.com/satyrius/gonx.MapReduce.func1
	/home/runner/go/pkg/mod/github.com/satyrius/[email protected]/mapreduce.go:43 +0xd6

goroutine 13 [runnable]:
github.com/satyrius/gonx.MapReduce.func1.1(0xa42a0a0, 0xa432180, 0xa546000, 0x8775d10, 0xa40c090, 0xa5055c0)
	/home/runner/go/pkg/mod/github.com/satyrius/[email protected]/mapreduce.go:46 +0x66
created by github.com/satyrius/gonx.MapReduce.func1
	/home/runner/go/pkg/mod/github.com/satyrius/[email protected]/mapreduce.go:43 +0xd6

goroutine 16 [runnable]:
reflect.(*structType).FieldByName(0x85fa060, 0x868ca05, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/opt/hostedtoolcache/go/1.14.15/x64/src/reflect/type.go:1343 +0x23c
reflect.(*rtype).FieldByName(0x85fa060, 0x868ca05, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/opt/hostedtoolcache/go/1.14.15/x64/src/reflect/type.go:936 +0x6f
reflect.Value.FieldByName(0x85fa060, 0xa47e000, 0x199, 0x868ca05, 0x6, 0x199, 0x8050d14, 0xa)
	/opt/hostedtoolcache/go/1.14.15/x64/src/reflect/value.go:888 +0x66
ktbs.dev/teler/pkg/teler.Analyze(0xa50f440, 0xa47e000, 0x0, 0x8050ebf)
	/home/runner/work/teler/teler/pkg/teler/teler.go:23 +0xdb
ktbs.dev/teler/internal/runner.New.func3.1(0xa5800a0, 0xa50f440, 0x0, 0xa502158, 0xa47e000)
	/home/runner/work/teler/teler/internal/runner/runner.go:72 +0x72
created by ktbs.dev/teler/internal/runner.New.func3
	/home/runner/work/teler/teler/internal/runner/runner.go:69 +0x93

[BUG] Incorrectly accumulates parameters that cause false-positives to detect CVE

test: on hold issue

[FEATURE] Add a whitelist

Is your feature request related to a problem? Please describe.
Add whitelists in teler configuration file; which supports all threat categories.

Describe the solution you'd like
N/A

Describe alternatives you've considered
N/A

Additional context
Reducing false-positive results.

[BUG] CVE detect produce a bunch of false-positives

Separate log request format

$request format log must be separated again into 3 parts:

• Method name,
• Request URI, and
• Protocol

For example:

NGINX Ingress

• 127.0.0.1 - [127.0.0.1] - - [22/Jul/2020:00:34:14 +0000] "GET /_next/static/images/logo_ktbs_word_white-e12c3b97d3137c13e35f664a66b03096.png HTTP/2.0" 200 45088 "https://kitabisa.com/service-worker.js" "Mozilla/5.0 (Linux; Android 6.0; CPH1609) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Mobile Safari/537.36" 83 0.324 [kanvas-kanvas-app-prod-http] 10.0.0.100:9001 45088 0.324 200 eb344f13d3e1a7b72ef8ea9b37f157ad

Parser format as:

• $remote_addr - [$remote_addr] - - [$time_local] "$method $request_uri $protocol" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_length $request_time [$proxy_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status $req_id

Returns a record of type Entry (which is customized map[string][string]):

&{map[
	body_bytes_sent:45088
	http_referer:https://kitabisa.com/service-worker.js
	http_user_agent:Mozilla/5.0 (Linux; Android 6.0; CPH1609) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Mobile Safari/537.36
	method:GET
	protocol:HTTP/2.0
	proxy_upstream_name:kanvas-kanvas-app-prod-http
	remote_addr:127.0.0.1
	req_id:eb344f13d3e1a7b72ef8ea9b37f157ad
	request_length:83
	request_time:0.324
	request_uri:/_next/static/images/logo_ktbs_word_white-e12c3b97d3137c13e35f664a66b03096.png
	status:200
	time_local:22/Jul/2020:00:34:14 +0000
	upstream_addr:10.0.0.100:9001
	upstream_response_length:45088
	upstream_response_time:0.324
	upstream_status:200
]}

[FEATURE] Google Hack Database resource from Exploit-DB

Is your feature request related to a problem? Please describe.
Added external resources to check $request_uri on Google Hack Database from Exploit-DB.

Describe the solution you'd like
N/A.

Describe alternatives you've considered
N/A.

Additional context
Of course this will slow down analysis and alerting because it's calling the API.

[FEATURE] Telegram notification alerts

[BUG][ASK] : teller can not be run message "analyzing" appears

Describe the bug

when I run the command tail -f /var/log/apache2/access.log | teler -c /var/www/html/teler/teler.yaml -x 25 but the response from teler analyzing is not complete

To Reproduce

Steps to reproduce the behavior:

tail -f /var/log/apache2/access.log | teler -c /var/www/html/teler/teler.yaml -x 25

# To write log format, see https://github.com/kitabisa/teler#configuration
log_format: |
  $remote_addr - [$remote_addr] $remote_user - [$time_local] 
  "$request_method $request_uri $request_protocol" $status $body_bytes_sent 
  "$http_referer" "$http_user_agent" $request_length $request_time 
  [$proxy_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status $req_id
rules:
  cache: true
  threat:
    excludes:
      - "Common Web Attack"
      - "CVE"
      - "Bad IP Address"
      - "Bad Referrer"
      - "Bad Crawler"
      - "Directory Bruteforce"

    # It can be user-agent, request path, HTTP referrer, IP address and/or request query values parsed in regExp
    whitelists:
      # - "(curl|Go-http-client|okhttp)/*"
      # - "^/wp-login\\.php"
      # - "https://www\\.facebook\\.com"
      # - "192\\.168\\.0\\.1"

# prometheus:
  # active: false
  # host: "localhost"
  # port: 9099
  # endpoint: "/metrics"

alert:
  active: true
  provider: "slack"

notifications:
  slack:
    token: "xxxxxxxxxx"
    color: "#ffd21a"
    channel: "teler"

 # telegram:
   # token: "123456:ABC-DEF1234...-..."
   # chat_id: "-111000"

 # discord:
   # token: "NkWkawkawkawkawka.X0xo.n-kmZwA8aWAA"
   # color: "16312092"
   # channel: "700000000000000..."

Screenshots

https://drive.google.com/file/d/1vvbFHk9e-AVJgzUhIeaLynskmG24jYeT/view?usp=sharing

Environment (please complete the following information):

• OS: linux
• OS version: Linux kali 5.7.0-kali1-amd54 #1 SMP Debian 5.7.6-1kali2 (2020-07-01) x86_64 GNU/Linux
• teler Version : teler 1.0.1

Log Format Configurations

• NCSA Combined Log Format

%h %^[%d:%t %^] "%r" %s %b "%R" "%u"" "%u"%R%^,"%R","%u"

• NCSA Combined Log Format with Virtual Host

%^:%^ %h %^[%d:%t %^] "%r" %s %b "%R" "%u"%R%^,"%R","%u"

• Common Log Format (CLF)

%h %^[%d:%t %^] "%r" %s %b %s %b "%R" "%u"%R%^,"%R","%u"

• Common Log Format (CLF) with Virtual Host

%^:%^ %h %^[%d:%t %^] "%r" %s %b"%u"" "%u"%R%^,"%R","%u"

• W3C

%d %t %h %^ %^ %^ %m %r %^ %s %b %^ %^ %u %R%^,"%R","%u"

• CloudFront (Download Distribution)

%d\t%t\t%^\t%b\t%h\t%m\t%^\t%r\t%s\t%R\t%u\t%^,"%R","%u"

• Google Cloud Storage

"%x","%h",%^,%^,"%m","%U","%s",%^,"%b","%D",%^,"%R","%u"

• AWS Elastic Load Balancing (HTTP/S)

%dT%t.%^ %^ %h:%^ %^ %T %^ %^ %^ %s %^ %b "%r" "%u","%u"