crowdsecurity/crowdsec-docs

Crowdsec documentation

This repository holds the documentation for the crowdsec project.

Online version of this documentation is available here: https://doc.crowdsec.net/

crowdsec-docs's Issues

dashboard: improve documentation "how to install the dashboard using docker"

openresty-bouncer : indicate centos-based support

Current doc only refers to debian-based environements

Inform user on apply_on in Grok node list

Hey, I will create a PR for these changes!

When a user supplies a list of Grok nodes in a parser and they want all nodes to be evaluated they must omit the apply_on key from all nodes. If they do not the node that evaluates first and / or the only node with apply_on will be the value for all parsed information this was confusing and was not explained in docs!

I will do this shortly, just finding the right words!

Here a link to the caddy parser which I was following for my own to explain what I mean

Define tainted in docs

Suggest defining "tainted" (in Crowdsec context) somewhere in the docs. I didn't know what it meant without asking in Discord. As long as it's findable via search (example) then users like me would be okay.

Clarify docker pieces for multi server setup

https://doc.crowdsec.net/docs/user_guides/multiserver_setup
it'd really help if you clarified the docker environment variables you needed for a multi server setup on the agent end:

DISABLE_LOCAL_API=true
LOCAL_API_URL=http://host.ip.address:8080
AGENT_USERNAME=username
AGENT_PASSWORD="password"

The blog post you reference certainly helps, but there are a bunch of steps you'd do on bare metal that don't make sense in the context of containerisation, and would require editing the dockerfile to do.

The only other thing is that it's not immediately clear (at least it wasn't to me) that the LAPI and agent are part of the same container - when i was looking at the multi server setup initially, i was looking for an agent container I didn't need.

Error in Docs

Not sure where to post this, just a quick note that in https://doc.crowdsec.net/docs/parsers/create there is a invalid command cscli hubtest inspect (mentioned twice) I believe it should be cscli hubtest explain

Metabase_without_docker

Related to https://docs.crowdsec.net/blog/metabase_without_docker/

The repo for liberodark's helper script has been stale for quite some time.

I've made some significant improvements to the script, increasing automation for users, and making it easier to use and install.

Originally had hoped that my PR would have been looked at and merged by the owner, but doesn't seem to be any interest in doing so.

I'd be happy to take over maintenance of the script and apply any changes required if needs be.

Please let me know and if you're happy to, I'll sort out my repo to take over, and create a PR here to update the docs.

Ensure "install" section links to how to build from source as well

Document bultin pprof

document expression in grok patterns

cf. crowdsecurity/crowdsec#830

db_config section mentions both "host" and "db_host"

(but only "host" seems to work)

Installation should warn beforehand of the available ports that are required

otherwise, it might confuse people that already have services using port 8080

Hello

in observability/prometheus#exploitation-with-prometheus-server--grafana

is it a really hard thing to simply write about how to configure grafana's connection to prometheus and then configure the board ?????????????

https://doc.crowdsec.net/docs/observability/prometheus#exploitation-with-prometheus-server--grafana

is very rediculous that "hei , i show how crowdsec can be observed,use grafana !" and nothing else then , what XXXX of the writer's thought ?????

I say , God , teache me how , OK??

doc build warning

The readme says I should build the docs with yarn, however:

warning package-lock.json found. Your project contains lock files generated by tools other than Yarn. It is advised not to mix package managers in order to avoid resolution inconsistencies caused by unsynchronized lock files. To clear this warning, remove package-lock.json.

Should this be replaced by yarn.lock?

How to use captcha with WordPress bouncer?

There isn't any documentation for the wordpress bouncer on how to configure captcha

https://github.com/crowdsecurity/crowdsec-docs/blob/main/crowdsec-docs/versioned_docs/version-v1.3.0/bouncers/wordpress.mdx

Contributing: add doc on cscli hubtest and how to generate tests when adding new parser/scenario

centos7 / centos8: mistake in documentation

For crowdsec documentation (https://docs.crowdsec.net/docs/getting_started/install_crowdsec), it is mentionned that with must install with yum on centos8 and dnf on centos7, which is the invert.

For bouncers, we don't provide different installation method for centos7/centos8. (eg. https://docs.crowdsec.net/docs/bouncers/firewall)

CrowdSec build instructions

CrowdSec since 1.5 has alot more options to build RE2 / WASM / STATIC we should document these options better within docs as currently if you are not internal you dont know how to compile with these options unless you love reading makefiles

Add 1.5 options to configuration page

We added some option to config.yaml these should be documented in https://docs.crowdsec.net/docs/next/configuration/crowdsec_configuration maybe #384 already does?? but I think it just moving next -> 1.5.0

Maybe @blotus can confirm

Add db_config.use_wal documentation

Add blurb about use_wal support and what improvements it has over not using it?

Linked to
crowdsecurity/crowdsec#1860

Add documentation for Go/Python CTI clients

Mistake in example configuration for Gmail

What happened?

The example configuration for Gmail shows:

smtp_port: 587
encryption_type: ssltls # Required

Which yields the following error on attempted email send:

Mail Error on dialing with encryption type SSL/TLS: tls: first record does not look like a TLS handshake error

The Gmail SMTP docs say port 465 is for "SSL" and 587 for "TLS/STARTTLS".

Either of these work for me:

smtp_port:       465
encryption_type: ssltls

Or:

smtp_port:       587
encryption_type: starttls

Also, even though starttls works it isn't listed as a valid value in my email.yaml template:

# One of "ssltls", "none"
encryption_type: 

What did you expect to happen?

Email via Gmail works per docs example.

How can we reproduce it (as minimally and precisely as possible)?

I think just set up email with gmail per the docs example.

Anything else we need to know?

Suggest:

• Update example configuration for Gmail with valid port/encryption combo.
• Add starttls to email.yaml comment: # One of "ssltls", "starttls", "none"

Crowdsec version

2023/02/21 20:04:48 version: v1.4.6-debian-pragmatic-5f71037b40c498045e1b59923504469e2b8d0140
2023/02/21 20:04:48 Codename: alphaga
2023/02/21 20:04:48 BuildDate: 2023-02-09_14:41:04
2023/02/21 20:04:48 GoVersion: 1.19.2
2023/02/21 20:04:48 Platform: linux
2023/02/21 20:04:48 Constraint_parser: >= 1.0, <= 2.0
2023/02/21 20:04:48 Constraint_scenario: >= 1.0, < 3.0
2023/02/21 20:04:48 Constraint_api: v1
2023/02/21 20:04:48 Constraint_acquis: >= 1.0, < 2.0

OS version

# On Linux:
$ cat /etc/os-release
PRETTY_NAME="Raspbian GNU/Linux 11 (bullseye)"
NAME="Raspbian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"

$ uname -a
Linux gitea 5.15.84+ crowdsecurity/crowdsec#1613 Thu Jan 5 11:58:09 GMT 2023 armv6l GNU/Linux

Enabled collections and parsers

$ cscli hub list -o raw
LePresidente/gitea,enabled,0.2,Gitea Support : parser and brute-force detection,collections
crowdsecurity/linux,enabled,0.2,core linux support : syslog+geoip+ssh,collections
crowdsecurity/sshd,enabled,0.2,sshd support : parser and brute-force detection,collections
LePresidente/gitea-logs,enabled,0.4,Parse gitea logs,parsers
crowdsecurity/dateparse-enrich,enabled,0.2,,parsers
crowdsecurity/geoip-enrich,enabled,0.2,"Populate event with geoloc info : as, country, coords, source range.",parsers
crowdsecurity/sshd-logs,enabled,2.0,Parse openSSH logs,parsers
crowdsecurity/syslog-logs,enabled,0.8,,parsers
LePresidente/gitea-bf,enabled,0.2,Detect gitea bruteforce,scenarios
crowdsecurity/ssh-bf,enabled,0.1,Detect ssh bruteforce,scenarios
crowdsecurity/ssh-slow-bf,enabled,0.2,Detect slow ssh bruteforce,scenarios

Acquisition config

# On Linux:
$ cat /etc/crowdsec/acquis.yaml /etc/crowdsec/acquis.d/*
#Generated acquisition file - wizard.sh (service: sshd) / files : /var/log/auth.log
filenames:
  - /var/log/auth.log
labels:
  type: syslog
---
#Generated acquisition file - wizard.sh (service: linux) / files : /var/log/syslog /var/log/kern.log /var/log/messages
filenames:
  - /var/log/syslog
  - /var/log/kern.log
  - /var/log/messages
labels:
  type: syslog
---
# Hand written file
filenames:
 - /mnt/foo/gitea/log/gitea.log
labels:
  type: gitea
cat: '/etc/crowdsec/acquis.d/*': No such file or directory

Config show

$ cscli config show
Global:
   - Configuration Folder   : /etc/crowdsec
   - Data Folder            : /var/lib/crowdsec/data
   - Hub Folder             : /etc/crowdsec/hub
   - Simulation File        : /etc/crowdsec/simulation.yaml
   - Log Folder             : /var/log/
   - Log level              : info
   - Log Media              : file
Crowdsec:
  - Acquisition File        : /etc/crowdsec/acquis.yaml
  - Parsers routines        : 1
  - Acquisition Folder      : /etc/crowdsec/acquis.d
cscli:
  - Output                  : human
  - Hub Branch              :
  - Hub Folder              : /etc/crowdsec/hub
Local API Server:
  - Listen URL              : 127.0.0.1:8080
  - Profile File            : /etc/crowdsec/profiles.yaml
  - Trusted IPs:
      - 127.0.0.1
      - ::1
  - Database:
      - Type                : sqlite
      - Path                : /var/lib/crowdsec/data/crowdsec.db
      - Flush age           : 7d
      - Flush size          : 5000

Prometheus metrics

$ cscli metrics
Acquisition Metrics:
╭────────────────────────┬────────────┬──────────────┬────────────────┬────────────────────────╮
│         Source         │ Lines read │ Lines parsed │ Lines unparsed │ Lines poured to bucket │
├────────────────────────┼────────────┼──────────────┼────────────────┼────────────────────────┤
│ file:/var/log/auth.log │ 6          │ -            │ 6              │ -                      │
│ file:/var/log/syslog   │ 25         │ -            │ 25             │ -                      │
╰────────────────────────┴────────────┴──────────────┴────────────────┴────────────────────────╯

Parser Metrics:
╭─────────────────────────────────┬──────┬────────┬──────────╮
│             Parsers             │ Hits │ Parsed │ Unparsed │
├─────────────────────────────────┼──────┼────────┼──────────┤
│ child-crowdsecurity/sshd-logs   │ 20   │ -      │ 20       │
│ child-crowdsecurity/syslog-logs │ 31   │ 31     │ -        │
│ crowdsecurity/sshd-logs         │ 2    │ -      │ 2        │
│ crowdsecurity/syslog-logs       │ 31   │ 31     │ -        │
╰─────────────────────────────────┴──────┴────────┴──────────╯

Local Api Metrics:
╭──────────────────────┬────────┬──────╮
│        Route         │ Method │ Hits │
├──────────────────────┼────────┼──────┤
│ /v1/decisions/stream │ GET    │ 11   │
│ /v1/heartbeat        │ GET    │ 10   │
│ /v1/watchers/login   │ POST   │ 2    │
╰──────────────────────┴────────┴──────╯

Local Api Machines Metrics:
╭──────────────────────────────────────────────────┬───────────────┬────────┬──────╮
│                     Machine                      │     Route     │ Method │ Hits │
├──────────────────────────────────────────────────┼───────────────┼────────┼──────┤
│ 6fc549d69f0b4cfb9f14fef65c2d23d2PHFTzoaJ3bNUgHwU │ /v1/heartbeat │ GET    │ 10   │
╰──────────────────────────────────────────────────┴───────────────┴────────┴──────╯

Local Api Bouncers Metrics:
╭────────────────────────────┬──────────────────────┬────────┬──────╮
│          Bouncer           │        Route         │ Method │ Hits │
├────────────────────────────┼──────────────────────┼────────┼──────┤
│ FirewallBouncer-1676929718 │ /v1/decisions/stream │ GET    │ 11   │
╰────────────────────────────┴──────────────────────┴────────┴──────╯

Local Api Decisions:
╭───────────────────────────┬────────┬────────┬───────╮
│          Reason           │ Origin │ Action │ Count │
├───────────────────────────┼────────┼────────┼───────┤
│ crowdsecurity/ssh-bf      │ CAPI   │ ban    │ 13109 │
│ crowdsecurity/ssh-slow-bf │ CAPI   │ ban    │ 4236  │
╰───────────────────────────┴────────┴────────┴───────╯

Local Api Alerts:
╭───────────────────────┬───────╮
│        Reason         │ Count │
├───────────────────────┼───────┤
│ LePresidente/gitea-bf │ 36    │
╰───────────────────────┴───────╯

Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.

FreeBSD: Could you please elaborate how to auto start service in the install docs?

After installing the FreeBSD this message is shown. Could you please elaborate how in the docs?

Message from crowdsec-1.1.1:

--
crowdsec is installed.

You need to edit the agent config file /usr/local/etc/crowdsec/crowdsec.yaml and
enable rc via sysrc.

# sysrc crowdsec_enable="YES"

Add documentation for `cscli decisions import`

cscli decisions import is missing documentation.

In new docs console command is wrong regarding FreeBSD

In new docs in https://docs.crowdsec.net/docs/user_guides/hub_mgmt/ (and everywhere else in that section) it isn't taken into consideration that cscli is renamed to crowdsec-cli on FreeBSD

Document file acquisition exclude function

Just noted when helping a user that we have not documented how to use exclude_regexps from version >= 1.4.2

Placing this here to remind me or if anyone has the time to do it 👍🏻

CentOS: Description for CentOS 7 and 8 have been switched

In the install instructions for CentOS the instructions for CentOS 7 and 8 appear to have been switched:

DNF doesn't exist on CentOS 7 - but on CentOS 8. On CentOS 7 one should use YUM instead.

Dashboard + ARM: inform user from cscli

As the dashboard is not compatible with ARM architectures, it should be great to inform the user and blocking him directly from the cscli command, else, patient users will wait for long long time... 👴

admin@ip-XXXXXXXXXX:~$ sudo cscli dashboard setup --listen 0.0.0.0
INFO[01-07-2021 10:24:19 AM] /var/lib/crowdsec/data/metabase.db exists, skip.
INFO[01-07-2021 10:24:19 AM] Pulling docker image metabase/metabase:v0.37.0.2
............................................................................................................
INFO[01-07-2021 10:24:29 AM] creating container '/crowdsec-metabase'
INFO[01-07-2021 10:24:33 AM] waiting for metabase to be up (can take up to a minute)
...........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

rpm doc should refer to dnf

for centos8 and fc3{3,4}, documentation should refer to dnf install instead of yum as it's the now preferred way to install stuff.

Install : copy-paste a command to create crowdsec_crowdsec.list file

In https://packagecloud.io/crowdsec/crowdsec/install#bash-deb

For the /etc/apt/sources.list.d/crowdsec_crowdsec.list section

It should be great to just have to copy-paste a command, @see Docker install doc for example:

 echo \
  "deb [arch=arm64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

fix the installation instructions for the repository

now that we switch to packagecloud, we need to edit the install where it's written

echo "deb https://packagecloud.io/crowdsec/crowdsec/debian/ $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/crowdsec.list > /dev/null

and add the same for ubuntu

echo "deb https://packagecloud.io/crowdsec/crowdsec/ubuntu/ $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/crowdsec.list > /dev/null

CentOS 8 is EOL

CentOS 8 is EOL. It would make sense to replace CentOS 8 with AlmaLinux in the file install.mdx .

[enhancement] HAproxy real IP

We should include in haproxy bouncer some information on how to set the src IP before it hits the lua code to prevent users from getting the proxy IP.

// Will update with more information

crowdsec tour : show how to manually ban IPs / ranges ?

I think it would be good to have in the crowdsec tour documentation 👍

problem to understanding for average people

Hi,

it's related to https://doc.crowdsec.net/Crowdsec/v1/getting_started/concepts/ essentially but it can be a global approach about your whole approach.
The problem is that you directly use concepts like API, framework etc...
Where actually average people or lambda people are familiarized with software. Not abstract concept. They are used to their usage and their command line usage if they have installed a few ubuntu servers for example for their personal use.

Its a reproach that I would like have done to many other tech companies, so don't worry it's a common thing among devs etc. But if you ever teached among students even older students that would maybe be used at some level to manage the servers at work etc and who would learn computer science in a continuing training (formation continue).... and as your tech corp is a french initiative, I know you will be sensitive to what I'm going to say... they are used to what they are used to write in their command line ssh. Not to an abstract approach who would require that you actually are quite knowladgeable in the field already.

So what would be nice is that there is like an overview-bis or something alike where average people can rely on.
What does know an average people ? or What is he familiar with?
iptables. That they know. And if they are a little bit advanced they would even know framework like ufw, firewalld or others that actually all rely on iptables. And if they are even more advanced they will know fail2ban.
We need to know at which level are we or how far are we from iptables.
So is it related to iptables? or is your "bouncer package" replacing iptables?

I would have thought for example that your package would download a blacklsit and then, as fail2ban iptables would integrate the whole list in its list.
But apparently not.
So do you see where the problem is here?

acquisition_dir behavior

It is unclear what happens when both configuration_path and configuration_dir are specified: if both are used, which one takes precedence, if any merging is done (don't think so)

https://github.com/crowdsecurity/crowdsec-docs/blob/main/crowdsec-docs/docs/configuration/crowdsec_configuration.md#acquisition_path

acquisition_path

Path to the yaml file containing logs that needs to be read.

acquisition_dir

(>1.0.7) Path to a directory where each yaml is considered as a acquisition configuration file containing logs that needs to be read.

Document minimal resources for crowdsec/metabase

The documentation should state the minimal needed resources for crowdsec itself and the other components :

• metabase
• a typical bouncer

update doc for raspberry pi os

update doc for raspberry pi os which is outdated as of now raspberry pi os is 64bits, and is basically debian.

notification plugins : specify if it's at agent level or LAPI level

This is not indicated in the documentation but it should : notification plugins are at LAPI level

raspbian : be more explicit

Either we should publish some packages, or make it explicit in the doc that we don't support it natively

doc fixes

hub management

• parser => reference (bad link)

• enricher => parsers link (bad link)

• decisions management => link vers cscli decisions pour le command usage

• manual installation => Build docker image titre pas formaté

• crowdsec_configuration => max_age pas formatté

• cloudwatch => no source directives

• datasources/monitoring => lien vers prometheus plutot que cscli ?

• scenarios / introduction => lien vers leaky bucket cassé

• simulation: preciser si niveau agent ou niveau API

• profiles: preciser si cest niveau lapi ou agent

• observability/intro => lien de cscli vers cscli metrics

• observability/dashboard => mettre un lien de cscli_dashboard command
  use triple` bash for bash command
  and don't put the $ in front

• localapi/intro => ### Server pas formatté , ### configuration pas formatté

• local api / intro => !!!tips pas formatté

• central APi / intro => point a la fin de phrase (scenario list)

• get statistics and insights on your alerts compared

• bouncer/intro => changer lien du hub par lien des bouncers dans le hub

• in v1.1 => Bouncers => rename page Bouncers par Contributing

lua-cs-bouncer/nginx/openresty : delete doc about "LOG_FILE" which is not available anymore

Database : CREATE USER 'crowdsec'@'%'

https://docs.crowdsec.net/docs/local_api/database#mysql-and-mariadb

On the databases page it's advises to mysql> CREATE USER 'crowdsec'@'%' IDENTIFIED BY '<password>';

MySQLTuner advises this is insecure

Restrict Host for 'crowdsec'@'%' to 'crowdsec'@LimitedIPRangeOrLocalhost
RENAME USER 'crowdsec'@'%' TO 'crowdsec'@LimitedIPRangeOrLocalhost;

Document usage of debconf

The debian package supports debconf, but we do not document how to take advantage of it anywhere.

OpenWRT docs: Link to 3rd party

Hi

In terms of providing documentation for CrowdSec on 3. party platforms it's probably a good idea to link to well-maintained documentation rather than trying to keep our own maintained. Therefore I want to suggest that in the case of OpenWRT we link to https://openwrt.org/docs/guide-user/services/crowdsec from our documentation.

Improve documentation on go templates for notification

While the doc gives some examples, it can be improved:

• Give the user more pointers on how to write go-template
• Make more explicit that we use sprig and they have access to more functions in the template
• Add something to help the user to simulate notifications without needing to actually trigger a scenario.

notification plugins : add an example event

for people dealing with go template, having a sample event at hand reach and pointers to online validation tools (such as https://camlittle.com/go-template-validation / https://github.com/apexskier/go-template-validation) might be useful !

document how to use cscli on macos

packaging doc

• Add the sudo apt-get update before sudo apt-get install crowdsec when installing crowdsec from repository

• On fresh buster , gnupg is not installed (please add the package to install in the documentation)