robcowart/synesis_lite_syslog

Syslog collection with the Elastic Stack.

sýnesis™ Lite for Syslog

sýnesis™ Lite for Syslog provides basic log analytics for syslog messages using the Elastic Stack.

Getting Started

sýnesis™ Lite for Syslog is built using the Elastic Stack, including Elasticsearch, Logstash and Kibana. Please refer to INSTALL.md for instructions on how to install and configure sýnesis™ Lite for Syslog.

If you are new to the Elastic Stack, this video goes beyond a simple default installation of Elasticsearch and Kibana. It discusses real-world best practices for hardware sizing and configuration, providing production-level performance and reliability.

Additionally local SSD storage should be considered as mandatory! For an in-depth look at how different storage options compare, and in particular how bad HDD-based storage is for Elasticsearch (even in multi-drive RAID0 configurations) you should watch this video...

Dashboards

The following dashboards are provided.

Overview

The Overview dashboard provides a summary of received Syslog messages by severity, node, process and facility.

Top-N

Log Browser

The Log Browser dashboard allows for easy browsing of the raw Syslog messages that have been received.

synesis_lite_syslog's Issues

Add docker support

@robcowart Hi. Thanks for the work put in here. Please consider adding docker support.

grokparserfail on CISCO syslog

hello

im facing a little problem now in syslog solution
the parser cant parse the cisco syslog format
(gns3 is creating the syslog... router 7200)
any help would be appreciated
thanks...

Removing event.message but log.message cannot be searched

Hello,

Since event.message and log.message are pretty much a duplicate of the logs, I decided to drop event.message since we have a cleaner log.message - this allows to save space as some logs are quite long.

But when I search using "query strings" in Kibana, it doesn't search log.message at all. It does search event.message when it's there as well as other fields such as log.process.

I don't know why Kibana refuses to search log.message when using "query strings" (just typing a word or sentence with double-quotes in the KQL box), can you help?

Thanks!

LS not ingesting syslog

Hi Rob,

I've installed this on a docker container, this is the current docker-compose.yaml:

version: '2'
services:
  elasticsearch:
    image: elasticsearch-img:6.3.2
    container_name: elasticsearch-container
    volumes:
      - /data/elasticsearch-1/:/usr/share/elasticsearch/data
    ports: 
      - 9200:9200 #Elasticsearch HTTP
      - 9300:9300 #Elasticsearch TCP transport
    network_mode: bridge
    restart: always
    environment:
      # - cluster.name=docker-cluster
      # - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms4g -Xmx4g"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    
  logstash:
    image: logstash-img:6.3.2
    container_name: logstash-container
    ports:
      - 5000:5000 #logstash TCP input
      - 514:5140  #listent to syslog on 514(host) map it to 5140(container) 541 is reserved and needs root
      - 514:5140/udp  #listent to syslog on 514(host) map it to 5140(container) 541 is reserved and needs root
    restart: always #restarts on reboot
    environment:
      - "LS_JAVA_OPTS=-Xms8g -Xmx8g"
      - "SYNLITE_SYSLOG_TEMPLATE_PATH=/usr/share/logstash/syslog/templates"
      - "SYNLITE_SYSLOG_GROK_PATTERNS_DIR=/usr/share/logstash/syslog/patterns"
      - "SYNLITE_SYSLOG_RESOLVE_IP2HOST=true"
      - "SYNLITE_SYSLOG_NAMESERVER=8.8.8.8"
      - "SYNLITE_SYSLOG_ES_HOSTS=elasticsearch:9200"
      # - "SYNLITE_SYSLOG_ES_USER=elastic"
      # - "SYNLITE_SYSLOG_ES_PASSWORD=changeme"
      - "SYNLITE_SYSLOG_TCP_HOST=0.0.0.0"
      - "SYNLITE_SYSLOG_TCP_PORT=514"
      - "SYNLITE_SYSLOG_UDP_HOST=0.0.0.0"
      - "SYNLITE_SYSLOG_UDP_PORT=514"
      - "SYNLITE_SYSLOG_MSG_TIMESTAMP=true"
      - "SYNLITE_SYSLOG_TZ=UTC"
    network_mode: bridge 
    links:
    - elasticsearch
    depends_on:
    - elasticsearch

Initially, port mapping of 514:514 made docker crib stating that permission was denied. I'm guessing this is because it's a port < 1000 and hence is previlaged. I've mapped 514:5140 within the container.

My /etc/rsyslog.conf looks like below:

...
...
# provides UDP syslog reception
module(load="imudp")
input(type="imudp" port="514")

# provides TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="514")
...
...

I'm able to see syslog being written on to /var/log/syslog. It works when I do something like: logger -s " This is a test "

However, I do not see anyting being picked up by LS/ES. What am I missing?

Thanks

ES auth variable not working

It seems that ES username/password defined in the systemd file don't take effect and logstash still tries to use the default "changeme". The workaround is to specify the ES password in the output conf file.

updating package

Hello.
We use elastiflow 4 with elasticsearch 7.8.1
Will you plan to update synesis_lite_syslog to work with 7.8.1 version .
It allows to use one elasticsearch cluster to collect netflow and syslog data

Listener doesn't start

Hi Rob, followed the instructions but the 514 port listener fails to start for some reason -

[2019-05-15T12:05:05,250][INFO ][logstash.inputs.tcp ] Starting tcp input listener {:address=>"0.0.0.0:514", :ssl_enable=>"false"} [2019-05-15T12:05:05,256][ERROR][logstash.javapipeline ] A plugin had an unrecoverable error. Will restart this plugin. Pipeline_id:synesis_lite_syslog Plugin: <LogStash::Inputs::Tcp host=>"0.0.0.0", dns_reverse_lookup_enabled=>false, id=>"dbb44d7e7b498a36341cfa76571b5c366d668ed2b496086bcc69ec34add2adb2", type=>"syslog", port=>514, enable_metric=>true, codec=><LogStash::Codecs::Line id=>"line_ed18c78c-5642-4b13-9fc6-67f140d700c5", enable_metric=>true, charset=>"UTF-8", delimiter=>"\n">, mode=>"server", proxy_protocol=>false, ssl_enable=>false, ssl_verify=>true, ssl_key_passphrase=><password>, tcp_keep_alive=>false> Error: Permission denied Exception: Java::JavaNet::SocketException Stack: sun.nio.ch.Net.bind0(Native Method) sun.nio.ch.Net.bind(sun/nio/ch/Net.java:461) sun.nio.ch.Net.bind(sun/nio/ch/Net.java:453) sun.nio.ch.ServerSocketChannelImpl.bind(sun/nio/ch/ServerSocketChannelImpl.java:227) io.netty.channel.socket.nio.NioServerSocketChannel.doBind(io/netty/channel/socket/nio/NioServerSocketChannel.java:128) io.netty.channel.AbstractChannel$AbstractUnsafe.bind(io/netty/channel/AbstractChannel.java:558) io.netty.channel.DefaultChannelPipeline$HeadContext.bind(io/netty/channel/DefaultChannelPipeline.java:1283) io.netty.channel.AbstractChannelHandlerContext.invokeBind(io/netty/channel/AbstractChannelHandlerContext.java:501) io.netty.channel.AbstractChannelHandlerContext.bind(io/netty/channel/AbstractChannelHandlerContext.java:486) io.netty.channel.DefaultChannelPipeline.bind(io/netty/channel/DefaultChannelPipeline.java:989) io.netty.channel.AbstractChannel.bind(io/netty/channel/AbstractChannel.java:254) io.netty.bootstrap.AbstractBootstrap$2.run(io/netty/bootstrap/AbstractBootstrap.java:364) io.netty.util.concurrent.AbstractEventExecutor.safeExecute(io/netty/util/concurrent/AbstractEventExecutor.java:163) io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(io/netty/util/concurrent/SingleThreadEventExecutor.java:403) io.netty.channel.nio.NioEventLoop.run(io/netty/channel/nio/NioEventLoop.java:463) io.netty.util.concurrent.SingleThreadEventExecutor$5.run(io/netty/util/concurrent/SingleThreadEventExecutor.java:858) io.netty.util.concurrent.FastThreadLocalRunnable.run(io/netty/util/concurrent/FastThreadLocalRunnable.java:30) java.lang.Thread.run(java/lang/Thread.java:834)