ekanite/ekanite

The Syslog server with built-in search

For detailed look at the goals, design, and implementation of this project, check out these blog posts.

Ekanite

Ekanite is a high-performance syslog server with built-in text search. Its goal is to do a couple of things, and do them well -- accept log messages over the network, and make it easy to search the messages. What it lacks in feature, it makes up for in focus. Built in Go, it has no external dependencies, which makes deployment easy.

Features include:

• Supports reception of log messages over UDP, TCP, and TCP with TLS.
• Full text search of all received log messages.
• Full parsing of RFC5424 headers.
• Log messages are indexed by parsed timestamp, if one is available. This means search results are presented in the order the messages occurred, not in the order they were received, ensuring sensible display even with delayed senders.
• Automatic data-retention management. Ekanite deletes indexed log data older than a configurable time period.
• Not a JVM in sight.

Search is implemented using the bleve search library. For some performance analysis of bleve, and of the sharding techniques used by Ekanite, check out this post.

Getting started

The quickest way to get running on OSX and Linux is to download a pre-built release binary. You can find these binaries on the Github releases page. Once installed, you can start Ekanite like so:

ekanited -datadir ~/ekanite_data # Or any directory of your choice.

To see all Ekanite options pass -h on the command line.

If you want to build Ekanite, either because you want the latest code or a pre-built binary for platform is not available, take a look at CONTRIBUTING.md.

Sending logs to Ekanite

For now, for Ekanite to accept logs, your syslog client must be configured such that the log lines are RFC5424 compliant, and in the following format:

<PRI>VERSION TIMESTAMP HOSTNAME APP-NAME PROC-ID MSGID MSG"

Consult the RFC to learn what each of these fields is. The TIMESTAMP field must be in RFC3339 format. Both rsyslog and syslog-ng support templating, which make it very easy for those programs to format logs correctly and transmit the logs to Ekanite. Templates and installation instructions for both systems are below.

rsyslog

# Send messages to Ekanite over TCP using the template. Assumes Ekanite is listening on 127.0.0.1:5514
$template Ekanite,"<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% - %msg%\n"
*.*             @@127.0.0.1:5514;Ekanite

Add this template to /etc/rsyslog.d/23-ekanite.conf and then restart rsyslog using the command sudo service rsyslog restart.

syslog-ng

source s_ekanite {
	system();	# Check which OS & collect system logs
	internal();	# Collect syslog-ng logs
};
template Ekanite { template("<${PRI}>1 ${ISODATE} ${HOST} ${PROGRAM} ${PID} - $MSG\n"); template_escape(no) };
destination d_ekanite {
	tcp("127.0.0.1" port(5514) template(Ekanite));
};

log {
	source(s_ekanite);
	destination(d_ekanite);
};

Add this template to /etc/syslog-ng/syslog-ng.conf and then restart syslog-ng using the command /etc/init.d/syslog-ng restart.

With these changes in place rsyslog or syslog-ng will continue to send logs to any existing destination, and also forward the logs to Ekanite.

Searching the logs

Search support is pretty simple at the moment. You have two options -- a simple telnet-like interface, and a browser-based query interface.

Telnet interface

Telnet to the query server (see the command line options) and enter a search term. The query language supported is the simple language supported by bleve, but a more sophisiticated query syntax, including searching for specific field values, may be supported soon.

For example, below is an example search session, showing accesses to the login URL of a Wordpress site. The telnet clients connects to the query server and enters the string login

$ telnet 127.0.0.1 9950
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
login
<134>0 2015-05-05T23:50:17.025568+00:00 fisher apache-access - - 65.98.59.154 - - [05/May/2015:23:50:12 +0000] "GET /wp-login.php HTTP/1.0" 200 206 "-" "-"
<134>0 2015-05-06T01:24:41.232890+00:00 fisher apache-access - - 104.140.83.221 - - [06/May/2015:01:24:40 +0000] "GET /wp-login.php?action=register HTTP/1.0" 200 206 "http://www.philipotoole.com/" "Opera/9.80 (Windows NT 6.2; Win64; x64) Presto/2.12.388 Version/12.17"
<134>0 2015-05-06T01:24:41.232895+00:00 fisher apache-access - - 104.140.83.221 - - [06/May/2015:01:24:40 +0000] "GET /wp-login.php?action=register HTTP/1.1" 200 243 "http://www.philipotoole.com/wp-login.php?action=register" "Opera/9.80 (Windows NT 6.2; Win64; x64) Presto/2.12.388 Version/12.17"
<134>0 2015-05-06T02:47:54.612953+00:00 fisher apache-access - - 184.68.20.22 - - [06/May/2015:02:47:51 +0000] "GET /wp-login.php HTTP/1.1" 200 243 "-" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/24.0.1309.0 Safari/537.17"
<134>0 2015-05-06T04:20:49.008609+00:00 fisher apache-access - - 193.104.41.186 - - [06/May/2015:04:20:46 +0000] "POST /wp-login.php HTTP/1.1" 200 206 "-" "Opera 10.00"

Perhaps you only want to search for POST accesses to that URL:

login -GET
<134>0 2015-05-06T04:20:49.008609+00:00 fisher apache-access - - 193.104.41.186 - - [06/May/2015:04:20:46 +0000] "POST /wp-login.php HTTP/1.1" 200 206 "-" "Opera 10.00"

A more sophisticated client program is planned.

Browser interface

The browser-based interface also accepts bleve-style queries, identical to those described in the Telnet section. By default the browser interface is available at http://localhost:8080. An example session is shown below.

Diagnostics

Basic statistics and diagnostics are available. Visit http://localhost:9951/debug/vars to retrieve this information. The host and port can be changed via the -diag command-line option.

Building New Parsers

The architecture now supports the easy implementation of new parsers beyond the stock syslog in 3 easy steps:

1. In input/parser.go expand supportedFormats() to capture the additional standard and name.
2. In parser/, create the new input format parser using appropriate regex statements.
   • Ensure that the new parser includes a timestamp field compatible with RFC3339, e.g. 2006-01-02T15:04:05Z07:00
3. Back in input/parser.go, update NewParser() to properly instantiate the new input format parser.

Project Status

The project is not actively maintained, though development may re-occur in the future.

ekanite's Issues

Wire up Lucene-type query language support

This will required a dedicate parser, which will generate an AST.

Create command-line client

Ekanite needs a proper command line client, telnet won't cut it for extended use. This client should support the following:

• Paging of results.
• Return errors if necessary.
• Show stats and diagnostics, about system as well as indexed data.

A more sophisticated client may, in time, require a richer marshalled response from the Ekanite daemon.

steps to diagnose lack of content in logs

I've just deployed an ekanite instance, and am not seeing any log data in the system. I'm not sure whether the issue is the configuration of my client, its connectivity to ekanite, or ekanite's ability to parse the log entries being sent to it.

I've verified that the host/port for ekanite is reachable from the client system, and enabled the -diag endpoint, but without documentation showing what the diagnostic JSON should look like it's hard to know what to look for

Here's what the initial chunk of the diagnostic output looks like:

{
  "bleve": {
    "bootDuration": 12030,
    "indexes": {}
  },
  "cmdline": [
    "/usr/local/bin/ekanited",
    "-datadir",
    "/data/index",
    "-tcp",
    ":5514",
    "-queryhttp",
    ":8080",
    "-retention",
    "168h",
    "-diag",
    ":9951"
  ],
  "ekanite": {},
  "engine": {
    "queriesRx": 5
  },
  "input": {},
  "memstats": {
    "Alloc": 1269032,
    "TotalAlloc": 1269032,
    "Sys": 4165632,
    "Lookups": 38,
    "Mallocs": 8128,
    "Frees": 417,
    "HeapAlloc": 1269032,
    "HeapSys": 2686976,
    "HeapIdle": 737280,
    "HeapInuse": 1949696,
    "HeapReleased": 0,
    "HeapObjects": 7711,
    "StackInuse": 458752,
    "StackSys": 458752,
    "MSpanInuse": 25080,
    "MSpanSys": 32768,
    "MCacheInuse": 3472,
    "MCacheSys": 16384,
    "BuckHashSys": 3055,
    "GCSys": 169984,
    "OtherSys": 797713,
    "NextGC": 4473924,
    "LastGC": 0,
    "PauseTotalNs": 0,
    "PauseNs": [
      0,
      0,
      0,

After than its a ton of 0 lines for PauseNs, and then PauseEnd with a ton of 0s, and then a bunch of memory information objects

Is "input": {}, significant? The key thing I'm trying to determine at this point is whether ekanite is receiving any messages, and then how many of those it's successfully parsing. Is the empty input section telling me it hasn't received anything? Would it look any different if ekanite was receiving messages but failing to parse them? If ekanite was receiving messages but failing to parse them, would I be able to get a look at what the messages it's receiving look like anywhere and/or what the parse error is?

Tokenise digits?

Two log messages, 1 contains "sshd", the other contains "ssh2". Other search systems pick up the latter with a search for "ssh". Ekanite picks up neither.

10x performance boost after replacing SyslogDelimiter with bufio#Reader.ReadString

I am not sure why SyslogDelimiter is used in input/collector.go. But when I replace it with ReadString, the overall log indexing throughput increase by ~10 times.

A rough code review shows s.regex.FindIndex is called for every byte we received. If it is assumed that the log messages are always separated by \n, the offset scan used in ReadString is enough and efficient.

Provide client examples

I'm trying to log to ekanite with logrus's syslog hook, but I can't get it working so far. Some examples for logging clients/libs would be nice.

Shared parsers results in invalid parsing

Issue that would be masked by #73

Error during execution - no tokenizer type 'regexp' registered

Something has changed in the bleve source, and the right tokenizers block index creation. I've sent a note to the bleve list.

Support automatic parsing of Apache Common Format

The system should detect when the body of a message is Apache Common Format, parse the body, and index the fields.

Support SSL transport for log messages

syslog-ng example config

Your syslog-ng example configuration is invalid. I don't know the correct configuration myself, just that it isn't correct and/or valid.

Replace regex parsing with state machine type parsing

Regex is heavyweight and consumes too much CPU during line parsing. A dedicated state-machine type parser would be much better.

Index source of parsed lines is actually entire line

The syslog header should be stripped, stored in a separate filed, but it should not be presented directly in search results. It should be accessible some other way.

Unparsed lines should be indexed as the source

If a line cannot be parsed at all, the entire line should be indexed as the message.

S3 backup and restore

Ekanite should provide backup and restore functionality so that the logging server is not a single point of failure. Broadly speaking, the equivalent to logstash-output-s3 and logstash-input-s3 where logs are shipped to s3 periodically and potentially restored during startup.

Add support for regex search queries

Try out performance with LevelDB (and goleveldb)

Some experimentation with the storage backend would be worthwhile.

Investigate stemming for better search results

Started with some issues raised here:

#18

From:

https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/bleve/9-PZ1c1E4ro/WEd3a-SfBgAJ

Getting stemming work may address these issues.

Allow search of fields

The query language needs to support syntax like:

apache.statuscode: 404

Windows support

Just confirming that this server does not support the Windows platform, correct?

Add "live tail" feature

The CLI could be put in this mode, allowing a specific refreshing search (perhaps).

ekanite cause a high cpu usage, any fix or how to debug?

top - 11:07:44 up 43 days, 17:30,  2 users,  load average: 1.06, 1.03, 1.05
Tasks: 127 total,   4 running, 123 sleeping,   0 stopped,   0 zombie
%Cpu0  : 60.4 us, 38.9 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.3 si,  0.3 st
KiB Mem:   1016332 total,   895712 used,   120620 free,    85044 buffers
KiB Swap:        0 total,        0 used,        0 free.   237724 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND                                                                                                                     
28941 root      20   0 1554440 115492  98036 R 99.0 11.4   8490:51 ekanited                                                                                                                    

thx.

Support time-ranged queries

Right now ekanite searches all indexes for log messages. The system should support only searching a specific time range, which may speed up searching significantly, depending on the data set.

feature of this package

i want to use something like this for log collector i want to use as store something like leveldb or badger...
Do you plan to invest time for this package or it not maintained anymore?

Parsing of "day" retention periods not supported

This is because Go does not not support parsing strings such as "7d".

Hour-long indexes may be excessive

It would probably be better to have day-long indexes.

example of client feeding logs into server

just to get going to try this out.

And also example of queries to run on the web interface.

is there a way to achieve the same as tail -f /var/log/syslog? with some search term?

is there a way to achieve the same as tail -f /var/log/syslog? with some specific bleve search term? thanks

Corrupted searching when shard number > 10

Hi,

I'm using goleveldb backend, setting shards = 16.

The problem can be reproduced by these steps:

1. start ekanite
2. send some logs to it.
3. search looks fine.
4. restart ekanite
5. search again with the same query. It shows a lots of empty items.

After digging into the code, some suspects are:

• ekanite does not call engine.Close() when quitting, so the underlining KVStore is not properly closed.
• When ekanite restart from a non-empty data dir, It loads the shards in alphabetical order, and add them into IndexAlias in the same order. It may be the cause when shard number > 10, so it loads actually 0, 1, 10, 11, ...

A quit test by setting shards = 8, the problem gone.

Confused on how to get ekanited built

Your readme instructions are incorrect

here's what i get when i follow them on my mac
alexs-mbp-2:ekanite alexnewman$ find . -type d -depth 3
./pkg/darwin_amd64/github.com
./src/github.com/blevesearch
./src/github.com/boltdb
./src/github.com/ekanite
./src/github.com/golang
./src/github.com/steveyen
./src/github.com/willf
alexs-mbp-2:ekanite alexnewman$ pwd
/Users/alexnewman/ekanite
alexs-mbp-2:ekanite alexnewman$

searching

I coded a simple ruby program to send logs to ekanited, see: https://gist.github.com/cleesmith/1d87a436023f746549ec
... for testing without rsyslog or syslog-ng
... maybe useful for others and could be altered to read the logs13k.txt file you provide

Some of the following may be issues for bleve, but I noticed:

• searches only work for exact matches: ssh won't match sshd
• no wildcard ssh* searches
• no word stemming: listen, listened, listening or similar
• are time range searches possible?
• is it possible to list the most recent logs, say, in descending order by timestamp
  ... i.e. a kind of near real time view of what's happening
• instead of telnet localhost 9950 doing a http://localhost:9950/ in the browser lists all logs
• running 2 simultaneous ruby programs sending syslogs takes twice as long: from 20 seconds for one to 50 seconds each for two ... i.e. sometimes logs are sent from multiple sources to a central server, or perhaps ekanite is only intended to receive from a single rsyslog/syslog-ng instance
• is there docs or more details about the folder that is created: 20151018_0000 which seems to use 1.2M of space for 40 syslogs ?
• 84,440 logs uses 1G <-- du -h 20151018_0000 ... this just seems like a lot, but I'm not sure ?

"ekanite": {"launch": 2015-10-18 15:22:40.011642743 +0000 UTC},
"engine": {"batchIndexed": 285, "batchTimeout": 5, "docsIDsRetrived": 1467, "eventsIndexed": 84440, "queriesRx": 60},
"input": {"tcpBytesRead": 9404505, "tcpConnReadEOF": 1674263446, "tcpConnReadError": 1674263449, "tcpConnections": 5, "tcpEventsRx": 84440, "unparsed": 84440},

I work on a lot of projects involving syslogs and intrusion detection (events from Snort/Suricata) so I find ekanite interesting. My future view is to replace elasticsearch with something similar but coded in Go for simpler deployments.
Thanks for your efforts.

End-to-end tests failed when DefaultNumShards = 1

Need to see what is going on here.

Increase BoltDB fill percentage

Since log data is mostly append only, increasing the BoltDB fill percentage may increase storage efficiency.