iq-scm/log-courier

The Log Courier Suite is a set of lightweight tools created to ship and process log files speedily and securely, with low resource usage, to Elasticsearch or Logstash instances.

Log Courier Suite

The Log Courier Suite is a set of lightweight tools created to ship and process log files speedily and securely, with low resource usage, to Elasticsearch or Logstash instances.

• Log Courier Suite
  • Log Courier
  • Log Carver
  • Philosophy
  • Documentation
    • Installation
    • Reference
  • Upgrading from 1.x to 2.x

Log Courier

Log Courier is a lightweight shipper. It reads from log files and transmits events over the Courier protocol to a remote Logstash or Log Carver instance.

• Reads from files or the program input, following log file rotations and movements
• Compliments log events with additional fields
• Live configuration reload
• Transmits securely using TLS with server and client verification
• Codecs for client-side preprocessing of multiline events and filtering of unwanted events
• Native JSON reader to support JSON files, even those with no line-termination that makes line-based reading problematic
• Remote Administration Utility to inspect monitored log files and connections in real time.
• Compatible with all supported versions of Logstash. At the time of writing this is >= 7.7.x.

Log Carver

Log Carver is a lightweight event processor and alternative to Logstash. It receives events over the Courier protocol and performs actions against them to manipulate them into the required format for storage within Elasticsearch, or further processing in Logstash. Connected clients do not receive acknowledgements until the events are acknowledged by the endpoint, whether that be Elasticsearch or another more centralised Log Carver, providing end-to-end guarantee.

• Receives events securely using TLS with client verification
• Supports If/ElseIf/Else conditionals to process different events in different ways
• Provides several powerful actions for date processing, grokking, or simply computing a new field
• Support for complex expressions when setting fields or performing conditionals
• Transmit events for storage using the elasticsearch transport immediately after processing
• Remote Administration Utility to inspect connections in real time.

Philosophy

• Keep resource usage low and predictable at all times
• Be efficient, reliable and scalable
• At-least-once delivery of events, a crash should never lose events
• Offer secure transports
• Be easy to use

Documentation

Installation

• Public Repositories
• Building from Source

Reference

• Log Courier Configuration

• Log Carver Configuration

• Administration Utility

• Change Log

• Command Line Arguments

• Event Format

• Logstash Integration

• Protocol Specification

• SSL Certificate Utility

Upgrading from 1.x to 2.x

There are many breaking changes in the configuration between 1.x and 2.x. Please check carefully the list of breaking changes here: Change Log.

Packages also now default to using a log-courier user. If you require the old behaviour of root, please be sure to modify the /etc/sysconfig/log-courier (CentOS/RedHat) or /etc/default/log-courier (Ubuntu) file.