loggly-pipe
- Eats lines from stdin.
- Sends batches to Loggly (or compatible).
- Poops unaltered lines to stdout.
- Poops
mark
JSON to stdout.
Installation
loggly-pipe
is built to be executed as a standalone file, so installing
directly from the raw github URL works great:
curl -L -o loggly-pipe https://raw.github.com/modcloth-labs/loggly-pipe/master/loggly_pipe.py
chmod +x loggly-pipe
You can also grab it via pip
, which will install a loggly-pipe
executable:
pip install loggly_pipe
Configuration
Configuration may be given via the command line or the env. The latest command line options are available like one might expect:
loggly-pipe --help
Here's a list of the environment variables that are used as the option defaults:
LOGGLY_TOKEN
REQUIRED one of the tokens from your Loggly accountLOGGLY_SERVER
the base server URI (defaulthttps://logs-01.loggly.com
)LOGGLY_SHIPPER_COUNT
number of shipper threads (default1
)LOGGLY_TAGS
comma-delimited tags to apply to the shipped records. This is passed directly as the last member ofPATH_INFO
, so it should be URL-encoded if necessary. (defaultpython
)LOGGLY_BATCH_SIZE
number of records to accumulate before shipping over HTTP (default100
)LOGGLY_MAX_LINES
exit after consuming this many lines (default0
, which disables)LOGGLY_STDIN_SLEEP_INTERVAL
interval in seconds to sleep when line read from stdin is empty (default0.1
)LOGGLY_FLUSH_INTERVAL
interval in seconds at which to flush the line buffer so that low activity doesn't cause records to back up (default10.0
)LOGGLY_SHIP_ATTEMPTS
number of times to retry each batch over HTTP (default1
)LOGGLY_ON_ERROR
behavior when exceptions are caught, which is usually around shipping over HTTP, and will only take effect onceLOGGLY_SHIP_ATTEMPTS
has been reached (choices areraise
,ignore
; defaultraise
)LOGGLY_DEBUG
print some debuggy stuff if non-empty
Usage
Given a stdin stream made of line-based JSON records, loggly-pipe
will
emit each record unaltered as well as a record with a single _mark
key
for each batch.
Here's an example of a node app that uses bunyan for logging as well as
the bunyan
command line tool for human readability:
node server.js | loggly-pipe | bunyan
A trivial example that's used in the black-box test of loggly-pipe
itself can be found in the examples dir.