michal-lyson/kinesalite

An implementation of Amazon's Kinesis built on LevelDB

Kinesalite

An implementation of Amazon's Kinesis, focussed¹ on correctness and performance, and built on LevelDB (well, @rvagg's awesome LevelUP to be precise).

The Kinesis equivalent of dynalite.

To read and write from Kinesis streams in Node.js, consider using the kinesis module.

NOTE: Starting with v1.0, SSL/HTTPS is no longer the default. Use --ssl to enable pre-v1.0 behaviour.

Example

$ kinesalite --help

Usage: kinesalite [--port <port>] [--path <path>] [--ssl] [options]

A Kinesis http server, optionally backed by LevelDB

Options:
--help                 Display this help message and exit
--port <port>          The port to listen on (default: 4567)
--path <path>          The path to use for the LevelDB store (in-memory by default)
--ssl                  Enable SSL for the web server (default: false)
--createStreamMs <ms>  Amount of time streams stay in CREATING state (default: 500)
--deleteStreamMs <ms>  Amount of time streams stay in DELETING state (default: 500)
--updateStreamMs <ms>  Amount of time streams stay in UPDATING state (default: 500)
--shardLimit <limit>   Shard limit for error reporting (default: 10)

Report bugs at github.com/mhart/kinesalite/issues

Or programmatically:

// Returns a standard Node.js HTTP server
var kinesalite = require('kinesalite'),
    kinesaliteServer = kinesalite({path: './mydb', createStreamMs: 50})

// Listen on port 4567
kinesaliteServer.listen(4567, function(err) {
  if (err) throw err
  console.log('Kinesalite started on port 4567')
})

Once running, here's how you use the AWS SDK to connect (after configuring the SDK):

var AWS = require('aws-sdk')

var kinesis = new AWS.Kinesis({endpoint: 'http://localhost:4567'})

kinesis.listStreams(console.log.bind(console))

Or with the kinesis module (currently only works in https mode, when kinesalite is started with --ssl):

var kinesis = require('kinesis')

kinesis.listStreams({host: 'localhost', port: 4567}, console.log)

CBOR protocol issues with the Java SDK

The Java AWS SDK recently changed their Kinesis client to default to the CBOR protocol, which kinesalite doesn't support – you may see an error like this:

com.amazonaws.AmazonServiceException: Unable to parse HTTP response content (Service: AmazonKinesis; Status Code: 404; Error Code: null;

You can set the AWS_CBOR_DISABLE environment variable to disable this (any value should work, eg true or 1) before invoking any of the Kinesis calls in the Java SDK.

Installation

With npm do:

$ npm install -g kinesalite

Footnotes

¹Hi! You're probably American (and not a New Yorker editor) if you're worried about this spelling. No worries – and no need to open a pull request – we have different spellings in the rest of the English speaking world 🐨