- Source: https://github.com/bitly/nsq
- Issues: https://github.com/bitly/nsq/issues
- Mailing List: [email protected]
- Twitter: @imsnakes or @jehiah
NSQ is a realtime message processing system designed to operate at bitly's scale, handling billions of messages per day.
It promotes distributed and decentralized topologies without single points of failure, enabling fault tolerance and high availability coupled with a reliable message delivery guarantee.
Operationally, NSQ is easy to configure and deploy (all parameters are specified on the command line and compiled binaries have no runtime dependencies). For maximum flexibility, it is agnostic to data format (messages can be JSON, MsgPack, Protocol Buffers, or anything else). Official Go and Python libraries are available out of the box and, if you're interested in building your own client, there's a protocol spec (see client libraries).
The latest stable release is 0.2.19 (ChangeLog). We publish binary releases for linux and darwin.
NOTE: master is our development branch and may not be stable at all times.
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
![]() |
NSQ was built as a successor to simplequeue (part of simplehttp) and as such was designed to (in no particular order):
- provide easy topology solutions that enable high-availability and eliminate SPOFs
- address the need for stronger message delivery guarantees
- bound the memory footprint of a single process (by persisting some messages to disk)
- greatly simplify configuration requirements for producers and consumers
- provide a straightforward upgrade path
- improve efficiency
If you're interested in more of the design, history, and evolution please read our design doc or blog post.
- no SPOF, designed for distributed environments
- messages are guaranteed to be delivered at least once
- low-latency push based message delivery (performance)
- combination load-balanced and multicast style message routing
- configurable high-water mark after which messages are transparently kept on disk
- few dependencies, easy to deploy, and sane, bounded, default configuration
- runtime discovery service for consumers to find producers (nsqlookupd)
- HTTP interface for stats, administrative actions, and producers (no client libraries needed!)
- memcached-like TCP protocol for producers/consumers
- integrates with statsd for realtime metrics instrumentation
- robust cluster administration interface with graphite charts (nsqadmin)
- nsq Go (official)
- pynsq Python (official) pypi
- libnsq C
- nsq-java Java
- TrendrrNSQClient Java
- nsq-client Node.js npm
- nodensq Node.js npm
- nsqphp PHP
- ruby_nsq Ruby rubygems
NSQ is composed of the following individual components, each with their own README:
- nsqd is the daemon that receives, buffers, and delivers messages to clients.
- nsqlookupd is the daemon that manages topology information
- nsqadmin is the web UI to view message statistics and perform administrative tasks
- nsq is a go package for writing
nsqd
clients
For more information see the docs directory.
DISCLAIMER: Please keep in mind that NSQ is designed to be used in a distributed fashion. Single node performance is important, but not the end-all-be-all of what we're looking to achieve. Also, benchmarks are stupid, but here's a few anyway to ignite the flame:
On a 2012 MacBook Air i7 2ghz (GOMAXPROCS=1
, go 1.1 beta2 4a712e80e9b1
, NSQ v0.2.19-alpha)
single publisher, single consumer:
$ ./nsqd --mem-queue-size=1000000
$ ./bench_writer
2013/04/09 23:25:54 duration: 2.46904784s - 77.250mb/s - 405014.429ops/s - 2.469us/op
$ ./bench_reader
2013/04/09 23:27:53 duration: 5.996050461s - 31.810mb/s - 166776.448ops/s - 5.996us/op
The following steps will run NSQ on your local machine and walk through publishing, consuming, and archiving messages to disk.
-
follow the instructions in the INSTALLING doc (or download a binary release).
-
in one shell, start
nsqlookupd
:$ nsqlookupd
-
in another shell, start
nsqd
:$ nsqd --lookupd-tcp-address=127.0.0.1:4160
-
in another shell, start
nsqadmin
:$ nsqadmin --lookupd-http-address=127.0.0.1:4161
-
publish an initial message (creates the topic in the cluster, too):
$ curl -d 'hello world 1' 'http://127.0.0.1:4151/put?topic=test'
-
finally, in another shell, start
nsq_to_file
:$ nsq_to_file --topic=test --output-dir=/tmp --lookupd-http-address=127.0.0.1:4161
-
publish more messages to
nsqd
:$ curl -d 'hello world 2' 'http://127.0.0.1:4151/put?topic=test' $ curl -d 'hello world 3' 'http://127.0.0.1:4151/put?topic=test'
-
to verify things worked as expected, in a web browser open
http://127.0.0.1:4171/
to view thensqadmin
UI and see statistics. Also, check the contents of the log files (test.*.log
) written to/tmp
.
The important lesson here is that nsq_to_file
(the client) is not explicitly told where the test
topic is produced, it retrieves this information from nsqlookupd
and, despite the timing of the
connection, no messages are lost.
NSQ was designed and developed by Matt Reiferson (@imsnakes) and Jehiah Czebotar (@jehiah) but wouldn't have been possible without the support of bitly:
- Dan Frank (@danielhfrank)
- Pierce Lopez (@ploxiln)
- Will McCutchen (@mccutchen)
- Micha Gorelick (@mynameisfiber)
- Jay Ridgeway (@jayridge)
- Justin Hines (@jphines)
- Phillip Rosen (@phillro) for the Node.js Client Library
- David Gardner (@davidgardnerisme) for the PHP Client Library
- Clarity Services (@ClarityServices) for the Ruby Client Library
- Harley Laue (@losinggeneration)
- Justin Azoff (@JustinAzoff)
- Michael Hood (@michaelhood)
- Xianjie (@datastream)
- Dustin Norlander (@dustismo)
- Funky Gao (@funkygao)
- Dan Markham (@dmarkham)
- Francisco Souza (@fsouza)
- galvinhsiu (@galvinhsiu)
- Eric Lubow (@elubow)