Giter Site home page Giter Site logo

tddmonkey / elasticmq Goto Github PK

View Code? Open in Web Editor NEW

This project forked from softwaremill/elasticmq

0.0 1.0 0.0 1.5 MB

Message queueing system with an actor-based Scala and Amazon SQS-compatible interfaces. Runs stand-alone or embedded.

Home Page: http://www.softwaremill.com

License: Apache License 2.0

Scala 100.00%

elasticmq's Introduction

ElasticMQ

tl;dr

  • message queue system
  • runs stand-alone (download) or embedded
  • Amazon SQS-compatible interface
  • fully asynchronous implementation, no blocking calls

Summary

ElasticMQ is a message queue system, offering an actor-based Scala and an SQS-compatible REST (query) interface.

ElasticMQ follows the semantics of SQS. Messages are received by polling the queue. When a message is received, it is blocked for a specified amount of time (the visibility timeout). If the message isn't deleted during that time, it will be again available for delivery. Moreover, queues and messages can be configured to always deliver messages with a delay.

The focus in SQS (and ElasticMQ) is to make sure that the messages are delivered. It may happen, however, that a message is delivered twice (if, for example, a client dies after receiving a message and processing it, but before deleting). That's why clients of ElasticMQ (and Amazon SQS) should be idempotent.

As ElasticMQ implements a subset of the SQS query (REST) interface, it is a great SQS alternative both for testing purposes (ElasticMQ is easily embeddable) and for creating systems which work both within and outside of the Amazon infrastructure.

The future will most probably bring even more exciting features :).

Community

Installation: stand-alone

You can download the stand-alone distribution here: https://s3/.../elasticmq-server-0.8.12.jar

Java 6 or above is required for running the server.

Simply run the jar and you should get a working server, which binds to localhost:9324:

java -jar elasticmq-server-0.8.12.jar

ElasticMQ uses Typesafe Config for configuration. To specify custom configuration values, create a file (e.g. custom.conf), fill it in with the desired values, and pass it to the server:

java -Dconfig.file=custom.conf -jar elasticmq-server-0.8.12.jar

The config file may contain any configuration for Akka, Spray and ElasticMQ. Current ElasticMQ configuration values are:

include classpath("application.conf")

// What is the outside visible address of this ElasticMQ node (used by rest-sqs)
node-address {
    protocol = http
    host = localhost
    port = 9324
    context-path = ""
}

rest-sqs {
    enabled = true
    bind-port = 9324
    bind-hostname = "0.0.0.0"
    // Possible values: relaxed, strict
    sqs-limits = relaxed
}

By default the maximum SQS message wait time is 20 seconds, and the maximum duration of a request is set to 21 seconds. To change that (e.g. if you want longer message wait times), adjust the spray.can.server.request-timeout configuration property.

You can also provide an alternative Logback configuration file (the default is configured to log INFO logs and above to the console):

java -Dlogback.configurationFile=my_logback.xml -jar elasticmq-server-0.8.12.jar

Starting an embedded ElasticMQ server with an SQS interface

val server = SQSRestServerBuilder.start()
// ... use ...
server.stopAndWait()

If you need to bind to a different host/port, there are configuration methods on the builder:

val server = SQSRestServerBuilder.withPort(9325).withInterface("localhost").start()
// ... use ...
server.stopAndWait()

You can also provide a custom ActorSystem; for details see the javadocs.

Embedded ElasticMQ can be used from any JVM-based language (Java, Scala, etc.).

Using the Amazon Java SDK to access an ElasticMQ Server

To use Amazon Java SDK as an interface to an ElasticMQ server you just need to change the endpoint:

client = new AmazonSQSClient(new BasicAWSCredentials("x", "x"))
client.setEndpoint("http://localhost:9324", "sqs", "")

The endpoint value should be the same address as the NodeAddress provided as an argument to SQSRestServerBuilder or in the configuration file.

The rest-sqs-testing-amazon-java-sdk module contains some more usage examples.

Using the Amazon boto (Python) to access an ElasticMQ Server

To use Amazon boto as an interface to an ElasticMQ server you set up the connection using:

region = boto.sqs.regioninfo.RegionInfo(name='elasticmq',
                                        endpoint=sqs_endpoint)
conn = boto.connect_sqs(aws_access_key_id='x',
                        aws_secret_access_key='x',
                        is_secure=False,
                        port=sqs_port,
                        region=region)

where sqs_endpoint and sqs_port are the host and port.

ElasticMQ dependencies in SBT

// Scala 2.11
val elasticmqSqs        = "org.elasticmq" %% "elasticmq-rest-sqs"         % "0.8.12"

// Scala 2.10
val elasticmqSqs        = "org.elasticmq" %% "elasticmq-rest-sqs"         % "0.7.1"

If you don't want the SQS interface, but just use the actors directly, you can add a dependency only to the core module:

val elasticmqCore       = "org.elasticmq" %% "elasticmq-core"             % "0.8.12"

If you want to use a snapshot version, you will need to add the https://oss.sonatype.org/content/repositories/snapshots/ repository to your configuration.

ElasticMQ dependencies in Maven

Dependencies:

<dependency>
    <groupId>org.elasticmq</groupId>
    <artifactId>elasticmq-rest-sqs_2.11</artifactId>
    <version>0.8.12</version>
</dependency>

If you want to use a snapshot version, you will need to add the https://oss.sonatype.org/content/repositories/snapshots/ repository to your configuration.

Replication, journaling, SQL backend

Until version 0.7.0, ElasticMQ included optional replication, journaling and an SQL message storage. These modules have not yet been reimplemented using the new Akka core.

Current versions

Stable: 0.8.12

Development: 0.8.13-SNAPSHOT

Logging

ElasticMQ uses Slf4j for logging. By default no logger backend is included as a dependency, however Logback is recommended.

Performance

Tests done on a 2012 MBP, 2.6GHz, 16GB RAM, no replication. Throughput is in messages per second (messages are small).

Directly accessing the client:

Running test for [in-memory], iterations: 10, msgs in iteration: 100000, thread count: 1.
Overall in-memory throughput: 21326.054040

Running test for [in-memory], iterations: 10, msgs in iteration: 100000, thread count: 2.
Overall in-memory throughput: 26292.956117

Running test for [in-memory], iterations: 10, msgs in iteration: 100000, thread count: 10.
Overall in-memory throughput: 25591.155697

Through the SQS REST interface:

Running test for [rest-sqs + in-memory], iterations: 10, msgs in iteration: 1000, thread count: 20.
Overall rest-sqs + in-memory throughput: 2540.553587

Running test for [rest-sqs + in-memory], iterations: 10, msgs in iteration: 1000, thread count: 40.
Overall rest-sqs + in-memory throughput: 2600.002600

Note that both the client and the server were on the same machine.

Test class: org.elasticmq.performance.LocalPerformanceTest.

Building, running, and packaging

To build and run with debug (this will listen for a remote debugger on port 5005):

~/workspace/elasticmq $ sbt -jvm-debug 5005  
> project elasticmq-server
> run

To build a jar-with-dependencies:

~/workspace/elasticmq $ sbt
> project elasticmq-server
> assembly

Technology

  • Core: Scala and Akka.
  • Rest server: Spray, a high-performance, asynchronous, REST/HTTP toolkit.
  • Testing the SQS interface: Amazon Java SDK; see the rest-sqs-testing-amazon-java-sdk module for the testsuite.

Change log

Version 0.8.12 (30 Sep 2015)

  • checking queue length if limits are strict

Version 0.8.11 (3 Sep 2015)

  • downgrading typesafe-config to keep Java6 compatibility

Version 0.8.10 (3 Sep 2015)

  • numeric attributes support (thx @sf-git)

Version 0.8.9 (10 Aug 2015)

  • binary attributes support (thx @brainoutsource)
  • dependency updates

Version 0.8.8 (10 Apr 2015)

  • dependency updates, bug fixes

Version 0.8.6, 0.8.7 (7 Feb 2015, 13 Feb 2015)

  • adding support for the QueueArn attribute

Version 0.8.5 (11 Dec 2014)

  • supporting PurgeQueue action instead of a custom one

Version 0.8.4 (2 Dec 2014)

  • custom action for clearing all messages from a queue

Version 0.8.3 (22 Oct 2014)

  • bug fixes
  • updating dependencies
  • publishing to Maven Central

Version 0.8.2 (6 Jun 2014)

  • increasing the bind timeout
  • initial support for String-valued message attributes (thx @hayesgm)

Version 0.8.1 (29 May 2014)

  • fixing Node.JS compatibility
  • fixing a bug when calculating queue attributes
  • updating to Scala 2.11.1, Akka 2.3.3

Version 0.8.0 (29 April 2014)

  • updating to Scala 2.11, Akka 2.3.2
  • updating libraries to latest versions

Version 0.7.1 (22 August 2013)

  • bug fixes

Version 0.7.0 (5 June 2013)

  • reimplemented using Akka and Spray (actor-based, no blocking)
  • long polling support
  • bug fixes

Version 0.6.3 (21 January 2013)

  • Scala 2.10 support
  • Changing the way the stand-alone server is configured

Version 0.6.2 (13 December 2012)

  • bug fixes
  • properly handling SQS receipt handles - message can be deleted only when passing the most recent receipt handle

Version 0.6.1 (18 November 2012)

  • using Sonatype's OSS repositories for releases
  • library upgrades

Version 0.6 (19 October 2012)

  • batch operations in SQS (send, receive, delete, change visibility)
  • changed SQSRestServerFactory to SQSRestServerBuilder
  • "strict" and "relaxed" modes when creating an SQS server: the limits enforced by SQS are optionally checked, e.g. for batch operations (max 10 messages), maximum message size (64KB). Strict by default.

Version 0.5 (26 May 2012)

  • stand-alone distribution (download)
  • file log for message storage (journal)
  • factoring out storage-database module, to decrease the dependencies of the core modules

Version 0.4 (27 Mar 2012)

  • replication

Version 0.3 (6 Feb 2012)

  • in-memory storage
  • new native API
  • bug fixes

Version 0.2 (12 Jan 2012)

  • new SQS functions support
  • testing with Amazon Java SDK
  • bug fixes

Version 0.1 (12 Oct 2011)

  • initial release
  • DB storage
  • SQS interface support

elasticmq's People

Contributors

adamw avatar alexwei avatar averyregier avatar bitdeli-chef avatar brainoutsource avatar cancan101 avatar conniec avatar ddossot avatar ephemeralsnow avatar gasi avatar hayesgm avatar iristyle avatar jaceklaskowski avatar jcogilvie avatar maciej avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.