Giter Site home page Giter Site logo

squaremo / rabbitmq-streams Goto Github PK

View Code? Open in Web Editor NEW
38.0 7.0 5.0 42.24 MB

Data streams management using RabbitMQ. OF ARCHAEOLOGICAL INTEREST ONLY

License: Other

Shell 0.93% JavaScript 22.51% Python 36.75% Java 19.40% Erlang 8.43% Scala 11.23% Io 0.31% Emacs Lisp 0.18% XSLT 0.02% Awk 0.07% OpenEdge ABL 0.16%

rabbitmq-streams's Introduction

What is this?

"RabbitMQ Streams" is our name for the open source project developed with the BBC to power the "BBC Feeds Hub". It is a data streams management system.

For background information, see

You can think of RabbitMQ Streams as a distributed, robust, scalable, secure, user-friendly and manageable version of Unix pipes.

The basic logical building blocks are Sources and Destinations of data and Pipelines. The latter are composed of PipelineComponents which can route (e.g. based on regexp matches on Atom feed entries), merge and transform the data in arbitrary ways.

Data arrive at sources, and leave from destinations, via Gateways, which talk various protocols to the outside world.

Gateways as well as pipeline components (jointly referred to as Plugins) can currently be written in Java and Python, and require little boilerplate (see e.g. regexp_replace.py Support for other languages can be added straightforwardly by creating a Harness; plugins are essentially just programs following a simple protocol, with the harness taking care of much of the detail.

The message wiring and plugin processes are managed by an Erlang/OTP application called the Orchestrator. This is in a sense the core of Streams.

Current state

No longer being developed. May be interesting to look at though.

Getting started

The Makefile has a number of targets useful for development.

make setup installs build dependencies and sets up a development environment. This is geared towards apt-get, but it's fairly easy to do the equivalent for e.g., macports.

We also need RabbitMQ and CouchDB; make setup builds these from source in build/opt; make install-dev-debs will just install build dependencies without building RabbitMQ and CouchDB.

make all builds the things . This currently relies on a Maven repository for the Java harness and plugins, which we'll make available.

make create-fresh-accounts will install a minimal configuration, and create a user in RabbitMQ for Streams to use.

Descriptions of the model items (sources, gateways, etc.) are kept in CouchDB. sbin/import_config.py DIR can be used to import whole configurations at once; e.g., sbin/import_config.py examples/showandtell_demo.

make run will start RabbitMQ and CouchDB from the local builds, start the development code, and tail the logs for you.

make listen-orchestrator start-orchestrator-nox starts just the Streams orchestrator and tail its log in an xterm.

There is a more detailed "Getting started" guide in doc/getting_started_dev.org.

rabbitmq-streams's People

Contributors

aschmolck avatar msackman avatar squaremo avatar tonyg avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.