kureuil/batch-rs

Background job library for Rust supporting RabbitMQ

batch-rs's Issues

Write a Faktory adapter

Faktory is a new message broker (written by the author of sidekiq) that aims to be as language agnostic as possible. It probably shouldn't be too hard to write an adapter for it given how it works.

There already exists a client faktory-rs but its a high-level one, and it only supports synchronous I/O for now.

Improve Worker parallelism

Currently, the Worker only executes jobs one by one. It should be possible to spawn as many jobs as available core on the current machine, greatly improving the performance.

Make sure macros work with Rust 2018

In Rust 2018 Edition, paths have been reworked and the code generated by our macros might be incompatible with this rework.

Fix README example

The "Batch in action" example currently written in the README doesn't compile. We should set up a tool such as rust-skeptic to automatically check that the README stays up-to-date with the state of the project.

Also it would seem there's tabs instead of spaces.

The guide isn't deployed after committing on master

May be related to:

• https://blog-staging.travis-ci.com/2018-06-21-build-conditions-v1-is-available
• https://docs.travis-ci.com/user/conditions-v1

Emit a warning when a Task can't be associated to a Queue

When building a Worker, if a Task cannot be associated to any Queue using its exchange & routing key a warning should be emitted to inform the user as their's a high chance it's an error.

Allow integration of error reporting services

e.g when a task execution fails, we should provide a simple hook for people to publish the error to their favourite error reporting service (Sentry, Raygun, Rollbar, Bugsnag, etc.). Integrations should not be tied to the batch crate.

Refactor task name generation

When using batch-derive, the task of the name is directly inferred from the struct's location (its absolute module path) and from the struct's name which might give a name such as webapp::tasks::SendSignupEmail. However this task name format isn't often used, the separator is usually a dot (.) instead of a double colon (::), this should be changed. It will probably need to use the lazy-static crate as Task::name() should still return a &'static str and String operations on literals are not executed at compile-time.

Improve task name generation

The auto-generated task name is just the type name, making it too easy to get name collisions. We should use the module_path! macro to prefix the task name with its containing module.

lazy-static shouldn't be peer dependency

Since the task name refactor (#45), users of this library are required to specify lazy-static as a direct dependency of their project, even though they never have to explicitly use it.

A way of having the same behavior without forcing users to install lazy-static should be found.

Change terminology from Task to Job

There seems to be more momentum behind the term Job, and keeping the term Task might induce some confusion as the futures is using the same word for their work unit.

Add work-flow primitives

Probably at least Chain and Group, and be able to compose these primitives.

Officially support macOS

Even if the library should be able to work correctly on macOS, no tests were made and the CI doesn't run the tests on a macOS system.

Write an Amazon SQS adapter

Amazon SQS seems to be a pretty popular choice when needing a managed queue service on AWS.

The simplest way to use it will probably be to use one of the Rusoto crates, the AWS SDK for Rust.

Remove unwrap in examples

See C-QUESTION-MARK in the API Guidelines:

Like it or not, example code is often copied verbatim by users. Unwrapping an error should be a conscious decision that the user needs to make.

Write a Disque adapter

Disque is a message broker written by the author of Redis. Disque aims to be as simple as Redis while incorporating the high-level blocks of a message broker (message TTL, acknowledgements & retries, etc.). It is currently pending a rewrite as a Redis module.

Add integration tests for proc-macros

Proc macro code is currently rather messy and can't be easily refactored because it lacks tests.

We should integrate [compiletest-rs] to make it easy to work on the *-codegen crates.

compiletest-rs: https://github.com/laumann/compiletest-rs

Add a contributing guide

Add a changelog

Maintain a changelog, allowing users to quickly understand what will be required to upgrade from one version to another.

See Keep a Changelog

Migrate to tokio reform

Tracking issue for the upcoming futures/tokio 0.2 ecosystem migration. It should be pretty straightforward once our dependencies will be updated:

• https://github.com/rust-lang-nursery/futures-rs
• https://github.com/tokio-rs/tokio
• https://github.com/sozu-proxy/lapin
• https://github.com/sozu-proxy/lapin-futures-tls

Document version support

We should document:

• Minimum supported version of the Rust compiler
• Minimum supported version of RabbitMQ server

Add a persistent result backend

Such a backend will allow us to store the status of past, current and upcoming tasks in a sensible and will allow us to easily build dashboards to visualize tasks & stats, and more.

In the first version, the backend will probably by SQL based though we might want to loosen the requirement on this one at a later point.

Investigate using Travis for Windows CI

Travis recently announced support for Windows on their CI platform. Might be interesting to have all CI on one service ?

The Travis build is failing on Linux

During the installation of RabbitMQ 3.7. Doesn't seem to be related to batch's code:

$ if [ "$TRAVIS_OS_NAME" = linux ]
  then
    sudo mv /opt/jdk_switcher/jdk_switcher.sh /tmp
    sudo apt-get install rabbitmq-server
    sudo mv /tmp/jdk_switcher.sh /opt/jdk_switcher/
    sudo service rabbitmq-server start
  fi
  
Reading package lists...
Building dependency tree...
Reading state information...
The following packages were automatically installed and are no longer required:
  couchdb-common libmozjs185-1.0
Use 'sudo apt autoremove' to remove them.
The following packages will be upgraded:
  rabbitmq-server
1 upgraded, 0 newly installed, 0 to remove and 186 not upgraded.
Need to get 9,105 kB of archives.
After this operation, 4,850 kB of additional disk space will be used.
Get:1 https://dl.bintray.com/rabbitmq/debian trusty/main amd64 rabbitmq-server all 3.7.6-1 [9,105 kB]
Fetched 9,105 kB in 3s (2,465 kB/s)
(Reading database ... 100104 files and directories currently installed.)
Preparing to unpack .../rabbitmq-server_3.7.6-1_all.deb ...
 * Stopping message broker rabbitmq-server       * message broker already stopped
Unpacking rabbitmq-server (3.7.6-1) over (3.6.14-1) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Processing triggers for ureadahead (0.100.0-16) ...
Setting up rabbitmq-server (3.7.6-1) ...
Installing new version of config file /etc/init.d/rabbitmq-server ...
Installing new version of config file /etc/logrotate.d/rabbitmq-server ...
/var/lib/dpkg/info/rabbitmq-server.postinst: 8: /etc/profile.d/travis-haskell.sh: [[: not found
 * Starting message broker rabbitmq-server       * FAILED - check /var/log/rabbitmq/startup_\{log, _err\}
fail]
invoke-rc.d: initscript rabbitmq-server, action "start" failed.
dpkg: error processing package rabbitmq-server (--configure):
 subprocess installed post-installation script returned error exit status 1
Processing triggers for ureadahead (0.100.0-16) ...
Errors were encountered while processing:
 rabbitmq-server
E: Sub-process /usr/bin/dpkg returned an error code (1)
 * Starting message broker rabbitmq-server
 * FAILED - check /var/log/rabbitmq/startup_\{log, _err\}
   ...fail!

Write a Redis adapter

Make it possible to dispatch job to a Redis instance. Inspiration can be taken from the sidekiq and dramatiq projects on how to successfully use Redis as a reliable message broker.

Add issue templates

Update RabbitMQ version on Travis

When running the tests on Ubuntu Trusty (14.04), Travis provides version 3.6 of RabbitMQ. We should upgrade it to 3.7 to test the same version on all 3 OS.
Also, it would seem that RabbitMQ 3.6 support has recently stopped.

http://www.rabbitmq.com/install-debian.html

Add a task scheduler

Create a Scheduler object, able to schedule tasks execution.

Refactor manual Debug implementations

Manual Debug implementation of batch's types format their output using a string as a template but they should be replaced by the debug_struct method.

Improve builers discoverability

Following the API docs, it might not be obvious that Exchange is the result of ExchangeBuilder::build() and that Queue is the result of QueueBuilder::build().

A simple solution for now would be to remove the new constructors on ExchangeBuilder & QueueBuilder and replace them by a builder static function on Exchange & Queue respectively.

Have the worker use a process pool

When the worker receives a job from the broker, it spawns a brand new process per job. While this is easy to implement and monitor, it is sub-optimal on the performance side.

One possible solution to this performance problem would be to have a pool of processes, which are kept alive as long as possible. The main worker process would send over to these "worker worker" processes the jobs to execute, and they would send a message back to acknowledge or reject the job execution. If one the processes dies, it should be restarted and the job should be sent back to the broker as "rejected".

Customize the process title of the worker's child processes

Celery can use the setproctitle package to rename a child process, making it easier to identify the task being executed by a worker child process when visualizing them in ps or top for example.

setproctitle is based on code written by the PostgreSQL project: https://doxygen.postgresql.org/ps__status_8c_source.html

It could be interesting to port this code to Rust and integrate its functionality to batch.

Officially support Windows

Even if the library should be able to work correctly on Windows, no tests were made and the CI doesn't run the tests on a Windows system.

Add a web accessible dashboard

Depends on #13.

Add a web dashboard, allowing administrators to:

• visualize the status of past / current / upcoming tasks
• access logs of current / past tasks
• retry failed tasks

The dashboard will be made available through middlewares for web frameworks, allowing for maximum flexibility.

Add task priorities

RabbitMQ natively supports priority queues, adding support in batch should be rather straightforward.

Upstream poll gave error

When I run multiple instances, a connection error appears one after the other.

root@10f834387d04:/app/batch# cargo run --example simple-client
    Finished dev [unoptimized + debuginfo] target(s) in 0.0 secs
     Running `/app/target/debug/examples/simple-client`
Starting RabbitMQ client example
root@10f834387d04:/app/batch# cargo run --example simple-client
   Compiling batch v0.1.1 (file:///app/batch)
    Finished dev [unoptimized + debuginfo] target(s) in 19.29 secs
     Running `/app/target/debug/examples/simple-client`
Starting RabbitMQ client example
root@10f834387d04:/app/batch# cargo run --example simple-client
    Finished dev [unoptimized + debuginfo] target(s) in 0.1 secs
     Running `/app/target/debug/examples/simple-client`
Starting RabbitMQ client example
root@10f834387d04:/app/batch# cargo run --example simple-client
    Finished dev [unoptimized + debuginfo] target(s) in 0.1 secs
     Running `/app/target/debug/examples/simple-client`
Starting RabbitMQ client example
^[[AERROR 2018-04-20T21:39:03Z: lapin_futures::transport: upstream poll gave error: Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error { inner:

An error occured in the RabbitMQ broker: Connection reset by peer (os error 104) }', libcore/result.rs:945:5
note: Run with `RUST_BACKTRACE=1` for a backtrace.

Register tasks on Queue instead of the Worker ?

This might reduce duplication and remove some string-y API.

let exchanges = vec![exchange("default.exchange")];
let queues = vec![
    queue("default.queue")
        .task::<SayHello>("default.exchange")
];

WorkerBuilder::new()
    .exchanges(exchanges)
    .queues(queues)
    .build();