inwork/queue_dispatcher

Executes asynchronous tasks in the background.

QueueDispatcher for Rails3 / Rails4¶ ↑

<img src=“https://codeclimate.com/github/InWork/queue_dispatcher.png” />

This Rails3 Gem implements a method to perform long running methods in the background. Background tasks will be executed by persistent workers.

Install¶ ↑

Inside your Gemfile:

gem "queue_dispatcher"

and then run:

bundle install

To install the queue_worker_dispatcher script, which starts all workers, execute the following rake command:

rake queue_dispatcher:sync

Database Setup¶ ↑

Use

rails g queue_dispatcher:migration

This will create a database migration for the models Task and TaskQueues.

If you update the queue_dispatcher from pre 1.5.1 you have to use the following command to update your migrations:

Use

rails g queue_dispatcher:migration --skip

This will create all new database migrations for the models Task and TaskQueues and leve the existing as they are.

Gem Dependencies¶ ↑

Please check if all those requirements are satisfied on your environment.

• rails >= 3.0.0

• sys-proctable >= 0.9.1

Inside your application¶ ↑

To enqueue a long running task, simple call a method through enque. E.g.: Assume, you have a long running job:

LongRunningMailJob.send_mail

Now we’d like to execute it in the background by simply calling:

task = LongRunningMailJob.enqueue.send_mail

If you like to put the job in a queue, you can do this by execute it the following way:

task = LongRunningMailJob.enqueue(queue: 'queue_name').send_mail

Jobs inside a queue are executed serialized, not in parallel. You can define dependencies. A task is then executed only after all dependent tasks are finished. The dependencies could also be in another queue. This way you could ensure, that a task is only executed when another task from another queue is successfully finished. Code to add Task dependencies:

task.dependent_tasks = another_task

Queue Worker Dispatcher¶ ↑

The QueueWorkerDispatcher-script starts the workers (default are 10 workers). A worker waits for a new queue and executes all tasks of this queue. Start the QueueWorkerDispatcher by executing the following command:

script/queue_worker_dispatcher

To start the QueueWorkerDispatcher as a daemon, use the option -b.

-b, --background        work in background mode

License¶ ↑

This project is licenced under the MIT license.

Author¶ ↑

Philip Kurmann (philip (at) kman.ch)

queue_dispatcher's Issues

Replace Protoyp periodically_call_remote and onclick_to_remote with jquery functions

In _my_event view we use an the prototyp function periodically_call_remote and in _task_my_event the prototyp function onclick_to_remote.

We should not have dependency on the prototyp-rails gem has, we didn't acquire to used it with our gem.

'can't be called from trap context'-Error when a INT-signal is sent with Ruby >= 2.0.0

If a Sig-Int is sent to the Worker-Dispatcher under Ruby >= 2.0.0, Ruby reports a 'can't be called from trap context'-Error.
It seems, that Ruby 2.0 tries to be more thread-safe and raise an exception, if something is called which could cause a deadlock.

The workaround could be, putting the problematic code (like writing to a DB or to a Log-File) inside a new thread.

ERROR: Fatal error in method 'run!': can't modify frozen NilClass

When I start a new task I get the following error, with queue_dispatcher version "2.5.2"

  /Users/myname/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/bundler/gems/queue_dispatcher-b3addf0ba342/lib/queue_dispatcher/acts_as_task_queue.rb:293:in `run!'
  script/queue_worker_dispatcher:221:in `worker_runner'
  script/queue_worker_dispatcher:185:in `block in spawn_and_monitor_workers'
  /Users/myname/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/spawnling-2.1.6/lib/spawnling.rb:122:in `block in run'
  /Users/myname/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/spawnling-2.1.6/lib/spawnling.rb:184:in `block in fork_it'
  /Users/myname/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/spawnling-2.1.6/lib/spawnling.rb:160:in `fork'
  /Users/myname/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/spawnling-2.1.6/lib/spawnling.rb:160:in `fork_it'
  /Users/myname/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/spawnling-2.1.6/lib/spawnling.rb:122:in `run'
  /Users/myname/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/spawnling-2.1.6/lib/spawnling.rb:102:in `initialize'
  script/queue_worker_dispatcher:184:in `new'
  script/queue_worker_dispatcher:184:in `spawn_and_monitor_workers'
  script/queue_worker_dispatcher:128:in `daemon_runner'
  script/queue_worker_dispatcher:158:in `daemon_start'
  script/queue_worker_dispatcher:281:in `<main>'

The error is caused when no @engine instance variable is set. My debugging showed that the if statement in

queue_dispatcher/lib/queue_dispatcher/acts_as_task_queue.rb

Line 293 in 6e8e9f5

@engine.logger = @logger if defined? @engine && @engine.methods.include?(:logger=)

is not correct, which came with the commit d2f9e3a.

See debuggin:

puts "oh no" if defined? @engine
=> nil
[9] pry(#<TaskQueue>)> puts "oh no" if defined? @engine && @engine.methods.include?(:logger=)
oh no

The correct statement should be

pry(#<TaskQueue>)> puts "oh no" if (defined? @engine) && (@engine.methods.include?(:logger=))
=> nil

Mysql2::Error: Data too long for column message

It looks like that MySQL 5.6.13 is more striker than the previews version.

If you want to but a long string into message field from task table you get the error.
Mysql2::Error: Data too long for column message

To fix the problem, we should change the field typ from string to text.

  create_table "<%= options[:tasks_table_name] %>" do |t|
     .....
      t.string :message
     .....
    end

  create_table "<%= options[:tasks_table_name] %>" do |t|
     .....
      t.text :message
     .....
    end

https://github.com/InWork/queue_dispatcher/blob/master/lib/generators/queue_dispatcher/migration/templates/tasks.rb#L9

I would to it, but I am not sure how the migration in a gem behave. To you know is there a way like on a rails application to create an update migration?

No closing string in javascript

<% @deleted_task_ids.each do |id| %>
  jQuery('#task_<%= id %>).fadeOut();
<% end %>

Replace Prototyp Event Expander

The view is based on jQuery except the expand_event javascript.

page.visual_effect :toggle_blind, "task_#{@task.id}_details", :duration => 0.1

We should only use one javascript framework to avoid conflicts.

Active Job

We should check if we can support Active Job which is part of Rails 4.2.

http://weblog.rubyonrails.org/2014/12/19/Rails-4-2-final/

http://api.rubyonrails.org/classes/ActiveJob/QueueAdapters.html