fmmfonseca/work_queue

A tunable work queue, designed to coordinate work between a producer and a pool of worker threads.

work_queue's Issues

Deadlock when spawning threads

I have gotten a deadlock when spawning threads in Ruby 1.9.3. (This has only happened once that i am aware.)

The create_work_queue method that caused the deadlock simply creates a WorkQueue then starts up the threads and makes them do a little work upon initialization. The implementation of the code that caused the deadlock and the backtrace are below:

def self.create_work_queue
  work_queue = WorkQueue.new(WORK_QUEUE_SIZE)

  # Force threads to be available - the work_queue gem seems to not always create the threads
  WORK_QUEUE_SIZE.times do |i|
    work_queue.enqueue_b do
      1000.times { |i| i ** 1000 }
    end
  end

  work_queue
end

fatal:deadlock detected
/usr/local/rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/monitor.rb:185:in `lock'
/usr/local/rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/monitor.rb:185:in `mon_enter'
/usr/local/rvm/rubies/ruby-1.9.3-p125/lib/ruby/1.9.1/monitor.rb:209:in `mon_synchronize'
/home/bkr/.bundle/4218576002/614614351/ruby/1.9.1/gems/work_queue-2.5.2/lib/work_queue.rb:197:in `spawn_thread'
/home/bkr/.bundle/4218576002/614614351/ruby/1.9.1/gems/work_queue-2.5.2/lib/work_queue.rb:190:in `enqueue'
/home/bkr/.bundle/4218576002/614614351/ruby/1.9.1/gems/work_queue-2.5.2/lib/work_queue.rb:106:in `enqueue_b'
/home/bkr/.bundle/4218576002/614614351/ruby/1.9.1/gems/parallizer-0.2.2/lib/parallizer.rb:60:in `block in create_work_queue'
...

Unable to create new native thread

Here's the backtrace:

ThreadError: unable to create new native thread
initialize at org/jruby/RubyThread.java:301
new at org/jruby/RubyThread.java:230
spawn_thread at /home/tools/gems/jruby/1.9/gems/work_queue-2.0.1/lib/work_queue.rb:198
mon_synchronize at /usr/jruby-1.6.7/lib/ruby/1.9/monitor.rb:201
mon_synchronize at /usr/jruby-1.6.7/lib/ruby/1.9/monitor.rb:200
spawn_thread at /home/tools/gems/jruby/1.9/gems/work_queue-2.0.1/lib/work_queue.rb:196
enqueue at /home/tools/gems/jruby/1.9/gems/work_queue-2.0.1/lib/work_queue.rb:171
mon_synchronize at /usr/jruby-1.6.7/lib/ruby/1.9/monitor.rb:201
mon_synchronize at /usr/jruby-1.6.7/lib/ruby/1.9/monitor.rb:200
enqueue at /home/tools/gems/jruby/1.9/gems/work_queue-2.0.1/lib/work_queue.rb:166
enqueue_b at /home/tools/gems/jruby/1.9/gems/work_queue-2.0.1/lib/work_queue.rb:129
process_data at ./script/loader.rb:129
each at org/jruby/RubyArray.java:1615
process_data at ./script/loader.rb:127
file at ./script/loader.rb:214
measure at /usr/jruby-1.6.7/lib/ruby/1.9/benchmark.rb:295
file at ./script/loader.rb:211
each at org/jruby/RubyArray.java:1615
(root) at ./script/loader.rb:207

This happens at around 40 min of runtime with only 10 worker threads. Any possibility WorkQueue is leaking threads or something?

Running under:
jruby 1.6.7 (ruby-1.8.7-p357) (2012-02-22 3e82bc8) (Java HotSpot(TM) 64-Bit Server VM 1.6.0_27) [linux-amd64-java]
CentOS 6.2 x86_64

stacktrace from within thread

I’m not sure if I’m just not seeing it, but is there a nice way to get the stacktrace/error from an enqueued thread? I know failed Threads return nil, but I have yet to find a way to access the exception without including or monkey patching a rescue block that prints the stackttrace. Any hints would greatly appreciated.

1.9.3p194 :004 > wq.enqueue_b { put "Hello from the WorkQueue" }
 => [#<Thread:0x000000010079a8 sleep>] 
1.9.3p194 :007 > wq.enqueue_b { raise "Hello from the WorkQueue" }
 => nil 

(In my specific case, just printing the error and trace to the terminal is the most helpful approach)