Jesque not removing workers from Resque upon end()

[ruckus]

Jesque not removing workers from Resque upon end()

[ruckus]

Well, I see that I am not calling .join() on the worker thread, ala

try { workerThread.join(); } catch (Exception e){ e.printStackTrace(); }

in the example. Calling that after I call worker.end(true) does seem to remove the Worker from Resque, throwing a

WARN  net.greghaines.jesque.worker.WorkerImpl - Terminating in response to exception
java.lang.InterruptedException: sleep interrupted
    at java.lang.Thread.sleep(Native Method)
    at net.greghaines.jesque.worker.WorkerImpl.poll(WorkerImpl.java:459)
    at net.greghaines.jesque.worker.WorkerImpl.run(WorkerImpl.java:206)
    at java.lang.Thread.run(Thread.java:680)

in the process though

[gresrun]

Does the worker thread stop? Use jstack to see if the worker threads are still running.

Worker.end(true) can't stop the thread while the thread is busy processing a job. How long running are your jobs?

On Oct 29, 2011, at 7:52 PM, Cody [email protected] wrote:

I am spinning up Workers and they're processing jobs just fine. I use the existence of a control file to request workers to shut down.

Even though I am calling

worker.end(true);

The worker is still in Resque, waiting for a job. I am monitoring the redis stream using redis-cli monitor and dont see any commands to remove this worker.

         for(Worker w : avatarWorkers) {
           if(w != null) {
             log.info("Stopping avatar worker: " + w.toString());
             w.end(true);
           }
         }

Results log messages like:

2011-10-29 16:42:58,436 [main] INFO  com.batch.App - Stopping avatar worker: dev:worker:Codys-MacBook-Pro.local:10065-6:JAVA_DYNAMIC_QUEUES,avatars

But they're stil in Resque / Redis.

Am I doing something wrong?

Reply to this email directly or view it on GitHub:
#11

[gresrun]

Calling join() simply tells the current thread to wait until the worker thread stops. The true in end(true) is teling the worker to attempt to interrupt a job in-progress and it does this by calling Thread.interrupt() on the worker thread. Now, a job will only stop if it is in a method checks for interrupts (e.g.: Thread.sleep(), any I/O task, Object.wait(), etc.)

The warning in your last post is from the worker being interrupted while waiting on a job to arrive and this is the desired behavior.

If you're still having problems with the workers not shutting down, use the JDK's jstack command-line tool to get a stack trace of all currently running threads in your process and I'll help you interpret the output.

[gresrun]

Let me know if you need any help with this; threads are a sticky subject and I always ask for a second pair of eyeballs when I'm writing a multi-threaded app.

[ruckus]

Thank you for being on top of this.

I've made some changes to the code but it won't get deployed today. Namely, I was not waiting for the thread to finish after calling worker.end(true) - that is, I wasn't calling Thread.join on the worker thread. I don't know how important that is.

Anyways, I will try and dig into this more and keep you posted.

Thanks.

On Oct 31, 2011, at 8:07 AM, Greg Haines wrote:

Let me know if you need any help with this; threads are a sticky subject and I always ask for a second pair of eyeballs when I'm writing a multi-threaded app.

Reply to this email directly or view it on GitHub:
#11 (comment)

[ruckus]

It appears this problem was on my end the whole time.

I am using a Sun JVM and was capturing the signal via:

    Signal.handle(new Signal("TERM"), new SignalHandler() {
      public void handle(final Signal sig) {
        log.info("Caught TERM signal");
        shutdown();
        System.exit(0);
      }
    });

And I was running my code via a terminal session, and using ctrl+c to kill the process. My error was that I assumed ctrl+c sends a TERM signal, but in fact it sends a INT signal, which I wasnt registered to receive.

Thus I was never giving Jesque a chance to clean up after itself.

Sorry to waste your time.

[gresrun]

No problem. Glad you figured it out.

[bendilley]

@ruckus I found this tip to avoid Sun-specific code: http://stackoverflow.com/a/2541950/418602

I got to the bottom of my similar problem by reading through this issue, so thanks for raising it 👍 even if it wasn't technically a 🐛 😉