breamware/sidekiq-batch

Sidekiq Batch Jobs Implementation

sidekiq-batch's Issues

on_success callback not called after job retried

If a job inside of a batch is retried, the on_complete callback will be fired (before the failed job is retried) but the on_success callback will never fire, even after the failed job is retried successfully.

Non-safe JSON arguments Sidekiq warning

Sidekiq ~> 6 warns about passing non-safe JSON arguments to jobs via this message:

Job arguments to #{item["class"]} do not serialize to JSON safely. This will raise an error in
Sidekiq 7.0. See https://github.com/mperham/sidekiq/wiki/Best-Practices or raise an error today
by calling Sidekiq.strict_args! during Sidekiq initialization.

Moreover, as you can see, Sidekiq ~> 7 will raise an error (it already does if Sidekiq.raise_args! is invoked)
This gem, unfortunately, pushes some symbols instead of strings (e.g. ./sidekiq/batch:176). It can be resolved by forcibly converting events names (e.g. :complete) to string
If that's an issue, I'd like to propose a PR (if required)

Batches Tab

Does this gem add "Batches" tab in the Web UI, I don't see it. Am I doing something wrong?

Callback success fired too soon with ActionMailer

Hi,

The success callback is called too soon due to ActionMailer. (class ApplicationMailer < ActionMailer::Base)

I have a complex workflow with six steps (to start).
And when a mail is sent by ActionMailer in the middle of step 2, it fires success callback even the step 2 is not finished.

So, the step 3 begins whereas the step 2 is still in progress.

When I disable the sent of mail, the workflow is followed perfectly.

Question about Batch Callback

Thank you for this gem, I was trying to implement Callbacks from my batches, but everything I try does not work. Can someone point me out the mistake? Thanks

I call: Testworker.perform_async("hello")

class TestWorker
  include Sidekiq::Worker
  include Sidekiq::Status::Worker
  sidekiq_options queue: 'default'

  def perform(name)
    batch = Sidekiq::Batch.new
    batch.on(:success, 'TestWorkerCallback#perform')
    batch.jobs do
      TestWorkerTwo.perform_async(name)
    end
  end
end




class TestWorkerCallback
  include Sidekiq::Worker
  include Sidekiq::Status::Worker
  sidekiq_options queue: 'default'

  def perform
    puts "You called TestWorkerCallback as CALLBACK!"
  end
end




class TestWorkerTwo
  include Sidekiq::Worker
  include Sidekiq::Status::Worker
  sidekiq_options queue: 'default'

  def perform(name)
    puts "You called TestWorkerTwo called: #{name}"
  end
end

After installation when try to use it show synatx error

gems/sidekiq-batch-0.1.6/lib/sidekiq/batch.rb:232: syntax error, unexpected '.' (SyntaxError)
cb_opts = callback_args.first&.at(2) || opts

Support for Sidekiq::BatchSet

Do you have plans to support Sidekiq::BatchSet API?
eg https://github.com/sidekiq/sidekiq/wiki/Batches#api

bs = Sidekiq::BatchSet.new
bs.each do |status|
  puts status.bid
end

Any plan to support join?

I'm wondering if you have any plan to support the join method on the status class?

command_builder should should allow SADD to pass an array as an argument

Using 0.1.9 when I add my job to the batch I get this error: Unsupported command argument type: Array because it passes the job ids to the SADD command as an array but the argument validation in command_builder does not allow this.

Fix is trivial. PR to follow.

Callback is triggered just after perform is call

I am trying to get the callback (success or complete) when the worker is full finished not just after it is called.

Are there any way to get it done?

I tried with Worker.perform_in(4.seconds,data)
and also in config sidekiq:
:batch_push_interval: 3

But Success or complete event is called just after the action is called.

I also request the status inside the complete function and inside it complete is false:
{
:bid => "XXXX",
:total => 0,
:failures => 0,
:pending => 0,
:created_at => nil,
:complete => false,
:failure_info => [],
:parent_bid => nil,
:child_count => 0
}

Thank you

Change API naming

Hi. I don't mind you reimplementing functionality similar to my commercial functionality. You'll find that you spend hundreds of hours implementing, tuning and supporting your users in return for no money. Such is the life of an open source developer.

What I do mind is that you copy my API and method names exactly, sowing confusing among developers who might see my documentation and think it applies to your gem or vice versa. Please do not use Sidekiq::Batch. You are welcome to call it "Sidekiq::JobGroup" or something else but Sidekiq::Batch is my API and I don't want this gem confusing Sidekiq users by providing something with the exact same name.

Callback implementation doesn't support class#method

Following the specification on https://github.com/mperham/sidekiq/wiki/Batches under the "Callbacks" header, there is the following example for callbacks.

batch.on(:complete, 'AnotherClass#method', 'uid' => current_user.id)

This behaviour does not seem supported by sidekiq-batch.

Deprecations in logs

After sidekiq update to 4.5.5, there is deprecation in console.
Redis#sadd will always return an Integer in Redis 5.0.0. Use Redis#sadd? instead.(called from: /usr/share/rvm/gems/ruby-2.7.6/gems/sidekiq-batch-0.1.8/lib/sidekiq/batch.rb:46:in block (2 levels) in on').
As I see, it is new problem and Sidekiq already merged PR with 4.5.6, that fixes their deprecations, but there is something from sidekiq-batch.

on_success callback is fired when some workers are still ongoing

For a few days, I'm facing a problem with success callback:

class ParentWorker
  include Sidekiq::Worker

  def perform
    batch = Sidekiq::Batch.new

    batch.on(:success, ParentWorker::Success)

    batch.jobs do
      ChildOneWorker.perform_async

      ChildTwoWorker.perform_async
    end
  end

  class Success
    def on_success(_status, _options)
      Rails.logger.info "On Success Callback"
    end
  end
end

class ChildOneWorker
  include Sidekiq::Worker

  def perform
    Rails.logger.info "ChildOne Start"

    a, b = 2, 4
    Rails.logger.info "a + b = #{a + b}" # simple and quick action

    Rails.logger.info "ChildOne Finish"
  end
end

class ChildTwoWorker
  include Sidekiq::Worker

  def perform(page = 1)
    Rails.logger.info "ChildTwo Start"

    # make API request and process data
    5.times do |n|
      Rails.logger.info "Page: #{page} - Second #{n}"
      sleep 1
    end

    # re-call when seconds page of data needs to be fetched
    self.class.perform_async(page + 1) if page < 2

    Rails.logger.info "ChildTwo Finish"
  end
end

Current behavior

ChildTwo Start
Page: 1 - Second 0
ChildOne Start
a + b = 6
ChildOne Finish
Page: 1 - Second 1
Page: 1 - Second 2
Page: 1 - Second 3
Page: 1 - Second 4
ChildTwo Finish
ChildTwo Start
Page: 2 - Second 0
On Success Callback
Page: 2 - Second 1
Page: 2 - Second 2
Page: 2 - Second 3
Page: 2 - Second 4
ChildTwo Finish

Expected behavior

ChildTwo Start
Page: 1 - Second 0
ChildOne Start
a + b = 6
ChildOne Finish
Page: 1 - Second 1
Page: 1 - Second 2
Page: 1 - Second 3
Page: 1 - Second 4
ChildTwo Finish
ChildTwo Start
Page: 2 - Second 0
Page: 2 - Second 1
Page: 2 - Second 2
Page: 2 - Second 3
Page: 2 - Second 4
ChildTwo Finish
On Success Callback

Is there any workaround for this?

Parameterize BID_EXPIRE_TTL to make it variable.

Redis memory usage is squeezing as BID_EXPIRE_TTL has been extended to 2_592_000 with v0.1.5 change.

v0.1.4...v0.1.5

We strongly desire be able to make it variable according to our use case.

on_success / on_complete does not trigger when jobs finishes too fast.

I have a pretty common code which was supposed to call the on_success method when all jobs get finished.

batch = Sidekiq::Batch.new
batch.on(:complete, Scheduling::SidekiqBatch::Callbacks)
   batch.jobs do
   salesmen.each do |salesman|
      company_id = salesman[0]
      salesman_id = salesman[1]
      
      ExportationWorker.perform_async({
         company_id: company_id,
         salesman_id: salesman_id
      })
   end
end

in general the ExportationWorker take some time to perform, but in some edge cases it finishes pretty quickly, in this case the on_success/on_complete is never called.

How to poll batch status ?

Hi guys,

I found in official documentation that endpoints for polling batch status are ready-to-use.
Just need to import and use the module:

The problem is that we don't have sidekiq pro (we are here because of that lol).

Is there any alternative module to require to use the polling with this gem ? Or should i make my own custom batch status endpoint ?

Thanks.

Success Callback calling before all jobs are complete

With more logging, I've been able to spot jobs executing after the on_success callback execution. I'm not sure what detail I can provider, but here we go. I am happy to provide more I'm just not sure what else would be helpful.

Redacted logs:

Nov 08 05:09:41pm info Starting batch b-5j1infmvnMQw
Nov 08 05:09:43pm info Running in batch b-5j1infmvnMQw
Nov 08 05:09:44pm info Running in batch b-5j1infmvnMQw
Nov 08 05:09:46pm info Running in batch b-5j1infmvnMQw
Nov 08 05:09:47pm info Running in batch b-5j1infmvnMQw
Nov 08 05:09:48pm info Completed batch b-5j1infmvnMQw
Nov 08 05:09:48pm info Successfully ran batch b-5j1infmvnMQw {"total":0,"failures":0,"pending":0,"created_at":null,"complete":false,"failure_info":[],"parent_bid":null}
Nov 08 05:10:00pm info Running  in batch b-5j1infmvnMQw
Nov 08 05:10:01pm info Running  in batch b-5j1infmvnMQw
Nov 08 05:10:04pm info Running  in batch b-5j1infmvnMQw

Thought it was strange the total field in the data was 0, testing this locally as well. The total is correct up until the on_success is called, it then goes to 0.

Code that creates the batch:

id = '1234' # fake example of parameters
batch = Sidekiq::Batch.new
batch.on(
  :success,
  Integration::MethodWorker,
  id: id
)
batch.on(:complete, MyWorker)
batch.callback_queue = 'queue'
batch

Workers enqueuing

batch.jobs
  MyWorker.perform_in(2.seconds)
  MyWorker.perform_in(3.seconds)
  MyWorker.perform_in(4.seconds)
  MyWorker.perform_in(5.seconds)
  MyWorker.perform_in(6.seconds)
  MyWorker.perform_in(7.seconds)
  MyWorker.perform_in(8.seconds) # there is an if statement on this (wouldn't think that matters)
en

Environment:

Ruby: 2.5.0
Rails: 5.1.4

Skips on_success callback for some batches

While running multiple batches simultaneously, on_success callback is being skipped for few batches.
In sidekiq logs, there are no logs saying DEBUG: Finalize success batch id: for the respective batches.
No issue noticed with on_complete callback.

`cleanup_redis` should not be called immediately after batch finishes

Relates to #17

It seems like when batches complete they call cleanup_redis, which removes all useful batch information from the store. This prevents you from seeing the number of failed jobs, whether the job was complete, etc. Is there a reason to do this immediately?

Success Callback always triggered for single failing job

Hey everyone. There is an issue when I create a batch and supply a single job to it. Let me supply some code for anyone to replicate:

# app/workers/create_cluster_worker.rb
class CreateClusterWorker

  def perform
    batch = Sidekiq::Batch.new
    batch.description = 'test'
    batch.on(:success, CreateClusterWorker::Created, { "cluster_id" => 999 })
    batch.on(:complete, CreateClusterWorker::Created, { "cluster_id" => 999 })

    batch.jobs do
      1.times { |i| CreateServerWorker.perform_async(i) }
    end
  end

  class Created
    def on_success(status, options)
      puts status, options
    end

    def on_complete(status, options)
      puts status, options
    end
  end
end

# app/workers/create_server_worker.rb
class CreateServerWorker
  include Sidekiq::Worker
  def perform(id)
    i = rand(1..10)
    sleep i
    raise StandardError unless 'a'.eql? 'b' # Deliberatly create a failure
  end
end

Executing CreateClusterWorker.new.perform from the rails console triggers the on_complete callback (which correctly passes status.failures to equal 1) but also triggers the on_success callback (which incorrecly passes status.failures to equal 0).

Environment

sidekiq-batch (0.1.4)
sidekiq (5.0.4)
redis (4.1.0)
ruby (2.4.4)
rails (5.2.2)

Disclaimer

#26 might fix the issue but given that the branch has merge conflicts I am unable to confirm. I wanted to create this issue just incase it is a slightly different edge case to the currently open issues.

Deprecation messages in the logs

It seems there are already two PRs (#52 and #53) for fixing this issue:

Pipelining commands on a Redis instance is deprecated and will be removed in Redis 5.0.0.

redis.multi do
redis.get("key")
end

should be replaced by

redis.multi do |pipeline|
pipeline.get("key")
end

Incorrect logic in valid? method

The valid? method on a batch seems to have incorrect logic.

I specifically ran into an issue invalidating batches since the valid? method always returns false due to an integer now being returned by Redis#exists(key). resque/redis-namespace#195

sidekiq-batch/lib/sidekiq/batch.rb

Line 133 in e1cdc03

valid = !Sidekiq.redis { |r| r.exists("invalidated-bid-#{batch.bid}") }

Seems like changing this line in the valid? method would work:

valid = !Sidekiq.redis { |r| r.exists("invalidated-bid-#{batch.bid}") }

to

valid = Sidekiq.redis { |r| r.exists?("invalidated-bid-#{batch.bid}") } > 0

I ran into this with the following example, working to invalidate jobs within a batch. Here's an example job:

class Imports::Payments::ChunkJob < Imports::ChunkBase
  include Sidekiq::Job

  def perform(opts)
    return unless valid_within_batch?
  end
end

I run the batch to get things started and I expect the batch to fully run, but valid_within_batch? always returns false. I threw a debugger in the job and found the following:

(ruby) Sidekiq.redis { |r| r.exists("invalidated-bid-#{batch.bid}") }
0 
(ruby) !Sidekiq.redis { |r| r.exists("invalidated-bid-#{batch.bid}") }
false 

rescue nil swallowing exceptions

https://github.com/breamware/sidekiq-batch/blob/master/lib/sidekiq/batch/callback.rb#L12 swallows all exceptions.

Ideally, this shouldn't be rescued, there by giving feedback to the developer

Web UI?

Any plans on support for this?

0.1.9 incompatible with redis-client which is used by sidekiq 7

Sidekiq 7 has switched to redis-client. redis-client does some validation on arguments passed to the redis commands.

In 0.1.9 this line https://github.com/breamware/sidekiq-batch/blob/master/lib/sidekiq/batch.rb#L99 was changed to place the @ready_to_queue jids in an array. This results in passing a nested array as an argument to the SADD command. This fails the redis-client validation check.

Their suggestion is to flatten the array.

Redis::CommandError: WRONGTYPE Operation against a key holding the wrong kind of value

any idea?

ruby 2.6.2, redis 5.0.5, sidekiq-batch (0.1.5), sidekiq (5.2.7), rails (5.2.3)

looks to be failing here:

Redis::CommandError: WRONGTYPE Operation against a key holding the wrong kind of value
from redis/pipeline.rb:156:in value' from redis/pipeline.rb:148:in _set'
from redis/pipeline.rb:75:in block in finish' from redis/pipeline.rb:74:in each'
from redis/pipeline.rb:74:in each_with_index' from redis/pipeline.rb:74:in each'
from redis/pipeline.rb:74:in map' from redis/pipeline.rb:74:in finish'
from redis/pipeline.rb:98:in finish' from redis/client.rb:164:in block in call_pipeline'
from redis/client.rb:306:in with_reconnect' from redis/client.rb:162:in call_pipeline'
from redis.rb:2462:in block in multi' from redis.rb:52:in block in synchronize'
from monitor.rb:230:in mon_synchronize' from redis.rb:52:in synchronize'
from redis.rb:2454:in multi' from sidekiq/batch/callback.rb:45:in block in complete'
from sidekiq.rb:97:in block in redis' from connection_pool.rb:65:in block (2 levels) in with'
from connection_pool.rb:64:in handle_interrupt' from connection_pool.rb:64:in block in with'
from connection_pool.rb:61:in handle_interrupt' from connection_pool.rb:61:in with'
from sidekiq.rb:94:in redis' from sidekiq/batch/callback.rb:44:in complete'
from sidekiq/batch/callback.rb:18:in perform' from sidekiq/processor.rb:192:in execute_job'
from sidekiq/processor.rb:165:in block (2 levels) in process' from sidekiq/middleware/chain.rb:128:in block in invoke'
from sidekiq/batch/middleware.rb:30:in call' from sidekiq/middleware/chain.rb:130:in block in invoke'
from raven/integrations/sidekiq.rb:9:in call' from sidekiq/middleware/chain.rb:130:in block in invoke'
from sidekiq/middleware/chain.rb:133:in invoke' from sidekiq/processor.rb:164:in block in process'
from sidekiq/processor.rb:137:in block (6 levels) in dispatch' from sidekiq/job_retry.rb:109:in local'
from sidekiq/processor.rb:136:in block (5 levels) in dispatch' from sidekiq/rails.rb:43:in block in call'
from active_support/execution_wrapper.rb:87:in wrap' from active_support/reloader.rb:73:in block in wrap'
from active_support/execution_wrapper.rb:87:in wrap' from active_support/reloader.rb:72:in wrap'
from sidekiq/rails.rb:42:in call' from sidekiq/processor.rb:132:in block (4 levels) in dispatch'
from sidekiq/processor.rb:250:in stats' from sidekiq/processor.rb:127:in block (3 levels) in dispatch'
from sidekiq/job_logger.rb:8:in call' from sidekiq/processor.rb:126:in block (2 levels) in dispatch'
from sidekiq/job_retry.rb:74:in global' from sidekiq/processor.rb:125:in block in dispatch'
from sidekiq/logging.rb:48:in with_context' from sidekiq/logging.rb:42:in with_job_hash_context'
from sidekiq/processor.rb:124:in dispatch' from sidekiq/processor.rb:163:in process'
from sidekiq/processor.rb:83:in process_one' from sidekiq/processor.rb:71:in run'
from sidekiq/util.rb:16:in watchdog' from sidekiq/util.rb:25:in block in safe_thread'

on_complete called before batched jobs finish running

I saw some similar issues reported where it seems that this should have been resolved #20 (resolved? in #26) for on_success callbacks but this seems to still be broken for on_complete. I started with ~25.5k jobs queued up within a batch and the on_complete method was called with ~7.5k jobs still queued up to run. Any thoughts?

Does it matter how many threads the batched jobs are running on?

Also noticed the BID on the callback job was different from the BID on the batched jobs, is that how it should be?

class DisclosureService
...
def trigger_send
  ...
  batch = Sidekiq::Batch.new
  batch.on(:complete, DisclosureService, disclosure_batch_id: batch_id)
  batch.jobs do
    accounts_to_disclose.each do |account|
      SendDisclosureJob.perform_async(account.account_id, batch_id)
    end
  end
end
...

def on_complete(status, options)
   ...
end
end

Add support for on_death callback

Sidekiq Pro has a death callback:

Sidekiq can notify you when a Batch is complete or successful with batch.on(event, klass, options={}):

1. complete - when all jobs in the batch have run once, successful or not.
2. success - when all jobs in the batch have completed successfully.
3. death - the first time a batch job dies

Regarding success, if a job fails continually it's possible the success event will never fire. If a job fails all retries and dies, it will fire any :death callbacks. The :death callback is always fired for first job in the batch that fails and does not retry. Even if you configure jobs to disable retries or job death, it will still fire the :death callback.

:death and :success are not mutually exclusive but the death callback firing means that the batch will not fire success without manual intervention. If you deploy a fix and manually re-enqueue a dead batch job, the batch can still fire :success.

But sidekiq-batch (lib/sidekiq/batch/callback.rb:8) silently fails if :death is passed.

Stubbing in rspec in tests

This gem uses Sidekiq.redis directly so but with the following snippet:

if Rails.env.test?
  # This is in here instead of rails_helper.rb so that E2E tests work as well.
  require 'sidekiq/testing'
  Sidekiq::Testing.fake!
end

There is no Redis client, it fails.

Any known workarounds?

Job never gets set to complete and stats get deleted

I've added a batch of jobs:

batch = Sidekiq::Batch.new
batch.on(:success, Publishing::Status)
batch.jobs do
  # enqueue jobs
end

But as it progresses and I check the status I get this:

# Just started
batch_status = Sidekiq::Batch::Status.new(batch.bid)
batch_status.data => {:bid=>"D9u_HoEX5AAA7A", :total=>37, :failures=>0, :pending=>37, :created_at=>"1670976746.5037887", :complete=>false, :failure_info=>[], :parent_bid=>nil, :child_count=>0}}
# Part wy through
batch_status = Sidekiq::Batch::Status.new(batch.bid)
batch_status.data => {:bid=>"D9u_HoEX5AAA7A", :total=>37, :failures=>0, :pending=>21, :created_at=>"1670976746.5037887", :complete=>false, :failure_info=>[], :parent_bid=>nil, :child_count=>0}}
# Completed (no errors)
batch_status = Sidekiq::Batch::Status.new(batch.bid)
batch_status.data => {:bid=>"D9u_HoEX5AAA7A", :total=>0, :failures=>0, :pending=>0, :created_at=>nil, :complete=>false, :failure_info=>[], :parent_bid=>nil, :child_count=>0}}

It seems that after the last batch finishes, it wipes out the counts and never sets the complete flag to true.

Any ideas of what could be going on here?

0.1.9 still incompatible with redis-client which is used by Sidekiq 7

The issue #66 is still happening, this was not fixed, but for some reason was marked as closed. The latest version of sidekiq-batch is not compatible with Sidekiq 7.

Extend batch expire time

The issue:
We have batches that have a lot of (throttled) jobs, therefore a batch can take several days to complete. What we saw with those batches is, that callbacks are not executed.

What I think the reason is:
As far as I can see the callback is stored in "BID-#{batch_id}-callback-complete", which is expiring after 30 hours (

sidekiq-batch/lib/sidekiq/batch.rb

Line 13 in 1e92bc2

BID_EXPIRE_TTL = 108_000

), whereas Sidekiq Pro Batch uses 30 days (https://github.com/mperham/sidekiq/wiki/Batches#expiration).

Currently we just override it with Sidekiq::Batch::BID_EXPIRE_TTL = 604_800. Is it possible to either increase the default value to match Sidekiq pros setting, or even make it configurable?

Thanks!

On complete is called while there are pending jobs

We do add more jobs to the batch from an existing job. Something like this:

batch = Sidekiq::Batch.new

batch.on(:complete, MyJob, 'arg' => 123)

batch.jobs do
  MyJob.perform_async
end

And then in MyJob we do:

def perform
  batch.jobs do
    AnotherJob.perform_async
  end
end

Jobs are being added to the queue and are reflected in batch #pending but #complete? returns true and on_complete callback is called in MyJob once the first job is finished.

As a workaround we have to schedule another job like this:

batch.jobs do
  MyJob.perform_async
  ExtraJob.perform_in(5.minutes)
end

Batch details:

(byebug) Site.find(options['site_id']).batch_status("pages").complete?
true
(byebug) Site.find(options['site_id']).batch_status("pages").pending
25
(byebug) Site.find(options['site_id']).batch_status("pages").data
{:bid=>"AsffjPpN7BwOKA", :total=>29, :failures=>0, :pending=>25, :created_at=>"1631141513.511842", :complete=>true, :failure_info=>[], :parent_bid=>nil, :child_count=>0}

Is this gem thread safe?

Looking to identify possible thread safety issues in a Rails 4.2 stack running Unicorn in a docker production environment with Sidekiq running our background jobs. Any info would be helpful, thanks.s

RSpec testing and Stats not updating

Here is my block of code:

it "has the jobs completed" do |example|
  b = Sidekiq::Batch.new
  b.jobs do
    3.times {Sidekiq::Worker.perform_async(Time.now.to_i,nil)}
  end
  Sidekiq::Worker.perform_one
  status = Sidekiq::Batch::Status.new(b.bid)
end

Here is my status output:
#<Sidekiq::Batch::Status:0x000000010c682808 @bid="4tR00OKDnnKWdA", @completed=0, @failures=0, @pending=3, @props={"created_at"=>"1675329197.732366", "callbacks"=>"{}", "pending"=>"3", "total"=>"3"}, @total=3>

For pending, it is still stuck at 3, where it should have been 2 from perform_one, where it will execute and finish the 1st job in the batch.

I have also used the flag stub_batches: false to use a real Sidekiq Worker to execute these dummy jobs.
May I know how is the Status for pending calculated?
Thank you!

String#constantize

Hi! Nice gem!

I came across a little problem when using callbacks. Sidekiq::Batch::Callback::Worker#perform makes use of ActiveSupport String#constantize method. For non-Rails apps for which ActiveSupport is not embedded by default, it crashes. I solved the issue by providing my own version of String#constantize.

We can either add an explicit dependency to ActiveSupport or directly provide a version of String#constantize.

Nested batches

I noticed in the "Projects" that this is a wanted feature.

Any chance of this being done soon? If not, could you point me in the right direction on how you would see this being developed? What the right approach should be?

The problem I'm trying to solve is as follows:

WorkflowWorker will spawn 1 overall batch, which will spawn X batches (from database, so don't know how many).
Each batch will create a new batch of 1 worker, which when finished (callback), will start a new batch with X workers (from database, so don't know how many).

When everything's finished, the WorkflowWorker overall batch should call a callback, spawning a final worker.

So, basically. If this implementation succeeds, I would be able to do what I want.

Question about status on callbacks

Hi,

I'm starting with sidekiq-batch and I'd like to know if the result I'm getting is really expected/intended:

Whenever I finish a job batch when I see the status object that I receive both on on_success and on_complete I see they always have empty values when everything works as it should.
{:total=>0, :failures=>0, :pending=>0, :created_at=>nil, :complete=>false, :failure_info=>[], :parent_bid=>nil}
This is one example from a Batch that had a single job that was executed successfully. I was expecting to see total 1 and complete true but that didn't happen.

Whenever I have a batch containing one job that failed I see the info correctly set though.

Is this really intended ? Because I was expecting to be able to log such information on database upon completion but apparently I can't rely on this object I receive on the callback.

Thanks

Doesn't seem to call the on_success callback during tests with Sidekiq::Testing.inline!

I'm using Sidekiq::Testing.inline! to run all the jobs inline during tests. I ran the test, and the on_success method is not being called. But the batch jobs are all completing successfully.

Does this gem support Sidekiq::Testing.inline!? Or should I just fire the on_success callback manually?

Error: Unsupported command argument type: TrueClass

When running some jobs, I'm getting this error:

Unsupported command argument type: TrueClass

It comes from:

sidekiq-batch/lib/sidekiq/batch.rb

Line 212 in 663debd

pipeline.hset(batch_key, event_name, true)

This is happening on sidekiq-batch 0.1.9, with sidekiq 6.4.2 and redis-rb 5.0.7, which uses redis-client 0.17.0

This happens because redis-client only accepts Strings, Symbols and Numerics as parameters. See:

https://github.com/redis-rb/redis-client/blob/1ab081c1d0e47df5d55e011c9390c70b2eef6731/lib/redis_client/command_builder.rb#L28

In the line I pasted above you can see sidekiq-batch always sends true. I can solve the problem locally by using true.to_s. So the result is like this:

pipeline.hset(batch_key, event_name, true.to_s)

It seems pretty trivial to fix. I could send a PR myself but first would like to hear your comments, just in case I'm doing something wrong.