hired/reactor

A Sidekiq-backed pub/sub layer for your Rails app.

reactor.gem

A Sidekiq-backed pub/sub layer for your Rails app.

This gem aims to provide the following tools to augment your ActiveRecord & Sidekiq stack.

1. Barebones event API through Sidekiq to publish whatever you want
2. Database-driven API to manage subscribers so that users may rewire whatever you let them (transactional emails, campaigns, etc...)
3. Static/Code-driven API to subscribe a basic ruby block to an event.
4. A new communication pattern between your ActiveRecord models that runs asynchronously through Sidekiq. a. describe model lifecycle events and callbacks with class-level helper methods/DSL

Installation

Add this line to your application's Gemfile:

gem 'reactor'

And then execute:

$ bundle

Or install it yourself as:

$ gem install reactor

Usage

Well, this is evolving, so it's probably best to go read the specs as well as below.

Barebones API

Reactor::Event.publish(:event_name, any: 'data', you: 'want')

Publishable

Describe lifecycle events like so

publishes :my_model_created

Schedule an event to get published at a specific time. Note: if timestamp is a property on an ActiveRecord::Model then updating that property will re-schedule the firing of the event

publishes :something_happened, at: :timestamp

Schedule an event to get published at a specific time using a method to generate the timestamp and following some other property. In this case the :something_happened event will be fired 72 hours after your model is created. The event will be re-scheduled if created_at is changed.

def reminder_email_time
  created_at + 72.hours
end

publishes :reminder_sent, at: :reminder_email_time, watch: :created_at

Scheduled events can check conditionally fire -- eg: in 2 days fire reminder_email if the user hasn't already responded. Note that this check will occur at the time the event is intended to fire, after which the state of the model may have changed.

publishes :reminder_sent, at: :reminder_email_time, if: -> { user.responded == false }

It is also possible to conditionally trigger events so that the check happens within the context of the model, at the moment an update occurs, rather than later in the context of the Reactor::Event

publishes :model_changed_in_some_important_way, enqueue_if: -> { important_change_occurred? }

Subscribable

You can now bind any block to an event in your models like so

on_event :any_event do |event|
  event.target.do_something_about_it!
end

Static subscribers like these are automatically placed into Sidekiq and executed in the background.

You may also have Sidekiq process a subscriber block on a specific queue or supply any other Sidekiq::Worker options accordingly.

on_event :event_with_ui_bound, sidekiq_options: { queue: 'highest_priority' } do |event|
  speedily_execute!
end

Automatic Events

If you'd like to have events automatically fired for you around standard rails resource controller actions, you may want to write your own downstream abstraction for it. We attempted this once and it seems unwise to presume your application would want the same thing.

It probably only makes sense if you have a real Magestic Monolith and an intentionally small team because coupling resource names to event streams makes refactoring harder. (Though it may still be worth it for you, depending on your needs!)

Testing

Calling Reactor.test_mode! enables test mode. (You should call this as early as possible, before your subscriber classes are declared). In test mode, no subscribers will fire unless they are specifically enabled, which can be accomplished by calling

Reactor.enable_test_mode_subscriber(MyAwesomeSubscriberClass)

We also provide

Reactor.with_subscriber_enabled(MyClass) do
  # stuff
end

for your testing convenience.

Matchers

You can clean up some event assertions with these somewhat imperfect matchers.

# DRY up strict event & data assertions.
expect { some_thing }.to publish_event(:some_event, actor: this_user, target: this_object)

# DRY up multi-event assertions. Unfortunately can't test key-values with this at the moment.
expect { some_thing }.to publish_events(:some_event, :another_event)

Production Deployments

TLDR; Everything is a Sidekiq::Worker, so all the same gotchas apply with regard to removing & renaming jobs that may have a live reference sitting in the queue. (AKA, you'll start seeing 'const undefined' exceptions when the job gets picked up if you've already deleted/renamed the job code.)

Adding Events and Subscribers

This is as easy as write + deploy. Of course your events getting fired won't have a subscriber pick them up until the new subscriber code is deployed in your sidekiq instances, but that's not too surprising.

Validating Events On Publish

As of 1.0 you may inject your own validator lambda to handle the logic and flow-control of valid/invalid events.

This is entirely optional and the default behavior is to do nothing, to not validate any data being provided.

# in config/initializers/reactor.rb
Reactor.validator -> do |event|
  Activity.build_from_event(event).validate! # you own the performance implications here
end

We at Hired use this to validate the event's schema as we found having stricter schema definitions gave us more leverage as our team grew.

By injecting your own logic, you can be as permissive or strict as you want. (Throw exceptions if you want, even.)

Removing Events and Subscribers

Removing an event is as simple as deleting the line of code that publishes it. Removing a subscriber requires awareness of basic Sidekiq principles.

Is the subscriber that you're deleting virtually guaranteed to have a worker for it sitting in the queue when your deletion is deployed?

If yes -> deprecate your subscriber first to ensure there are no references left in Redis. This will prevent Reactor from enqueuing more workers for it and make it safe for you delete in a secondry deploy.

on_event :high_frequency_event, :do_something, deprecated: true

If no -> you can probably just delete the subscriber. In the worst case scenario, you get some background exceptions for a job you didn't intend to have run anyway. Pick your poison.

Managing Queues

There will likely be more queue theory here later, but for now here are the features.

Everything is done on Sidekiq default queue by default.

Subscribers can opt into certain queues with on_event :whatever, sidekiq_options: { queue: 'whatever' } argument.

You can also override all queue choices with ENV['REACTOR_QUEUE']. You may want to do this if you wish to contain the 'cascade' of events for more expensive or risky operations.

Executing in a Console

By default, running a Rails console in a production ENV['RACK_ENV'] will cause publish events to bomb out unless srsly: true is provided as an additional parameter to event publishing. To control this behavior, set ENV['REACTOR_CONSOLE_ENABLED'] to a value.

Contributing

1. Fork it
2. Create your feature/fix branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create new Pull Request

For testing Reactor itself we use Thoughtbot's appraisal gem. This lets us test against multiple versions of Sidekiq, Rails, etc. To install appraisal and set up multiple dependencies, do the following:

1. bundle install - this will install up-to-date dependencies and appraisal
2. appraisal install - installs dependencies for appraisal groups
3. appraisal rake - runs specs for each appraisal group

Open Source by Hired

We are Ruby developers ourselves, and we use all of our open source projects in production. We always encourge forks, pull requests, and issues. Get in touch with the Hired Engineering team at [email protected].

Releasing

If you are a gem maintainer, you can build and release this gem with:

$ bundle exec rake build
$ gem push pkg/reactor(version).gem

reactor's Issues

can't supply a 'data' key to an event

Reactor::Event.publish(:some_event, with: 'some', data: {stuffed_in: 'here'})

It appears that data key will collide with the internal implementation that uses an instance variable named @data and allows loose access to setters/getters on it through method_missing.

The end result is that you'll overwrite key/values previously defined in the event data packet when using the data key.

Surely we can just obfuscate the internal implementation a little more or something.

Break readme up into wiki pages

It's reached critical mass and should be converted to a link to the wiki page.

Each section on the readme will be a new page on the wiki, and ideally a nice "Getting Started" page will start the whole thing with a bang.

prevent registration of subscribers with no block

We should just throw an argument error if no block is provided to on_event.

Small but important quality of life thing for teams of any post-market-fit size.

#deliver_now is never a function on Mail::Message

reactor/lib/reactor/workers/mailer_worker.rb

Lines 26 to 29 in dd8a15a

	if msg.respond_to?(:deliver_now)
	# Rails 4.2/5.0
	msg.deliver_now
	else

I was stepping through this code, on rails 5.0.4, to debug an issue on my branch, and notice that Mail::Message never implements deliver_now only deliver. So this switch is unnecessary and misleading.

introduce ActionMailer mixin

The theory is tested in production and quite sound, we just need to add this to reactor.

class MyMailer < ActionMailer::Base
  include Reactor::EventMailer

  on_event :user_created do |event|
    user = event.actor
    mail to: user.email, subject: "Welcome to the app!" 
  end
end

It works as you'd expect, simply extending the functionality of a typical ActionMailer method such that it is also automatically run in a sidekiq job on that event name.

Reschedule Functionality Has Highly Variable Performance

Based on how many items might be in the scheduled set for Sidekiq, scanning over them all can end up being extremely expensive. We see average times of 3+ seconds in web requests using the publishes .... at: feature that automatically reschedules.

In short, we use functionality documented here: https://github.com/mperham/sidekiq/wiki/API , but ignored this warning:

You should not be scanning through queues/sets and deleting jobs unless something has gone wrong and you need to repair data manually.

Maintaining the functionality by allowing duplicate self-destructing scheduled jobs might be possible and avoid the performance issues.

RFC: UUIDs on events

It would be nice is Reactor generated and added UUIDs on events.

We're generating events and sending them off to two destinations: Postgres and Kinesis.

For isolation reasons, this is being handled by two different * listeners.

It might be good to have an identifier i.e. UUID to match events across the systems.

We can't really generate the same UUID in both listeners, if only because timestamps will likely differ.

So it would be nice if Reactor generated the UUID before the event reaches subscribers.

Reactor could skip generating the UUID if the event already has a uuid key.

Thoughts invited!

document `action_event`

It's undocumented... so when you go looking for where events might come from, you don't know to look for that one!

Reactor "future-publishing-jobs"

I'm moving a discussion started by @dashkb from a hired/hired issue so the discussion can live here in public. He has some really good ideas about how to reduce the complexities/opinions we recently thrust into reactor.

=== begin @dashkb quote ===

An idea I had while reading #995 that ended up getting kind of long and shouldn't jam up that PR.

All problems in computer science can be solved by another level of indirection, except of course for the problem of too many indirections. -- David Wheeler

We want a way to publish an event when the event actually happens, which is going to be in the future. Some concerns we have are:

• The time at which the event should fire, available to us now, may change, and so a job scheduled to fire an event at that moment may sometimes require rescheduling. Timestamps like this are typically stored conveniently in columns named things like start_at, end_at, expire_at, explode_at, etc.
• We have some guard code we want to use at the last possible second to decide whether to fire the event at all.

I imagine this API:

class SomeModel
  # this model has `start_at` and `end_at` columns
  at_time :start do
    publish :whatever if things_are_still_cool?
  end

  at_time :end do
    if relation.state.acceptable?
      do_things!
    end
  end
end

This could be accomplished with a recurring/polling job which monitors all of the timestamp columns that have callbacks like these registered, and when their values move into the past, execute the block.

• To reschedule a future-publishing job you just update the *_at column
• None of your state checking code runs until the time
• You don't even have to fire an event, or you can choose what events to fire conditionally, or whatever.

Test Helper Improvement: Reschedule

For events that can be rescheduled (Publishable::publishes :at), we should allow a test helper to verify the events are rescheduled.

robust rspec matchers

Neither of the following should cause a reactor matcher to fail (but they currently do)

1. adding a second event to the test lifecycle
2. changing the order in which events are fired in the test lifecycle

It only really works well when asserting that just one event was fired when indeed, only one event is fired.

It fails incorrectly when a second event (whether or not it's related) is also fired within the test's covered code. (Because it is only matching that one event gets called with a .should_receive.) For example, check this snippet from one of our tests where we had to ensure that a specific event was fired (even though other events were being fired in the context).

Ideally Reactor will provide more choices in the style of event assertions being made. Perhaps a "stub" vs "assert" pattern applied here like they do in the webmock gem, for example, would help alleviate this pain. It should be up to the consuming developer to decide if they should just stub an event (or any) at the beginning of the test (like they would stub a web request) or if they should make some kind of stronger assertion along the lines of "this event will fire"

Historically a "publishes_events" multi-event matcher was introduced, but there is still a tight coupling between the tests and the code because we have to be explicit about each event that is fired. (So when we add another event somewhere, we may have to go update the assertions to include it, lame.)

dynamically named worker classes can cause inconsistency

For some context, see discussion in https://github.com/hired/hired/pull/5691. Some of that reposted here.

Reactor dynamically defines ruby classes for each subscriber class based on the event name, or in the case of a wildcard event, WildcardHandlerX where X is an iterator. When the wildcard subscriber was deleted, we now have 4 WildcardHandlers instead of 5, so the class named WildcardHandler4 is no longer defined. It appears that there were a number of jobs in the Sidekiq retry queue referencing WildcardHandler4 that are now erroring upon deserialization due to the undefined class.

Thinking about this, I realized that this is potentially a huge bug or design flaw in Reactor. In most cases it should be fine to dynamically name subscriber classes as such, assuming the jobs process in near real-time. However, say that there are 5 wildcard handlers and one of them fails, the failing job will be sent into the retry queue with the dynamic class name WildcardHandlerX. Now, there's no guarantee that on the next code deploy that the subscribers defined will be loaded an named in the same order, so by the time this job retries, WildcardHandlerX may actually contain different functionality than it did when originally processed. This leaves open the possibility of some subscribers executing twice with the same data, and some subscribers possibly being skipped.

Also, this is related to another gripe of mine about reactor is that it's hard to tell from looking at the Sidekiq interface which subscribers are running, because they are named after the events, not the subscriber.

The proposed solution is to name subscribers after their subscriber class name.

changing dependencies from sidekiq to activejob

we made this thing work in rails 4, but we didn't really consider cleaning up the API.

i probably don't have to explain the benefits of depending on an adapter instead of a specific background job implementation.

primarily, i would love to be able to use this in side projects where I don't want to pay for a background worker just yet, but do want the same code patterns

i've started looking at all of the dependencies/bindings we have thus far, and it looks non-trivial to accomplish this kind of change, so I wanted to start the conversation here before gutting a bunch of shit.

Benefits

• uses new rails GlobalID instead of serializing activerecord objects as *_type and *_id parameters.
• allows starter projects to avoid paying for worker processes until needed
• major incentive to perform feature-removal & decomposition/simplification of gem responsibilities (something we've deprioritized lately)

Costs

• major version release / breaking changes
• Unfortunately I'm pretty sure we have to consider completing #21 before doing this because rescheduling just isn't really a concern of background jobs.
• will require a careful / delicate migration strategy for production setup

I'll begin considering this against other priorities at Hired, as a case can be made for the simplification of our toolset and removal of a potential time-bomb sitting in #21.

Defining 1.0

This has been in production for so long... and by semantic versioning standards it should have been 1.0 a long time ago.

So I'm taking time to define what 1.0 means to me, and it's essentially this:

Reducing Bloat To A Discrete Surface Area

Rather than try to make Reactor an opinionated set of conventions (that historically have been just plain wrong), I'd like to remove anything that isn't just "easy pub sub workers". If an application wants to add conventions on top of Reactor, it's all quite possible to do at the application layer. If someone wants to share open source conventions on top of this, make a gem!

A reactor ecosystem of opt-in libraries is going to be far stronger than a bloated mess.

So with that philosophy/direction, I'd like to do the following

• Reactor 1.0 Roadmap
  • let subscribers pass through sidekiq options to choose their queue, retry, etc (see #68)
  • kill database subscriber feature (see #69)
  • rebase (and later merge) #30 - production console publish confirmations
  • fix fired_at value's semantics (see #35)
  • kill in_memory feature (see #71)
  • kill controller/concerns, if someone wants to extract it to a gem go for it. (see #72)
  • apps should decide where to include Reactor::Subscribable & Publishable(see #64)
• Stretch Goals
  • kill test_mode and let applications use basic expect()/allow() syntax

Upgrade Guide For Users

• drop_table :subscribers, or keep it and write your own static * subscriber that uses it.
• on that note, pull any of the above removed features downstream into your app as needed, make a reactor_* gem and.
• update ActivityLogger-like classes to use published_at instead of fired_at, which is now written at the appropriate time.
• handle on_event sidekiq wrapper job migration detailed in #73
• include Reactor::Publishable and Reactor::Subscribable on ActiveRecord::Base or ApplicationRecord, or wherever you want it now if you don't want it there any more. (see #64)

publishes implementation firing erroneously

With contract time cards, I used publishes as seen below:

publishes :auto_approve_time_card, actor: :time_approver, target: :self,
    approval_time: -> { auto_approval_time.to_s }, at: :auto_approval_time, watch: :submitted_at,
    if: -> { !submitted_at.nil? && status == :pending_review_by_employer }

All time cards start out with submitted_at being nil and status being something different, however the event publishes multiple times before the conditions are met... I think this is because until the time card is submitted, auto_approval_time returns nil, and this causes it to fire without checking the if: condition... @wnadeau and I were looking into this at one point (it only happens in production for some reason) and weren't really getting anywhere, but it's clearly a bug

stub out tests correctly

What is this Reactor.test_mode! nonsense? Isn't it an anti-pattern to have test mode behavior leak out into the primary implementation?

Why can't we just
allow(Reactor::Event).to receive(:publish).with(anything, anything)
in spec_helper.rb and have individual unit tests opt-in to specific events with
allow(Reactor::Event).to receive(:publish).with(:some_event, hash_including(whatever: 'stuff')).and_call_original.

Maybe there was a good reason we didn't do this to begin with... been too long, can't remember. Seems weird and worth logging here to sleep on it though.