toptal / active-job-style-guide Goto Github PK
View Code? Open in Web Editor NEWThis Background Jobs style guide is a list of best practices working with Ruby background jobs.
This Background Jobs style guide is a list of best practices working with Ruby background jobs.
This is a bag of thoughts issue, I encourage everyone to file more specific detailed issues and send pull requests.
do not use keyword arguments in plain Sidekiq workers sidekiq/sidekiq#2372 (check if it works or not with ActiveJob)
do not schedule jobs inside a transaction if a record created/updated inside this transaction will be directly or indirectly accessed by a job: this comes with a risk of not finding a record, or getting a record with non-updated data in case job is performed before the transaction is committed. Use Sidekiq::Postpone.wrap
around each transaction do
https://github.com/palkan/isolator can be used to detect jobs scheduled from inside a transaction
avoid setting the queue through sidekiq_options
explain the difference between bucket and window
[ ] warn on the use
[ ] make sure examples are right
extend default limiter backoff.
it's in minutes, this is not so effective if you want to run jobs ASAP (seconds), as it re-schedules,
e.g. puts jobs after all other jobs. Do not set it to too low (e.g. 1s), otherwise overhead will start kicking in
[ ] It'd be nice if there was more concrete argumentation for when you need the commercial features
limiter policy: :skip not to raise
write about lack of concurrency on cpu-intensive jobshttps://www.mikeperham.com/2019/07/19/something-for-nothing/
think of a better way of dealing with limiter backoff, keeping all of them in a single file is weird (#default-limiter-backoff guideline)
mention that in Sidekiq-ent 2.0 it can be per-limiter
race condition in Business Logic in Jobs (stale data, the job should not be performed) - same problem without transactions, and also with, without locking)
avoid using unique jobs?
it's mentioned sidekiq/sidekiq@40cc629#diff-0469ba29524ee01a49be37ff05e1f499R12
precaution - don't use queue
settings in sidekiq_options
serialization: only simple objects or what ActiveJob supports. custom serializers
what happens if the record is removed and can't be deserialized. use discard_on ActiveRecord::RecordNotFound
discard_on ActiveJob::DeserializationError
in ApplicationJob
?
[ ] how does discard_on work with batches
adopt the style suggested by Dan Allen - have separate good and bad code examples with explanations in their captions.
https://asciidoctor.org/docs/asciidoc-syntax-quick-reference/#source-code
? ALLOW_CONCURRENCY=true vs RAILS_ENV=production in local testing, nested concurrent limiters misbehave, sometimes neither works. prefer to use EAGER_LOAD=true?
emphasize on the distinction between jobs that don't need automatic retries and ephemeral jobs
The retry property can be set on a worker or specific job to disable retries completely (job goes straight to Dead) or disable death (the failed job is simply discarded). If your Failed count is increasing but you don't see anything in the Retry or Dead tabs, it's likely you've disabled one or both of those:
class SomeWorker
# will be completely ephemeral, not in Retry or Dead
sidekiq_options retry: false
# will go immediately to the Dead tab upon the first failure
sidekiq_options retry: 0
mention limiters for a long-running one-off in addition to batches
improve limiter backoff guideline and provide an example of how:
a limiter can be shared across jobs
a limiter can be locally specified (injected?)
a limiter is something-specific, e.g. user_id
mention leaky bucket rate limiter
? set lock_timeout to longer than average job execution time to prevent premature lock release
lock_timeout option ensures a crashed Ruby process does not hold a lock forever. You must ensure that your operations take less than this number of seconds
The same Limiter (based on the name) may be used with different lock_timeout values which allow for different blocks of code to lock on the same resource with a different lock_timeout.
track your errors in error tracking software
infra: use persistent, reliable, dedicated redis (sentinels, ...)
https://github.com/mperham/sidekiq/wiki/Using-Redis
don't use Process.clock_gettime(Process::CLOCK_MONOTONIC) to share data across processes/machines
sidekiq/sidekiq#4105
timeout (includes Net::HTTP and libraries using it, provide a reference to its source)
Don't ever use Ruby's Timeout module. You will get mysterious stuck or hung processes randomly.
http://www.mikeperham.com/2015/05/08/timeout-rubys-most-dangerous-api
https://github.com/mperham/sidekiq/wiki/Problems-and-Troubleshooting#frozen-processes
net:http based: httparty, restclient, open-uri https://github.com/httprb/http#another-ruby-http-library-why-should-i-care
exception to pass by id for external ids - harmonize with cop (salesforce_external_id)
uncached
+# In Rails 5, ActiveJob execution is wrapped with code reloader, which
+# as one of its callbacks enables query cache. There are some background
+# jobs (e.g. {MidasReportJob}) that traverse many records and do not create
+# any. In such cases, the query cache is being filled with data that is not
+# being used ever again and it is huge. In Rails 4.2, jobs were running
+# without query cache and we didn't notice any slowdowns or database
+# overloads. The disabling cache will prevent from creating such cache
+# and will make job server use much less memory.
+#
class ApplicationJob < ActiveJob::Base
extend Memoist
+
block.call
recommend MALLOC_ARENA_MAX=2
Draper: Active Job Integration. Active Job allows you to pass ActiveRecord objects to background tasks directly and performs the necessary serialization and deserialization.
In order to do this, arguments to a background job must implement Global ID.
Decorated objects implement Global ID by delegating to the object they are decorating.
This means you can pass decorated objects to background jobs, however, the object won't be decorated when it is deserialized.
extend renaming jobs guideline regarding the changes to perform
signatures, there can be scheduled jobs with old signature serialized
avoid changing mailers' signatures just like regular job's signatures
avoid using Redis namespaces if you keep data for several different Sidekiq instances in one Redis instance. (check if advanced Sidekiq features using Lua can interfere and even respect namespaces when used with e.g. redis-namespace
)
when using Reliable Push, keep Redis timeouts (connection pool's network_timeout
, connection pool's number of connections, redis
's timeout
, connect_timeout
, read_timeout
, write_timeout
, reconnect_attempts
, reconnect_delay
, reconnect_delay_max
) to the minimum. Doing otherwise would result in 30s+ pushes when Redis is down/degraded
don't rely on queue weights for job prioritization
make sure to add mailers
to Sidekiq's configuration when using it with Active Job's deliver_later
use custom DeliveryJob to separate mailers between queues with different priorities
config: how to set up different pool sizes for Rails and Sidekiq processes (per-process RAILS_MAX_THREADS
, and pool
in config/database.yml
for Rails and Sidekiq, ...)
Thanks for this styleguide. I learnt a few good practices and sidekiq's features.
Does rubocop extension corresponding this style-guide already exist somewhere? I could not find any mentioned in this guide, nor in the rubocop docs.
I am planning to create such extension. Not all rules can be checked by rubocop, but some sure can and that would be helpful.
I thought of checking with maintainers here before creating the new extension to avoid duplicate work.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.