activesphere/exq-scheduler

Job scheduler for Exq

Exq Scheduler

Exq Scheduler is a cron like job scheduler for Exq, it's also compatible with Sidekiq and Resque.

Installation

defp deps do
  [{:exq_scheduler, "~> x.x.x"}]
end

Overview

Exq Scheduler pushes jobs into the queue at intervals specified by the schedule configuration. It is designed to run on more than one machine for redundancy without causing duplicate jobs to be scheduled.

Configuration

By default Exq Scheduler will read the configuration from application environment.

Storage

Exq Scheduler uses redis to store internal state of the scheduler. It uses "#{exq_namespace}:sidekiq-scheduler" for storing scheduler internal metadata.

config :exq_scheduler, :storage,
  exq_namespace: "exq" # exq redis namespace
  json_serializer: Jason # or Poison, which is the default if not provided

Redis Client

Exq Scheduler will start a Redis Client under it's supervisor tree. The name used in the child_spec and config should be the same.

config :exq_scheduler, :redis,
  name: ExqScheduler.Redis.Client,
  child_spec: {Redix, [host: "127.0.0.1", port: 6379, name: ExqScheduler.Redis.Client]}

NOTE: The child_spec API provided by Redix library has changed over time. Refer Redix.child_spec documentation of the specific version you use.

Schedules

config :exq_scheduler, :schedules,
  signup_report: %{
    description: "Send the list of newly signed up users to admin",
    cron: "0 * * * *",
    class: "SignUpReportWorker",
    include_metadata: true,
    args: [],
    queue: "default"
  },
  login_report: %{
    cron: "0 * * * *",
    class: "LoginReportWorker"
  }

• cron: required Refer cron documentation for syntax. Time zone of a single schedule can be changed by specifying the time zone at the end. Example 0 * * * * Asia/Kolkata.

• class: required Name of the worker class.

• queue: Name of the worker queue. Defaults to "default".

• args: List of values that should be passed to perform method in worker. Defaults to [].

• retry: Number of times Exq should retry the job if it fails. If set to true, Exq will use max_retries instead. Defaults to true.

• enabled: Schedule is enabled if set to true. Defaults to true. Note: if this config value is set, on restart it will override the any previous value set via Sidekiq web UI. Don't use this option if you want to enable/disable via Sidekiq web UI.

• include_metadata: If set to true, the schedule time in unix time format (example {"scheduled_at"=>1527750039.080837}) will be passed as an extra argument to perform method in worker. Defaults to nil.

• description: a text that will be shown in sidekiq web

Misc

Scheduling each and every job at the exact time might not be possible every time. The node might get restarted, the process might get descheduled by the OS etc. To solve this exq scheduler by default schedules any missed jobs in the last 3 hour. This interval can be configured by changing missed_jobs_window value. Note: this config value should be more than 3 hour to handle daylight saving properly

config :exq_scheduler,
  missed_jobs_window: 3 * 60 * 60 * 1000,
  time_zone: "Asia/Kolkata"

• missed_jobs_window: Missed jobs interval in milliseconds. Defaults to 3 * 60 * 60 * 1000

• time_zone: Default time zone for all schedules. Defaults to system time zone.

The scheduler will start by default when the app is started, to disable set start_on_application to false.

config :exq_scheduler,
  start_on_application: false

Web

Exq Scheduler is compatible with sidekiq-scheduler web UI. Make sure the exq_namespace value and the namespace in sidekiq are same.

Example

A Sample Mix project along with sidekiq web UI is avaialbe at demo directory to demonstrate the configuration. Sidekiq web interface requires Ruby to be installed.

To install dependencies

> cd demo
> mix deps.get
> cd sidekiq-ui
> bundle install

To start it

> cd demo
> ./start_demo.sh

Running tests

1. Install sidekiq (requires ruby 2.5.1 and bundler 1.16.1)

    $ (cd sidekiq && gem install bundler:1.16.1 && bundle install)
   

2. Setup services

    $ docker-compose up
   

3. Install dependencies

    $ mix deps.get
   

4. Run tests

    $ mix test
   

exq-scheduler's Issues

Schedule does not respect DST

I have the following config:

config :exq_scheduler,
  time_zone: "Europe/Copenhagen"

config :exq_scheduler, :schedules,
  generate_new_game: %{
    description: "Example",
    cron: "0 0 * * *",
    class: "MayApp.NightlyWorker",
    queue: "example",
    retry: false
  }

I would expect that to enqueue the NightlyWorker job at midnight local time (Copenhagen). However looking in my server log, it appears that the job was scheduled an hour late at 1 in the night instead:

Apr 07 01:00:00.086am info app web.1 Enqueued a job class: "MayApp.NightlyWorker" queue: "example" enqueued_at: 1554591600 args: []

So it appears that the schedule does not respect daylight savings, which started in Denmark on the night between March 30 and 31.

Here are my dependencies:

...
{:exq, "~> 0.13.2"},
{:exq_ui, "~> 0.10.0"},
{:exq_scheduler, "~> 0.6.2"},
{:tzdata, "~> 1.0.0-rc.1", override: true}

Notice that I've forcefully included the latest version of tzdata in hopes that this would solve the issue, since the previous version had trouble downloading the latest timezone database.

Can't run unit tests: Could not start application exq_scheduler

I get the following error when trying to run the tests:

** (Mix) Could not start application exq_scheduler: exited in: ExqScheduler.start(:normal, [])
    ** (EXIT) an exception was raised:
        ** (Protocol.UndefinedError) protocol Enumerable not implemented for nil of type Atom. This protocol is implemented for the following type(s): Ecto.Adapters.SQL.Stream, Postgrex.Stream, DBConnection.Stream, DBConnection.PrepareStream, Timex.Interval, Floki.HTMLTree, Scrivener.Page, Jason.OrderedObject, HashSet, Range, Map, Function, List, Stream, Date.Range, HashDict, GenEvent.Stream, MapSet, File.Stream, IO.Stream
            (elixir 1.11.2) lib/enum.ex:1: Enumerable.impl_for!/1
            (elixir 1.11.2) lib/enum.ex:196: Enumerable.slice/1
            (elixir 1.11.2) lib/enum.ex:819: Enum.empty?/1
            (exq_scheduler 1.0.0) lib/exq_scheduler.ex:21: ExqScheduler.start_link/1
            (kernel 7.1) application_master.erl:277: :application_master.start_it_old/4

I've added Exq.Mock.start_link(mode: :fake) to test_helper.exs, but I'm not being able to mock the scheduler as well.

Problem starting the scheduler in a separate node

I may be missing something here, but the following is happening to me: In my Phoenix web application, I'm trying to run exq and exq_scheduler on a separate worker node only, not the main web one, so I've set start_on_application: false in both configurations, this is the config.exs:

config :exq,
  name: Exq,
  host: "127.0.0.1",
  port: 6379,
  namespace: "exq",
  concurrency: 3,
  queues: ["default"],
  start_on_application: false,
  redis_timeout: 5000

config :exq_scheduler,
  storage: %{
    name: :exq_scheduler,
    exq_namespace: "exq",
    json_serializer: Jason
  },
  redis: %{
    name: ExqScheduler.Redis.Client,
    child_spec: {Redix, [host: "127.0.0.1", port: 6379, name: ExqScheduler.Redis.Client]}
  },
  schedules: %{
    test1: %{
      description: "Test",
      cron: "* * * * *",
      class: "Workers.Test",
      args: ["test"],
      queue: "default"
    },
    test2: %{
      description: "Test",
      cron: "*/2 * * * *",
      class: "Workers.Test",
      args: ["test2"],
      queue: "default"
    }
  },
  start_on_application: false

And this is in the application.ex start/2: worker_only_children = [Exq, ExqScheduler].

But I keep getting this error:

Now, if I put this in the application.ex:

    worker_only_children = [
      Exq,
      %{
        id: :exq_scheduler,
        start: {ExqScheduler, :start_link, [[schedules: []]]}
      }
    ]

The worker node successfully starts, so it seems to me like the exq_scheduler is not getting its config values when I try to manually start it with the application start. Does anyone know what could be causing this? Am I missing any config?

compatibility with Timex > 3.3

Hi, thanks for your great work!

Do you have plans for supporting Timex latest version? If not, please update mix.exs

{:timex, "~> 3.3.0"}

The problem is in

exq-scheduler/lib/exq_scheduler/schedule.ex

Line 148 in 6114c10

case Timezone.resolve(timezone, Timex.to_erl(naive_time), :wall) do

.

Problems:

• Since 3.4, Timex.resolve/3 doesn't support passing an erlang representation of the date
• This method now returns TimezoneInfo and AmbiguousTimezoneInfo structs instead of Datetime and AmbiguousDatetime

I attempted to solve by myself, but no success :(

Redefined ExqScheduler.Time module in test_helper.exs doesnt have correct precision

While scaling the time from seconds to hour we are losing the minutes and seconds. Due to this timer value in the test suit is always clamped to the nearest hour.

Need to get time with lower precision (such as microseconds instead of seconds) so that scaling doesn't affect.

`/exq/recurring` page returns Internal Server Error

I get an intermittent issue on production where the /exq/recurring page returns Internal Server Error. When this is happening, the related scheduled jobs are not processing. The only way to fix it is to restart the phoenix server.

I can’t figure out what is happening or how to get to the bottom of it. Does anyone have an idea of what might be going on?