Improve Scheduler design

[ysbaddaden]

Improve Scheduler design

[ysbaddaden]

Surprisingly, every isn't trivial to solve. Running something every interval sounds simple, but... since when should it start occurring? CRON solves that by having fixed date or time (e.g. run every Sunday at 5:00).

I'm thinking to solve this by memoizing Time.local when the recurring job was scheduled as the starting point. The first occurrence of every: 5.minutes will be now + 5 minutes, now + 10 minutes and so on. That sounds good for minutes or hours, but days, weeks or months may become kinda weird.

For example every: 1.week means that the first occurrence will be exactly 1 week from when the program is started, whereas with CRON @weekly is next Sunday at midnight, which is an arbitrary date, except that the job will be run at an expected time (stable), unrelated from the program's startup time (unstable) or the time the job got scheduled (unstable).

With continuously deployed software, we risk that infrequent jobs would never actually run. Imagine a job running every 6 hours (4 times a day), but is deployed every 3 or 4 hours during the day, each restart would delay the next occurrence, pushing the job that should have run once or twice during the day to later in the night. Worst: let's imagine the team working on the software works across the world, it could be delayed to the weekend. This is an extreme scenario, but: if something can happen it will happen.

Translating to CRON sounds impossible. How would you translate 3.days 😅

Maybe the solution is actually simple: since could be a stable time. For example "last Monday at midnight". Then we resolve job recurrence from this date. It's arbitrary but it's stable: 3.days would lead to running on Thursday -> then Sunday -> Wednesday (always at midnight), ... whereas 3.days + 13.hours + 10.minutes would be Thursday at 13:10 -> Friday at 02:20, ...

[ysbaddaden]

Note: in case the program is restarted across weeks, the start date will change, and the job schedule will change to be the last Monday at midnight which isn't the same Monday from the previous run, but that sounds acceptable and documentation should advise to prefer CRON when regularity must be exact across restarts.

EDIT: instead of last Monday at midnight, it could be an epoch, like 2023-01-01T00:00:00Z and suddenly the recurrence would be stable & exact 😆

Confirmed: using an epoch makes the number simple and stable over time.

We might be concerned about the number of occurrences between the epoch and now might grow to a very large number, but even if we go the whole 0001-01-01T00:00:00Z to 9999-12-31T23:59:59Z range at the minimum resolution of 1 minute, that's a mere ~5.259 billion occurrences, slightly above UInt32::MAX (~4.295 billion). Even with a 1 second resolution that's ~315.538 billion occurrences, well below Int64::MAX.

[ysbaddaden]

Another note: the whole problem is that the scheduler works in the program's memory alone, and can't depend on an external database.