bigfav/megacron

Wikimedia Distributed Cron

megacron's Issues

Don't recreate jobs that haven't been modified

Currently megacrontab removes all jobs for the user and creates new ones even if the jobs haven't been modified. The implication of this is that the last_time_run will be changed to the current datetime and the job may not run at the time that the user expects it to. If the interval and command of a job is identical to one that existed previously, then that job should persist in the database.

I don't personally think that we should handle modified jobs because once they are changed they aren't really the same job. The logic to handle this gets quite complicated too. How similar does a job have to be to be considered modified? What if there are multiple jobs that are similar to a previous job?

Add more stats to megacron-status

Right now megacron-status only prints out the number of workers that are up. There are many other useful stats that could be helpful to a user of our system or to us while we are developing it. A few examples that I can think of are:

• Number of Jobs in queue
• Number of schedules
• Number of users with schedules
• Time the next job is going to run

You should add whatever stats that you think will be useful. If you are not sure you can ask Marc what he thinks.

Allow specifying a user to megacrontab

Add an option to allow editing another user's crontab. For example "megacrontab -u bob" would edit the crontab of the user with the name bob instead of the current user (which should be the default if -u is not specified). Only the root user should be allowed to use this option and if another user tries to use it an error message should be printed and the program should exit (this is how other crontab implementations work). I recommend that you use Python's argparse library to parse the command line options (and provide useful help messages to the user).

Man pages and Online Documentation

We need to document the commands for functionality. We can utilize many online resources to aid us in this work.

Mishandling Jobs

Given the following crontab:

I run the command:
python edit.py -u crontab.tab

Then I run the scheduler. Next I run two worker threads in separate terminal windows. The jobs should be distributed among the two workers, and they should both print two msgs.

Once the minute comes, one worker correctly prints two msgs:

However, the other worker has odd behavior, as it only prints one msg:

Then after the refresh time for the worker (10 seconds), it prints the same msg again:

Given this setup, the bug appears semi-frequently.

The current setup for the bug is complex (multiple workers running, multiple jobs for each worker, and scheduler running). I have not been able to simplify the setup and still get this bug unfortunately, suggesting that it may be a problem with the handling of multiple workers accessing the flat file.

Create unit tests for api.py

We eventually plan on creating multiple api interfaces for different databases and let the user choose which one to use. It will be very important that each of these will work in exactly the same way, so that we don't have bugs that will appear when only using one type of database. We need an extensive set of unit tests that can be run when adding support for new databases.

After creating the tests you should configure setuptools to be able to run the tests by just using the "./setup.py test" command. Also, add a section to the README to let other people know how to run them.

Retain comments in crontab

If comments or blank lines are added to a crontab they should be preserved between edits.

File Naming Issues

If we are going to name our user interface file "crontab" we will have to edit the python file for our crontab reader library since it is named crontab as well. Is this something everyone is willing to do for this?

Kill megacron with the megacrond command

There should be a graceful way to kill megacron, as illustrated by John's issue with getting rid of the daemon.