ndejay/ripe

Middleware for your MOAB/Torque stack and pipeline

Ripe

ripe provides a domain-specific language (DSL) allowing the simple definition of a pipeline in Ruby consisting of tasks defined in shell, Liquid or ERB templating. ripe then spits out scripts that you can distribute across your compute cluster of choice.

Installation

Install it yourself as:

$ gem install ripe

Usage

TODO: Write usage instructions here

Contributing

1. Fork it ( https://github.com/ndejay/ripe/fork )
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create a new Pull Request

ripe's Issues

Write detailed high-level documentation on using ripe.

Support ERB task templates.

Create a test suite for testing `Controller#prepare` behaviour

This is the single most important case of the pipeline and should be generalized into a test.

Interface to a minimal qsub stack.

ripe currently supports PBS/TORQUE. Let's make sure that it can work on any stack that contains qsub.

Support multiple json config files.

• Allow users to specify multiple json files with the -c options. Each specified json file needs to be separated by comma.
• The json file(s) will be searched in:
  1. The working directory
  2. The $RIPELIB/config path
• When there are multiple configurations for a given key, issue a warning and use the value defined in the first configuration file where it is described

Change the permissions of the database to the same as the user's umask.

Keep a log file at the sample level

With all the .sh scripts being written in the .ripe (by default) folder, it's hard to know, for a given sample, which workflow(s) and tasks(s) were applied with what options. It would be nice to have a log file in each sample directory that keeps a trace of what was done.

Display updated workers on calling `sync`

It would be nice to have WorkerController#sync return a list of changed Worker records as was the case prior to v0.2. It currently returns a list of booleans rather than the list of workers.

Track project_name and user in the ripe metadata db.

By tracking which user submitted jobs, it prevents issues that occur when multiple users are tracking jobs on the compute cluster within the same ripe repo.

Transform ripe into a shell-script producing engine.

• Remove all dependencies with MOAB or Torque.
• Produce shell scripts instead.

Support config file for workflows.

Create a DSL for defining workflows and tasks

The current API is exposes too many internals to the end user. It is critical to define a DSL in order to allow the user to easily define workflows and tasks.

Properly separate the roles of `Controller`, `WorkerController`, `Worker` and `CLI`.

Controller, WorkerController, Worker and CLI currently share the burden for interfacing with the compute cluster. It would be nice to do as follows:

• Rename Controller to Repo and have it manage only the ripe repository (folders in which .ripe is found.)
• Delegate all job management tasks from Worker to WorkerController.
• It will then be possible to have the CLI properly interface to Controller and WorkerController.

Jobs script output per tasks

It would be nice for the engine to produce a shell file per task that exclusively contains the part of the Worker-level shell script it is involved with.

More informative message when -x directory does not exit