runefriborg/gwf2

Make-like utility for submitting workflows to cluster schedulers such as PBS

gwf2's Issues

Managing running jobs through GWF

Add the possibility to update running GWF workflows by deleting jobs on hold, failed jobs and queued jobs and replacing them with new jobs.

Look into the approach from the previous GWF workflow.

Add an gwf-abort tool, which can stop executions running.

Add parameters to xsub and change the default behaviour for PBS settings

Scheduler parameters given in a workflow file must be abstracted from real formats. I suggest the following.

:xsub queue=qfat1,cores=24,mem=8,name=sadf adf

Eg.
for setting in val.strip().split(','):
key, value = setting.strip().split('=')

Also, a default name must be set for all jobs listing the targets 'TargetA,Targe...TargetX' with a max length of 128.

Add live output of stdout/stderr to gwf-status

Use a command like
gwf-status -j 28383.in -t TargetX -live

to get live output from stdout/stderr.

dynamic target, to handle N input files

Given the case that a task generates an unknown amount of output files, this must be handled.

Currently it can be handled, by making the task wrap all the files into a single archive, but then only one task will be handle to handle all these files.

A new dynamic target, which can handle N inputs or outputs as a "master-worker" setup, would be able to utilise many nodes and not just one.

I propose adding a reg-exp feature to input/output files.

Add preprocessor to gwf-workflow

Add preprocessor to gwf, which allows for more flexible and dynamic creation of GWF workflows.

Remove redundant copies

Remove the need to copy to/from shared storage.

For files needed from shared storage, we can create symbolic links. To capture files to be written, we must catch the "write" setting through LD_PRELOAD.

If a file is check pointed, then the task must output directly to shared storage and the next task will read directly from shared storage.

Change :output to :outfile and :tempfile

Always save outfile to shared storage.

Change file distribution pattern

The file distribution is too serialised, as all copies are blocked operations and performed from the mother node.

Create a CopyHandler, which will do asynchronous copying while still handling failure of transfers.

Add reasons to SKIP/RUN output

The reason for a skip or run can be outputted, if gwf2 is given an extra flag.

e.g. gwf2 -r/--reason