pepkit/divcfg

DIvvy computing configuration files.

1. Overview of divvy configuration files

The divcfg repository contains divvy computing configuration files currently in use across several research computing environments (Stanford, NIH, UVA, CeMM, and Memorial Sloan-Kettering). These files describe computing environments so that any tool that uses divvy can run jobs in these environments. These files can be used as examples to help you set up cluster or containerized computing in your own environment.

2. Setting up your environment

Using divvy in pre-configured environments

If you're at one of the following places, set-up is very simple. Here's a list of pre-configured computing environments:

• uva_rivanna.yaml: Rivanna cluster at University of Virginia
• cemm.yaml: Cluster at the Center for Molecular Medicine, Vienna
• nih_biowulf2.yaml: Biowulf2 cluster at the NIH
• stanford_sherlock.yaml: Sherlock cluster at Stanford
• ski-cer_lilac.yaml: lilac cluster at Memorial Sloan Kettering
• local_containers.yaml: A generic local desktop or server (with no cluster management system) that will use docker or singularity containers.

To configure divvy to use one of these, all you have to do is:

1. Clone this repository (e.g. git clone https://github.com/pepkit/divcfg.git)

2. Point the $DIVCFG environment variable to the appropriate config file by executing this command:

   export DIVCFG=path/to/compute_config.yaml
   

   (Add this line to your .profile or .bashrc if you want it to persist).

3. Install divvy (e.g. pip install --user --upgrade divvy)

And that's it, you're done! You can run divvy list on the command line to show you available compute packages.

If the existing config files do not fit your environment, you will need to create a divvy config file to match your environment by following these instructions:

Configuring a new environment

To configure a new environment, we'll follow the same steps, but just point at the default file, compute_config.yaml, which we will then edit to match your local computing environment.

1. Clone this repository (e.g. git clone https://github.com/pepkit/divcfg.git)
2. Point the $DIVCFG environment variable to the default config file by executing this command:

export DIVCFG=path/to/compute_config.yaml

(Add this line to your .profile or .bashrc if you want it to persist).

3. Next, use compute_config.yaml as a starting point to configure your environment. If you're using SLURM and you're lucky, the only thing you will need to change is the partition variable, which should reflect your submission queue or partition name used by your cluster resource manager. To make more advanced changes, the documentation below will guide you through all components of the configuration.

4. Once you have it working, consider submitting your configuration file back to this repository with a pull request.

3. DIVCFG configuration explained

The divvy documentation includes detailed instructions for how to write your own divvy configuration file.

What is divvy?

Divvy enables any tool to seamlessly switch between cluster resource managers (SGE, SLURM, etc.), linux containers (docker, singularity, etc.), or other computing environments.

divcfg's Issues

Update compute key

compute changed to compute_packages in divvy

--ignore-interrupts in templates on macOS

I noticed that --ignore-interrupts file is created in the working directory every time I submit a local job with looper. The file consists of logs.

It looks like there is no long version option in tee command on macOS:

TEE(1)                    BSD General Commands Manual                   TEE(1)

NAME
     tee -- pipe fitting

SYNOPSIS
     tee [-ai] [file ...]

DESCRIPTION
     The tee utility copies standard input to standard output, making a copy
     in zero or more files.  The output is unbuffered.

     The following options are available:

     -a      Append the output to the files rather than overwriting them.

     -i      Ignore the SIGINT signal.

     The following operands are available:

     file  A pathname of an output file.

     The tee utility takes the default action for all signals, except in the
     event of the -i option.

     The tee utility exits 0 on success, and >0 if an error occurs.

STANDARDS
     The tee function is expected to be POSIX IEEE Std 1003.2 (``POSIX.2'')
     compatible.

BSD                              June 6, 1993                              BSD

[mstolarczyk@MichalsMBP bedbuncher](cfg2): tee --ignore-interrupts
tee: illegal option -- -
usage: tee [-ai] [file ...]

but the templates use it.

https://github.com/pepkit/divcfg/blob/master/templates/localhost_template.sub

Can we switch to a short version: -i?

slurm template doesn't work with looper lump

The local template works with looper --lump. the slurm template, though, will only add srun to the first command.

quotes around code

Here I added quotes around {CODE}

d86cae4

I'm not sure why I did that, but it doesn't work if the command itself uses a single-quote, which I am needing to do in refgenie to specify the description, which is a string that can contain spaces. So, I have removed those single-quotes, and it works now... but just posting this issue in case something comes up and the reason for why I needed the quotes arises again.

How can I create the configuration for the AWS Batch environment?

Divvy is a wonderful tool. But I cannot find any example of the configuration for the cloud environment, e.g. the AWS Batvh environment. Can you please give me some hints about how to do that or what code I should write to implement that.

Cheers,
Zhen Zhang

Interest to support JSON?

Same issue in peppy: pepkit/peppy#206

Error (maybe) in readme

Just reading through some of this README. I think there is an error in the example Slurm template in the README section.
From the section titled "Templates Files" (probably should be "template files" but thats a separate point?) the code should read "srun {CODE}" rather than just "{CODE}". You might also want to check the slurm_sig_template.sub which may need to be updated with "srun" as well?

Screenshot below.