leebardon/easy_slurm

Instantly generate Slurm scripts with the correct module and package dependancies.

Instantly generate Slurm scripts with the correct module and package dependancies.

Table of Contents

1. About Easy Slurm
2. Getting Started
   • Prerequisites
   • Installation
3. Usage
4. Contributing
5. License
6. Contact

Easy Slurm

  !==   Avoid the endless chore of module spider-ing and copy-pasta-ing 

  !==   Fahgeddabout 'finding that other script with the Launcher commands in and copy-pasta-ing'

Easy Slurm is a HPC shell utility for quickly generating a 'skeleton' Slurm scripts. It allows the user to select the software they would like to load for their batch submission, and automatically appends the correct package dependancies to the script. The user may also choose to append a 'skeleton' sequence of Launcher lines for high throughput computing workflows.

Easy Slurm was created to avoid the hassle of manually copying and pasting 'boilerplate' code from other Slurm scripts, and of manually searching for the correct package versions that the software packages in your pipeline depend on.

(back to top)

Getting Started

Prerequisites

The project was written for bash 4, and has not yet been tested on other shells (e.g. earlier bash versions, zsh, etc).

Adding Modules

Easy Slurm is currently intended to be customised by the user. That is, you must first manually add the software packages you use as part of your workflow, and the modules they depend on. You can add multiple versions of the same software. The good news is, you only need to do this once.

Open easy_slurm.sh in your preferred text editor. At the top, there is a section called "Packages & Modules Hash Table". Simply add your modules and dependancies such that the module name is the key, and the depandencies are in space-delimited string format, as follows:

["module_version"]="dependancy1 depandancy2 depandancyN module_version"

Installation

Download the script (preferably into a project utilities folder) and make executable:

curl -L https://raw.githubusercontent.com/leebardon/easy_slurm/main/easy_slurm.sh > easy_slurm.sh && chmod +x easy_slurm.sh

(back to top)

Usage

First, make the script executable:

$ chmod +x easy_slurm.sh

To see all modules currently stored in your Easy Slurm list, run the script with no arguments:

$ ./easy_slurm.sh

If your workflow involves a large number of software packages, or different versions of the same software, you can filter to see all stored versions that match a given pattern. For example:

$ ./easy_slurm.sh bow

will return all stored packages with the consecutive string pattern "bow" in the name:

(back to top)

Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement".

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Added amazing feature!')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

(back to top)

License

Distributed under the MIT License. See License.txt for more information.

(back to top)

Contact

Lee Bardon - @teatauri - [email protected]

Project Link: https://github.com/leebardon/easy_slurm

(back to top)

easy_slurm's Issues

Display packages in alphabetical order

At present, packages are stored in a bash hash table (associative array), which is unsorted.

Suggest modifying to sort, for readability. Might necessitate a rewrite in e.g. python.

Modify script to allow for multiple core software packages to be added

At present, we can only select one software package, before the script is auto-generated.

For example, selecting 'samtools/1.10' will add:

module purge
module load gcc/8.3.0
module load samtools/1.10

and other required boilerplate. This is not useful if we'd like to build a multi-stage script that runs several software packages.

Suggest modifying to allow user to select multiple packages.

Add functionality to allow user to add packages and dependancies via command line

At the moment, users need to hardcode packages and dependancies into the associative array manually.

Suggest adding CLI functionality to deal with this. Reconsider data storage. Possible rewrite in e.g. python.