pzelasko/plz

Helpers for running native Python functions as qsub jobs on the CLSP grid

plz

Helpers for running native Python functions as qsub jobs on the CLSP grid.

plz uses Dask and Dask-Jobqueue to distribute Python functions as SGE-managed jobs. No need to wrap python scripts into bash scripts anymore!

Installation

pip install git+https://github.com/pzelasko/plz

Examples

CPU jobs:

def task(x):
    return x ** 2

import plz
plz.map(task, range(1000), jobs=10)

GPU jobs:

def task(x):
    import torch
    return torch.tensor([x] * 100, device='cuda').sum()

import plz
plz.map(task, range(1000), jobs=1, gpus=1)

Iterate over multiple sequences:

def task(x, y):
    return (x + y) ** 2
    
import plz
plz.map(task, range(1000), range(1000, 2000), jobs=10)

Using logs:

def task(x):
    import logging
    logging.info(f'Running job with input {x}')
    return x ** 2

import plz
plz.map(task, range(1000), jobs=10, log_dir='/path/to/logs')

Single task:

def task(x):
    return x ** 2

import plz
plz.run(task, 1)

Running on CLSP or COE grid

By default, the map and run methods are configured to run on the CLSP grid. To run on the COE grid, additionally pass the argument grid='coe' to these methods.

Technical details

Under the hood, for each run or map call, it creates your own "mini-cluster" of Python worker "services" where the jobs are being distributed. This cluster has its own scheduling, load balancing etc. It automatically shuts down as soon as all the inputs are processed.