pombredanne/compmake

Compmake is a non-obtrusive module that provides Makefile-like facilities (including parallel processing) for batch Python applications.

Compmake

Compmake is an unobstrusive Python library that provides Make--like facilities to a Python application, including:

• Minimal effort job management and parallelization (multiple CPU on a single host, cluster computing using SGE, and experimental support for cloud computing using Multyvac).
• Caching of temporary results: you can interrupt your program and restart it without losing what was already computed.
• Failure tolerance: if a job fails, other jobs that do not depend on it continue to be executed.
• A console for inspecting failures and partial completion, with familiar commands such as make, clean, etc.

Please see the manual at:

http://purl.org/censi/compmake-manual

Issues

Please report any problem using Github's issue tracker at

http://github.com/AndreaCensi/compmake/issues

Acknowledgements

Your tax dollars at work! Compmake's development was supported by the National Science Foundation in the National Robotics Initiative program under grant #1405259.

Screencasts

• Screencast 1: Compmake overview

• Screencast 2: Compmake basic usage

• Screencast 3: Compmake + Multyvac

• Screencast 4: Compmake + SGE

• Demo with images used in the screencasts

Some animations of the job graph

Robustness to job failure

Simple dynamic jobs

Dynamic jobs and recursive parallel executions

Changelog

• v3.5 - Added why command --- compact error visualization. Removed color effects which might not be suitable for all console types.
• v3.4.1 - Bug fix; experimental multyvac_sync
• v3.4.0 - Changed the way that dynamic job IDs are generated to avoid a race condition in a corner case.
• v3.3.7 - Bug fix.
• v3.3.6 - Experimental Multyvac backend.