billtubbs/experiments

Tools to setup and manage machine learning tests and experiments on a remote machine

experiments's Issues

Improve subprocess exception handling

Haven't tested jobdispatcher.py on TimeoutExpired and CalledProcessError exceptions. Not sure if they are handled gracefully and how exception information is reported.

See: cpython/Lib/subprocess.py

Folder structure is not ideal

The way it's currently set up, the job dispatcher is located in a sub-directory (scripts) and creates the queue sub-directories in its parent which is the root of this repository.

Although this can be changed using the command-line-arguments, it means that when the job dispatcher launches shell scripts, by default they are moved to the repository root to be executed. Depending on what the users script is doing, this might lead the user to put various other files and folders (data inputs, outputs, logs, etc) in the git repository folder which would not be good file organization.

What I am planning to do is to set the default so that this repo is intended only for the job-dispatcher code and example scripts and set the default so that it launches each script in the parent folder of this repository.

This means that users can clone copies of this repo in separate "project folders" and have all their project data exist alongside this repo rather than inside it.