IDL Job Parallel Framework

Document Author:	Luke Domanski, CSIRO, IM&T, Scientific Computing
Document Format:	This document was written in reStructuredText. There a numerous converters and generators for exporting from reStructuredText to pretty document formats.

Contents

Introduction
Software requirements and files
- Environment and licensing requirements
- Collecting required packages
Compilation and Setup
- Compiling the code and documentation
Documentation

Introduction

A generic framework developed for generating job parallel jobs [1] for parallel platforms based on a user defined (calling application) set of preprocessing, work, and postprocessing IDL executables or routines, as well as a list of application tasks to perform with parallel workers.

Different target parallel platforms are supported via job generator plugins.

The parallel job can be configured and generated on any machine running IDL or GDL, for execution on a remote parallel machine supported by the chosen plugin. The generated job directory can then be copied to the remote parallel machine, and the job executed in a standalone fashion.

Note

The machines used for generation and execution could potentially be the same machine.

[1]	i.e. the method of parallism utilised in the generated job is "job level parallelism" as opposed to a finer grained parallelism

Software requirements and files

Environment and licensing requirements

Compilation

IDL + IDL Development Licence

Job Generation (Client) and Execution (Remote)

IDL + IDL Runtime Licence
GDL (optional alternative, see below)

Some job generator plugins support launching of worker processes using the open source GDL implementation of the IDL language and environment. The REMOTE user application to be run on the parallel machine must support IDL versions <7.1 (at time of writing) to utilise GDL.

Collecting required packages

You will required the source packages to compile this software. Which can be downloaded from the GitHub source code repository via:

git clone https://github.com/csiro-scientific-computing/parallel_framework.git

Compilation and Setup

Compiling the code and documentation

The compilation is a two part process carried out on both the client and remote machines:

Compile The IDL Job Parallel Framework and Documentation

See Compiling the Job Parallel Framework
Add the IDL Job Parallel Framework installation directory to your IDL_PATH and GDL_PATH

See Setting IDL_PATH and Setting GDL_PATH

The package can be installed as a source-binary combination, or as a binary only installation.

Compiling the Job Parallel Framework

The compilation processes has been automated in the build/build_parframewrk.pro and build/build_docs.pro scripts.

To compile:

Start an IDL Development Environment
Change to the Parallel_Framework base directory

Run the build scripts:

> @build/build_parframewrk
> @build/build_docs

build_parframewrk creates two types of binaries in the Parallel_Framework's bin directory:

The IDL Job Parallel Framework Library & API(bin/par_framework.sav): whose functions/procedures you can call from other IDL programs to discover information about available job generators and generate parallel jobs
Job generator plugins (bin/plugins/*.sav): which contain compiled versions of the job generator plugins present in the Parallel_Framwork's src/plugins/ directory. The plugins are loaded on demanded by par_framework.sav to generate different type of job output. They should not be called directly by user applications.

build_docs generates IDL html help files from the header comment blocks of functions and procedures in the IDL Job Parallel Framework source code, and places them in the doc/reference directory.

You can then view the output html files individually in a web browser. See the Reference Manual section of this document for table of contents.

Setting IDL_PATH

For other applications to use the framework, you must make the Parallel Framework Library visible on the IDL !PATH.

To do this, append +<path_to_Parallel_Framework>/bin and +<path_to_Parallel_Framework>/src to the IDL_PATH system or user environment variable, where "<path_to_Parallel_Framework>" is a place holder for the real directory path.

Important

The resulting IDL_PATH must contain the special token <IDL_DEFAULT> for IDL internal use. It is recommend you include this token when appending to IDL_PATH.

On Linux derivative systems

Use your shell's provided mechanism for setting environment variables.

e.g. For bash:

> export IDL_PATH=${IDL_PATH}:<IDL_DEFAULT>:+<path_to_Parallel_Framework>/bin:+<path_to_Parallel_Framework>/src

e.g. For tcsh:

> setenv IDL_PATH ${IDL_PATH}:<IDL_DEFAULT>:+<path_to_Parallel_Framework>/bin:+<path_to_Parallel_Framework>/src

Place this command in your user shell start-up script to ensure it is set on every login. e.g. .bashrc (bash), .cshrc (tcsh), .profile, etc. This file will be system and shell dependant, please check your system administrator's policy/recommendation.

On modern Windows systems

In the Systems Properties menu, accessible via "Control Panel>System and Security>System>Change Settings" or similar:

click "Advanced Tab>Environment Variables..."
to append an existing IDL_PATH
1. Select the IDL_PATH variable in either the "User variables" or "System variables" section
2. click "Edit.."
3. enter ;+<path_to_Parallel_Framework>/bin:+<path_to_Parallel_Framework>/src;<IDL_DEFAULT> after the existing text in the "Variable value" text box (note the leading semi-colon!)
OR to specify a new IDL_PATH variable if necessary
1. click "New.." in either the "User variables" or "System variables" section
2. enter IDL_PATH into the "Variable name" text box
3. enter +<path_to_Parallel_Framework>/bin:+<path_to_Parallel_Framework>/src;<IDL_DEFAULT> in the "Variable value" text box (note absence of leading semi-colon!).

Warning

"System variables" will apply to all users of the system.

Note

Depending on your system privileges, you might only be able to add or edit the "User variables".

Setting GDL_PATH

Append +<path_to_Parallel_Framework>/src to the GDL_PATH environment variable following similar steps to those outlined in Setting IDL_PATH, e.g. for Linux bash:

> export GDL_PATH=${GDL_PATH}:+<path_to_Parallel_Framework>/src

Note

For GDL you do not need to add the <IDL_DEAFAULT> token.

Documentation

If you have not yet followed the instructions in the Compiling the code and documentation section, please do so.

User Manual & Tutorial

The Job Parallel Framework can be used in two modes:

By the generator application: To generate parallel jobs using the generate_parallel_job procedure
By the remote worker(s): To take advantage of Job Parallel Framework helper functions which simplify worker application development

For details on usage please see the User Guide and Tutorial.

Extending the IDL Job Parallel Framework

The framework can be extended through user defined job generator plugins which allow for a broad range of target platforms and job management systems to be supported through a single library API.

The generator plugins MUST implement the plugins interface described in the developer guide and also the par_framework.pro documentation.

Reference Manual