microsoft/aiida-dynamic-workflows

An AiiDA plugin for dynamically defining workflows using Python functions

aiida-dynamic-workflows

An AiiDA plugin for dynamically composing workflows from Python functions that run as CalcJobs.

This is experimental, pre-alpha software.

Prerequisites

An environment where the development version of AiiDA is installed. This plugin makes use of a bugfix on the development branch, which will not be included in an AiiDA release until v2.0.

Installing

As pre-alpha software, this package is not released on PyPI. Currently the only way to install the plugin is to clone the repository and use pip:

pip install -e .

Initialization

This plugin uses Conda for managing Python environments on remote computers. Any Computers that you use with this plugin must have a conda_dir property that contains an absolute path to the Conda directory on the machine (typically something like /home/{username}/miniconda3. The add_extras.py script in example_cluster_setup/ can help you with this

Examples

The examples/ directory contains Jupyter notebooks that illustrate the main features of aiida-dynamic-workflows. The notebooks are in Markdown format, and so require the Jupyter plugin jupytext in order to run them.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

aiida-dynamic-workflows's Issues

Register on aiida plugin registry

Traditionally, this has been done via the setup.json but perhaps it makes sense to wait until @chrisjsewell has added support for the setup.cfg

Instructions on https://github.com/aiidateam/aiida-registry

document requirements on remote compute env

• cloudpickle needs to be installed
• I was using python 3.8 locally and 3.9 remotely and got the friendly error message XXX lineno: 13, opcode: 53. Turns out cloudpickle can only be used to send objects between the exact same version of Python (including patch versions)

example notebook that can run locally

It would be nice to be able to run an example notebook without needing to have access to a slurm cluster (which puts some limitation on the user base).

Infinite recursion calling `fetch_value()` on returned AiiDA Nodes

Hi,

I was trying to use a PyFunction to construct the inputs to another CalcJob. For this I need to be able to return preexisting AiiDA data nodes such as e.g. KPointsData. This leads to an infinite recursion when calling fetch_value().

MWE to reproduce:

import aiida_dynamic_workflows as flows
from aiida.orm import Int

cluster_env = flows.engine.execution_environment(
    "aiida-dev",   # conda environment
    "localhost")

@flows.step(returns=["v1"])
def test(nk: int) -> int:
    return Int(nk)

z = flows.engine.apply(test, nk=4)
t = aiida.engine.run(z.on(cluster_env))
t['return_values']['kpoints'].fetch_value()

scheduler plugin chokes when job exits immediately

If the remote job dies immediately, it can happen that the scheduler plugin raises an exception:

*** 2 LOG MESSAGES:
+-> ERROR at 2022-03-03 16:34:44.678009+00:00
 | the `parse_output` method of the scheduler excepted: not enough values to unpack (expected at least 1, got 0)
+-> REPORT at 2022-03-03 16:34:45.789686+00:00
 | [227|PyCalcJob|on_except]: Traceback (most recent call last):
 |   File "/opt/conda/lib/python3.8/site-packages/plumpy/process_states.py", line 231, in execute
 |     result = self.run_fn(*self.args, **self.kwargs)
 |   File "/home/aiida/aiida-core/aiida/engine/processes/calcjobs/calcjob.py", line 471, in parse
 |     exit_code_retrieved = self.parse_retrieved_output(retrieved_temporary_folder)
 |   File "/home/aiida/aiida-core/aiida/engine/processes/calcjobs/calcjob.py", line 560, in parse_retrieved_output
 |     exit_code = parser.parse(**parse_kwargs)
 |   File "/home/aiida/aiida-dynamic-workflows/aiida_dynamic_workflows/parsers.py", line 52, in parse
 |     job_infos = calc.computer.get_scheduler().parse_detailed_job_info(
 |   File "/home/aiida/aiida-dynamic-workflows/aiida_dynamic_workflows/schedulers.py", line 64, in parse_detailed_job_info
 |     fields, *job_infos = lines
 | ValueError: not enough values to unpack (expected at least 1, got 0)

Consider integration with `ContainerizedCode`

AiiDA 2.1 finally adds support for containerized codes, see changelog.

This could be interesting for aiida-dynamic-workflows, since it would allow to encapsulate the Python environment on the remote compute resource.

If the compute resource is SLURM, there are plugins like pyxis that make it straightforward to run containers there.

Mentioned to me by @giovannipizzi in today's AiiDA hackathon

Reorder parameters to `execution_environment`

Currently the first parameter is Optional[str] naming a conda environment, or None for "the conda environment with the same name as the currently active one". The second parameter is a str, naming the computer.

Instead, we should take the computer name first and have the conda environment name second, with None as the default.

This will be a breaking change, but is trivial to fix.