microcombustion/ctwrap

Python wrapper for batch simulations (e.g. Cantera)

ctwrap's Issues

Update Sphinx documentation

While a rudimentary sphinx documentation is now implemented, it is not sufficiently detailed.

To-do:

• Move README.md to REAMDE.rst and update syntax
• Create separate INSTALL.rst file
• Tie README.rst and INSTALL.rst into sphinx docs
• Add usage examples to module/class docstrings
• Update jupyter notebooks and add more of them to sphinx
• Add additional discussion on how things work
• Switch to alabaster theme?

camsym's documentation is probably one of the better examples of where this should go.

Example can not be run correctly

Win 11. python 3.9 cantera 2.6.0a
After I installed the ctwrap I try to run the example cases by running the commend line:
ctwrap run freeflame freeflame.yaml --parallel --strategy sequence

Error returned:

same result returned using the jupyter notebook as shown below

Address NaturalNameWarning

In the current implementation, HDF groups are named by simulation tasks, which may include a period ('.') or other characters that prevent names from being used as attributes. E.g. a name upstream.phi_2.4<mix> triggers the following warning:

NaturalNameWarning: object name is not a valid Python identifier: 
'upstream.phi_2.4<mix>'; it does not match the pattern ``^[a-zA-Z_][a-zA-Z0-9_]*$``; 
you will not be able to use natural naming to access this object; 
using ``getattr()`` will still work, though

The warning is inconsequential but annoying ...

Fix parallel runs on Windows

Currently the Simulation object is created from a fully instantiated module, which creates issues when running parallel on windows (Linux allows for pickling of objects, but Windows does not: currently, objects - i.e. simulation modules - are pickled when they are passed to workers in multiprocessing).

I believe a straight-forward solution would be to just grab the name of the module in Simulation.from_module, and only store the path of the module. For the actual simulation, the module can be re-loaded in Simulation.run.

Supersedes #3

Improve documentation (Sphinx?)

Currently, the documentation is rudimentary.

More adequate documentation could be easily created by improving doc strings (PEP 484), plus nbsphinx based on existing jupyter notebooks.

Fix adiabatic_flame.py module

Mixture-averaged and multi-component diffusion used to give different results, see old ctwrap. They no longer do ...

Make Parser more intuitive

Currently Parser requires a list to to specify parameter values with dimensions and comments, i.e.

    T: [300., kelvin, 'temperature']
    P: [1., atmosphere, 'pressure']
    phi: [.55, dimensionless, 'equivalence ratio']
    oxidizer: 'O2:1,AR:5'

It would be more intuitive to simply use a string and parse values using re from

    T: 300. kelvin (temperature)
    P: 1. atmosphere (pressure)
    phi: .55 (equivalence ratio)
    oxidizer: O2:1,AR:5

Improve unit tests

Unit tests should be Updated so they can be run from both root and test folder. Also, testing the truth value of a boolean (success) is not informative.

Improve warning/error messages

• Warning/error if Cantera is not installed
• List available strategies if omitted

Make save method optional

It would be more intuitive if the save method (in simulation modules) were optional (i.e. saving is skipped if the method is not defined). For those instances, the YAML input file also won't need an output entry.

Simplify syntax of variation

YAML specification could be more intuitive...

Improve unittests / fix unittest failures

Unittests should probably check whether content is saved correctly (rather than just perform some rudimentary checks).

On linux there are currently two unittest failures:

_________________________________________________________________ TestWrap.test_commandline __________________________________________________________________

self = <test_ctwrap.TestWrap testMethod=test_commandline>

    def test_commandline(self):
        cmd = 'ctwrap'
        name = self._module.__name__.split('.')[-1]
        yaml = "{}".format(Path(EXAMPLES) / self._yaml)
        pars = [name, yaml, '--parallel']
    
        self.maxDiff = None
        process = subprocess.Popen([cmd] + pars,
                     stdout=subprocess.PIPE,
                     stderr=subprocess.PIPE)
        _, stderr = process.communicate()
>       self.assertEqual(stderr.decode(), '')
E       AssertionError: 'Process Process-5:\nTraceback (most recen[518 chars]nt\n' != ''
E       - Process Process-5:
E       - Traceback (most recent call last):
E       -   File "/home/ischoegl/.pyenv/versions/miniconda3-latest/envs/ctwrap/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
E       -     self.run()
E       -   File "/home/ischoegl/.pyenv/versions/miniconda3-latest/envs/ctwrap/lib/python3.7/multiprocessing/process.py", line 99, in run
E       -     self._target(*self._args, **self._kwargs)
E       -   File "/work/GitHub/ctwrap/ctwrap/simulation.py", line 586, in worker
E       -     obj._save_metadata(metadata)
E       - UnboundLocalError: local variable 'obj' referenced before assignment

tests/test_ctwrap.py:85: AssertionError

and a related one in TestIgnition.test_commandline.

Ensure that custom.save works

It probably doesn't for a simulation module created from a file.

Freeflame does not generate CSV output

The following does not produce output:

# YAML file example for the `freeflame` module
# file specifies a parameter variation of the equivalence ratio
strategy:
  sequence:
    upstream.phi: { mode: linspace, limits: [0.4, 2.6], npoints: 12 }
  matrix:
    upstream.phi: { mode: linspace, limits: [0.4, 2.4], npoints: 6 }
    model.transport: ['mix', 'multi', 'soret']
defaults:
  upstream:
    T: 300. kelvin # temperature
    P: 1. atmosphere # pressure
    phi: .55 # equivalence ratio
    fuel: H2
    oxidizer: O2:1,AR:5
  model:
    mechanism: h2o2.yaml
    transport: mix
  domain:
    width: 30 millimeter # domain width
output:
  format: csv
  force: True
ctwrap: 0.3.0

Make interface more flexible

• Implement sub commands ctwrap list ans ctwrap run
• Implement Output class with file-type specific variants
• Implement equilibrium simulation module collecting CSV output

Implement sound simulation module with YAML output

Add test based on cantera ubuntu package

see title - now that 2.5.1 is out, the package no longer depends on the development version.

Use SolutionArray objects in simulation modules

Use Cantera's SolutionArray for consistency. Relationship to pandas needs to be clarified.

Create easy way to debug

Running a single simulation from the command line would reveal the full error stack.

Parallel simulation failures on Linux

Parallel output fails on a Linux machine (running on 6 cores). See also updated commandline notebook ... curiously, the issue does not appear consistently: verbose output -v leads to more failures, and number of cores may play a role (it is not clear how many cores are used for testing pipelines on GitHub).

_________________________________________________________________ TestWrap.test_commandline __________________________________________________________________

self = <test_ctwrap.TestWrap testMethod=test_commandline>

    def test_commandline(self):
        cmd = 'ctwrap'
        name = self._module.__name__.split('.')[-1]
        yaml = "{}".format(Path(EXAMPLES) / self._yaml)
        pars = [name, yaml, '--parallel']
    
        self.maxDiff = None
        process = subprocess.Popen([cmd] + pars,
                     stdout=subprocess.PIPE,
                     stderr=subprocess.PIPE)
        _, stderr = process.communicate()
>       self.assertEqual(stderr.decode(), '')
E       AssertionError: 'Process Process-1:\nTraceback (most recen[518 chars]nt\n' != ''
E       - Process Process-1:
E       - Traceback (most recent call last):
E       -   File "/home/ischoegl/.pyenv/versions/miniconda3-latest/envs/ctwrap/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
E       -     self.run()
E       -   File "/home/ischoegl/.pyenv/versions/miniconda3-latest/envs/ctwrap/lib/python3.7/multiprocessing/process.py", line 99, in run
E       -     self._target(*self._args, **self._kwargs)
E       -   File "/work/GitHub/ctwrap/ctwrap/simulation.py", line 695, in worker
E       -     obj._save_metadata(metadata)
E       - UnboundLocalError: local variable 'obj' referenced before assignment

tests/test_ctwrap.py:85: AssertionError
_______________________________________________________________ TestIgnition.test_commandline ________________________________________________________________

self = <test_ctwrap.TestIgnition testMethod=test_commandline>

    def test_commandline(self):
        cmd = 'ctwrap'
        name = self._module.__name__.split('.')[-1]
        yaml = "{}".format(Path(EXAMPLES) / self._yaml)
        pars = [name, yaml, '--parallel']
    
        self.maxDiff = None
        process = subprocess.Popen([cmd] + pars,
                     stdout=subprocess.PIPE,
                     stderr=subprocess.PIPE)
        _, stderr = process.communicate()
>       self.assertEqual(stderr.decode(), '')
E       AssertionError: 'Process Process-1:\nTraceback (most recen[3714 chars]')\n' != ''
E       - Process Process-1:
E       - Traceback (most recent call last):
E       -   File "/home/ischoegl/.pyenv/versions/miniconda3-latest/envs/ctwrap/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
E       -     self.run()
E       -   File "/home/ischoegl/.pyenv/versions/miniconda3-latest/envs/ctwrap/lib/python3.7/multiprocessing/process.py", line 99, in run
E       -     self._target(*self._args, **self._kwargs)
E       -   File "/work/GitHub/ctwrap/ctwrap/simulation.py", line 687, in worker
E       -     obj._save(task=task)
E       -   File "/work/GitHub/ctwrap/ctwrap/simulation.py", line 164, in _save
E       -     with h5py.File(filename, 'r') as hdf:
E       -   File "/home/ischoegl/.pyenv/versions/miniconda3-latest/envs/ctwrap/lib/python3.7/site-packages/h5py/_hl/files.py", line 427, in __init__
E       -     swmr=swmr)
E       -   File "/home/ischoegl/.pyenv/versions/miniconda3-latest/envs/ctwrap/lib/python3.7/site-packages/h5py/_hl/files.py", line 190, in make_fid
E       -     fid = h5f.open(name, flags, fapl=fapl)
E       -   File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
E       -   File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
E       -   File "h5py/h5f.pyx", line 96, in h5py.h5f.open
E       - OSError: Unable to open file (unable to lock file, errno = 11, error message = 'Resource temporarily unavailable')
E       - Process Process-3:
E       - Traceback (most recent call last):
E       -   File "/home/ischoegl/.pyenv/versions/miniconda3-latest/envs/ctwrap/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
E       -     self.run()
E       -   File "/home/ischoegl/.pyenv/versions/miniconda3-latest/envs/ctwrap/lib/python3.7/multiprocessing/process.py", line 99, in run
E       -     self._target(*self._args, **self._kwargs)
E       -   File "/work/GitHub/ctwrap/ctwrap/simulation.py", line 687, in worker
E       -     obj._save(task=task)
E       -   File "/work/GitHub/ctwrap/ctwrap/simulation.py", line 164, in _save
E       -     with h5py.File(filename, 'r') as hdf:
E       -   File "/home/ischoegl/.pyenv/versions/miniconda3-latest/envs/ctwrap/lib/python3.7/site-packages/h5py/_hl/files.py", line 427, in __init__
E       -     swmr=swmr)
E       -   File "/home/ischoegl/.pyenv/versions/miniconda3-latest/envs/ctwrap/lib/python3.7/site-packages/h5py/_hl/files.py", line 190, in make_fid
E       -     fid = h5f.open(name, flags, fapl=fapl)
E       -   File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
E       -   File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
E       -   File "h5py/h5f.pyx", line 96, in h5py.h5f.open
E       - OSError: Unable to open file (unable to lock file, errno = 11, error message = 'Resource temporarily unavailable')
E       - Process Process-2:
E       - Traceback (most recent call last):
E       -   File "/home/ischoegl/.pyenv/versions/miniconda3-latest/envs/ctwrap/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
E       -     self.run()
E       -   File "/home/ischoegl/.pyenv/versions/miniconda3-latest/envs/ctwrap/lib/python3.7/multiprocessing/process.py", line 99, in run
E       -     self._target(*self._args, **self._kwargs)
E       -   File "/work/GitHub/ctwrap/ctwrap/simulation.py", line 687, in worker
E       -     obj._save(task=task)
E       -   File "/work/GitHub/ctwrap/ctwrap/simulation.py", line 164, in _save
E       -     with h5py.File(filename, 'r') as hdf:
E       -   File "/home/ischoegl/.pyenv/versions/miniconda3-latest/envs/ctwrap/lib/python3.7/site-packages/h5py/_hl/files.py", line 427, in __init__
E       -     swmr=swmr)
E       -   File "/home/ischoegl/.pyenv/versions/miniconda3-latest/envs/ctwrap/lib/python3.7/site-packages/h5py/_hl/files.py", line 190, in make_fid
E       -     fid = h5f.open(name, flags, fapl=fapl)
E       -   File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
E       -   File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
E       -   File "h5py/h5f.pyx", line 96, in h5py.h5f.open
E       - OSError: Unable to open file (unable to lock file, errno = 11, error message = 'Resource temporarily unavailable')

tests/test_ctwrap.py:85: AssertionError

Make simulation modules independent of ctwrap code?

It would be nice if the simulation modules could be agnostic of ctwrap, i.e. be run without ctwrap being installed. The main motivation here is to make simulation modules portable.

For the implementation, this would mainly mean that Parser cannot be used to handle the default configuration. As Parser acts like a dict already, changes are presumably minimal.

Parallel batch jobs fail silently

While failing parallel jobs should not raise an error, some documentation needs to be generated

Allow for sampling of higher-dimensional parameter spaces

At the moment, ctwrap only samples in one dimension (using a sweep). An extension to a test matrix is straight-forward. Beyond, pseudo-randomly sampled parameter spaces are a logical extension (e.g. Sobol sequences). For Sobol sequences, there is some Python 2 / Cantera 2.1 code that was used for prior work (e.g. Ayoobi and Schoegl, PROCI 2015) and can be ported.

Terminate hung calculations

For marginal cases, simulations oftentimes 'hang', i.e. they do not progress and take hours to terminate. It would be neat if there were a way to 'detect' and 'kill' a process that is hung. Probably not a priority, but it would be nice to have.

Error when running multiple temperatures for adiabatic flame simulation

So I have an n-decane input model built from RMG and I converted to .cti format using cantera 2.4.0 on a Linux system. The file contains 283 species and 8753 reactions. Now I was trying to parallelise the flame simulation code for 4 different temperatures and phi values so that I can save time and memory. This entailed using the .yaml format given in the instructions of the ctwrap repository. However, when running this simulation on an interactive node on SLURM high-performance computing, the time it takes for the simulation to finish exceeds the time I have been allocated. Here is the .cti file as well as the jupyter notebook and the .yaml file.
adiabatic_flame_nc10.yaml

Flame_Speed_Convergence_RMG_nc10_detailed_model_ctwrap.ipynb

reduced_283.cti

Cant' pickle module error

I got the error message "Cant' pickle module" when running the parallel version of adiabatic_flame example, Ignition_example and minimal_example without altering the default yaml files.

Operating system: Windows 10
Cantera Version: 2.40

Streamline Parser usage

Implementation can be simplified by directly loading Parser from YAML

Implement capability to restart simulations

For one-dimensional simulations, restarting from an adjacent solution can save significant time. One possible example would be:

strategy:
  matrix:
    upstream.phi: [0.5, 0.6, 0.7, 0.8]
    settings.transport: ['Mix', 'Multi']

where Multi could be restarted from Mix. For Sobol sequences, new points can be added based on triangulated initial guesses.

PS: This is longer term and should not be considered for 0.2.0

Support saving of non-HDF output

The current implementation assumes that Cantera creates SolutionArray or FlameBase objects that are saved to HDF files. An alternative (YAML?) output for simple dictionaries would make sense.

Other output formats are possible, but probably shouldn't be handled by ctwrap itself.

Port to cantera 2.5

Cantera 2.5b is now stable, with substantial improvements of SolutionArray.

Much of the self-documenting (HDF) output capabilities are now available in 2.5, and should be removed from ctwrap to eliminate redundancies.

Add additional simulation modules for standard simulations

The YAML approach lends itself to be extended to other commonly used simulations.

• This could make ctwrap a useful teaching tool in combustion-related UG classes (albeit probably not with HDF5 output).
• Per a comment by @bryanwweber in Cantera/enhancements#69, there is a possible extension to SENKIN capabilities (e.g. use YAML as an intermediate step in CanSen)

Add tests for custom simulation modules

• At the moment, all tests use modules defined by ctwrap, I.e. it is not ensured that custom modules actually run as anticipated.
• Also, add unit tests for jupyter notebooks
• Add test for old YAML syntax.

Minor issues for 0.2.0

This is a running list of minor issues that should be taken care of:

• Replace CTI/XML mechanisms by YAML mechanism
• Fix #32 (same issue as #29)
• Check/minimize dependencies (e.g. pandas, pytables) ... ensure that environment.y ml already satisfies everything.
• Issue #26 ... not a good idea after all; improvements of docstrings take care of this
• Remove output from YAML configuration (it will always be HDF) ... enable passing of arguments to write_hdf (e.g. force overwriting of existing groups)
• Include #36
• Replace all instances of io by pathlib
• Address NaturalNameWarnings (#1)
• Include #35 (@ischoegl)
• Include #42/#43 (@ischoegl)
• Include #45 (@ischoegl)
• Finish implementation of Strategy, I.e. #25 (@ischoegl)
• Fix bug in #49
• Bump version to 0.2.0

Some remaining edits of Sphinx documentation

• Move jupyter notebooks into a dedicated docs/examples folder (from current docs/pages) (@ischoegl ... see #45)
• Add descriptions of how to run batch jobs from commandline to overview.rst, ignition.rst and adiabatic_flame.rst (see README.rst)
• Check and fix broken links in Sphinx documentation (e.g. several links in README.rst)
• Add instructions for conda environment to INSTALL.rst - see e.g. camsym
• Update docstrings of SimulationHandler ... make description more generic (not tied to minimal example)

Add CI runners

Github actions ... see https://github.com/marketplace/actions/sphinx-build ?