sayerhs / cpl Goto Github PK
View Code? Open in Web Editor NEWCaelus Python Library
Home Page: https://sayerhs.github.io/cpl/
License: Apache License 2.0
Caelus Python Library
Home Page: https://sayerhs.github.io/cpl/
License: Apache License 2.0
Users must be able to submit parallel jobs using the following workflows:
The following MPI implementations must be supported
CPL can handled multiple versions but that functionality is only accessible through the CPL module in a custom Python script. Running multiple versions through the caelus run
or caelus tasks
should be supported with, for example, a cmd option:
caelus --cml-version="8.04-dev" run
where "8.04-dev" would be a version specified in the configuration file.
To facilitate workflows that require collaboration between case directories, a mechanism to group tasks based on the case directory should be added.
A suggested solution would be to add a new task called task_group
where the case directory is used by all sub-tasks. E.g. yaml
tasks:
- task_group:
case_dir: <coarse mesh case dir>
- clean_case:
remove_zero: no
remove_mesh: no
- run_command:
cmd_name: simpleSolver
- task_group:
case_dir: <fine mesh case dir>
- run_command:
cmd_name: mapFields
cmd_args: "<coarse mesh case dir> -consistent -sourceTime latestTime"
- run_command:
cmd_name: decomposePar
cmd_args: "-force"
- run_command:
cmd_name: simpleSolver
parallel: True
Add an option in CML installer to allow user to install CPL alongside during installation process.
For CPL release, we should be able to run tutorials through CPL interface. The following tasks must be completed for this
Allrun.py
in tutorials that can be run with caelus_tutorials
Allclean.py
with caelus_tutorials
The current method to stop CPL actions, is to kill it with CTRL+C. It would be helpful if CPL had a simple event handler to more gracefully stop actions.
I suppose for the cfg
, env
, clone
, logs
, and clean
actions nothing particularly special needs to done under the hood, other than not proceeding past the next "operation". What constitutes an operation would have to be determined.
For the run
and tasks
actions, the proper signal would need to be passed to the CML application. CML applications can be aborted with the abort
function object whereby the application can perform various abort types if a particular file name exists in the case directory. E.g. in the controlDict
functions
{
controlOnNext
{
type abort;
functionObjectLibs ( "libjobControl.so" );
action nextWrite;
fileName "snapshot";
}
killNow
{
type abort;
functionObjectLibs ( "libjobControl.so" );
action noWriteNow;
fileName "killrun";
}
suspendNow
{
type abort;
functionObjectLibs ( "libjobControl.so" );
action writeNow;
fileName "suspend";
}
}
Perhaps CPL could automatically add a template version of the above example to the controlDict
for the run
or tasks
actions. Then upon receiving a particular event, e.g. CTRL+C would correspond to the noWriteNow action, CPL would create the specific file in the case direction. The function object automatically deletes the file so no further action would be required. Other keyboard events could be used for nextWrite and writeNow.
This would require some discussion about design before any implementation would occur,
Currently, the *.foam
file is created as part of process_logs
. Need a better solution for this.
For long runs, especially submitted on a queue it is desirable to allow user to monitor logs as the run progresses.
Multiple invocations of the -e
(--exclude-patterns
) option with the caelus_tutorials
script does not work correctly. It causes the script to find all the tutorials for each time -e
is used and excludes the patterns for one instance of each tutorial. E.g.
caelus_tutorials --clean -d incompressible/simpleSolver/ras -e "*pitzDaily*" -e "*turbineSiting*"
Should clean all the tutorials in the incompressible/simpleSolver/ras
directory except those with pitzDaily
and turbineSiting
in the path name. However, each tutorial is cleaned twice except the those matching the patterns are cleaned once.
When using the -w
option with caelus logs
, excluding certain fields with -e
option is ignored. E.g.
caelus logs -w -c -e "pCorr" simpleSolver.log
will still plot the pCorr residual. It works correctly when creating a plot.
caelus logs -p -c -e "pCorr" simpleSolver.log
Use following log file for debugging.
simpleSolver.log
When testing log processing on a log file from the VOF solver, CPL returns the following error:
Traceback (most recent call last):
File "/home/cnsidero/.virtualenv/bin/caelus", line 11, in <module>
load_entry_point('caelus', 'console_scripts', 'caelus')()
File "/home/cnsidero/Development/AppliedCCM/caelus/caelus-python/caelus/scripts/caelus.py", line 414, in main
cmd()
File "/home/cnsidero/Development/AppliedCCM/caelus/caelus-python/caelus/scripts/core.py", line 133, in __call__
self.args.func()
File "/home/cnsidero/Development/AppliedCCM/caelus/caelus-python/caelus/scripts/caelus.py", line 383, in process_logs
logfile=args.log_file)
File "/home/cnsidero/Development/AppliedCCM/caelus/caelus-python/caelus/post/logs.py", line 309, in __init__
logs()
File "/home/cnsidero/Development/AppliedCCM/caelus/caelus-python/caelus/post/logs.py", line 101, in __call__
self._process_file(patterns)
File "/home/cnsidero/Development/AppliedCCM/caelus/caelus-python/caelus/post/logs.py", line 126, in _process_file
pat.send(line)
File "/home/cnsidero/Development/AppliedCCM/caelus/caelus-python/caelus/utils/coroutines.py", line 49, in grep
tgt.send(mat)
File "/home/cnsidero/Development/AppliedCCM/caelus/caelus-python/caelus/post/logs.py", line 238, in continuity_processor
"\t".join(x for x in rexp.groups()) + "\n")
AttributeError: 'LogProcessor' object has no attribute 'time_str'
Reproduce by using the attached log file:
vofSolver-test.log
$> caelus logs vofSolver-test.log
To help CPL be consistent with CML, investigate parsing the build_config.py (or use the modules in site_scons) from the PROJECT_DIR to set, for example, MPI paths, USER paths, etc.
No support to use applications or libraries that exist in CAELUS_USER_APPBIN and CAELUS_USER_LIBBIN, respectively.
Additionally, caelus env
does not add the aforementioned environment variables the output file.
Set mpiexec
as the default instead of srun
for SLURM queue submits.
Create a utility to quickly plot QoIs from log files
Add/update CPL installation instructions for Windows. Add additional sections for conda
and virtualenv
setup for windows if necessary.
Upon stopping the log plot watcher before the solver has finished, e.g.
$> caelus logs -w simpleSolver.log
further processing of the log files, e.g.
$> caelus logs -p simpleSolver.log
won't process the log file past the time-step/iteration at which the plot watcher was stopped unless the .logs_state.json
file is removed from the logs
dir.
Ensure that all features work on Windows OS.
caelus.yaml
in APPDATA
caelus.yaml
in USERPROFILE
caelus.yaml
in current working directoryCAELUSRC_SYSTEM
and CAELUSRC
caelus-environment.cmd
conda
and virtualenv
(See #2)The clean_case task with the remove_mesh argument fails when the polyMesh dir doesn't exist.
I think it should continue more gracefully.
When specifying a custom log file name either in a task file with log_file
option or with the run action (e.g. caelus run simpleSolver -l test.log
), the default log file is always used, e.g. simpleSolver.log
for simpleSolver.
Add support for generating plots of forces and other data written by function objects during simulation runs. Some obvious ones to support might be:
They may be others to consider
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.