pstelzig / mopyregtest Goto Github PK
View Code? Open in Web Editor NEWMoPyRegtest: A Python enabled simple regression testing framework for Modelica models
License: MIT License
MoPyRegtest: A Python enabled simple regression testing framework for Modelica models
License: MIT License
As of v0.2.2, to install MoPyRegtest users have to clone the repository and install locally by checking out the required version and then run pip install --user .
pip install mopyregtest
and getting MoPyRegtest from the Python Package Index (PyPI).pip install mopyregtest==0.2.1
MoPyRegtest calls RegressionTest._unify_timestamps
by default in RegressionTest.compare_result
. This is convenient, but in some scenarios the implemented timestamp unification might prove unnecessary (e.g. for integral based error metrics), or unsuitable.
Users shall have the option to disable timestamp unification and compare results from actual and reference results by means of their own metric definition. Even if the results have different timestamps.
Numpy 2 is about to be released with some breaking changes. MoPyRegtest may not break as a result of users having Numpy 2 instead of Numpy 1 installed.
Remove all deprecated and obsolete elements of Numpy that are no longer present in Numpy 2, like np.trapz
in test_user_defined_metrics.py
np.trapz
The command line interface fails to generate test definition files if the Modelica model can't be translated/simulated.
To Reproduce
The automated test definition generation hides simulation errors for MyLibrary.M
:
package MyLibrary "Package"
model M "Model"
Real x(fixed=true, start=0);
equation
der(x) = -0.5*x + sqrt(0.5 - time); // Breaks simulation at time=0.5
end M;
end MyLibrary;
> mopyregtest generate .CI/mopyregtest/ "TestM" . MyLibrary.M
Simulating model MyLibrary.M using the simulation tools: omc
Using simulation tool omc
The resulting reference file .CI/mopyregtest/references/MyLibrary.M_res.csv
is missing half of the simulation, but mopyregtest
exited without any errors.
System information (please complete the following information):
Return an error when the simulation for reference file generation failed.
Print or log the error message returned by OpenModelica.
So something like:
> mopyregtest generate .CI/mopyregtest/ "TestM" . MyLibrary.M
Simulating model MyLibrary.M using the simulation tools: omc
Using simulation tool omc
Failed to simulate model MyLibrary.M
LOG_STDOUT | warning | A Modelica assert prevents the integrator to continue. For more information use -lv LOG_SOLVER
LOG_STDOUT | warning | can't continue. time = 0.500000
LOG_STDOUT | info | model terminate | Integrator failed. | Simulation terminated at time 0.5
The full log can be viewed at some/path/MyLibrary.M.log
# exit code 1
Describe the bug
Users report that under Windows simulation in the test may fail if the Modelica model that shall be tested uses an external "C"
library. They report that the simulation binary cannot find the external C library. The problem appears not to exist under Linux.
To Reproduce
A minimum working example to reproduce the bug will be constructed in the comments to this issue.
System information (please complete the following information):
Additional context
Users report that under Windows, when compiling a Modelica model the OpenModelica compiler omc
creates a .bat file alongside the .exe with the simulation binary. The issue only occurs if the .exe is executed, but not if the .bat file is used.
Right now, all tests generated from a mopyregtest.Generator
object use the same metric for all tests it generates. However, it could make sense to use different metrics for different library elements. Right now, this can only be achieved by creating regression test definitions manually, or use different mopyregtest.Generator
objects (each of which generates its own test definition file).
In mopyregtest.Generator
have to option to use dedicated comparison metrics for individual Modelica models, and use a default metric for all others.
Use the metric passed to the construtor of mopyregtest.Generator
as the default metric, and pass dedicated metrics in the mopyregtest.Generator.generate_tests
method, just like with the pre-existing references as a dictionary mapping model name to metric to be used.
Describe the bug
The GitHub Action for the unit tests fails, according to the logs as a consequence because of the unit tests for Python version 3.11 are failing. Apparently, pip is not found on the default installation for Python 3.11 that comes with the setup-python action.
To Reproduce
Run the unit tests associated with the file .github/workflows/job-unit-tests.yml
System information (please complete the following information):
Unit tests shall be exectued.
The test test_unify_timestamps_multiplicity
in test_compare_results.TestCompareResults
fails in the job Unit test job
.
Timestamps as computed by RegressionTest._unify_timestamps
are not sorted as they should be.
Reason might the version of pandas
and numpy
used in the Ubuntu package repositories (python3-pandas
) and in pip
, producing different behavior in pandas.DataFrame.fillna
.
Is your feature request related to a problem? Please describe.
Under Windows there is an issue #38 with the PATH variable. Apparently, this can be circumvented by
omc
in the system call in mopyregtest.RegressionTest._run_model
, orsimulate
, instead of buildModel
followed by calling the binary via system
Describe the solution you'd like
Explore the benefits by using simulate
in the model_simulate.mos.template
Describe alternatives you've considered
Potential fixes described by users in #38
Additional context
None
The setup.py
currently does not include the project dependencies. So on pip install --user .
the dependencies are not resolved. Thus, if the dependencies are not already satisfied, mopyregtest
will not work out of the box.
On installation via pip all the required dependencies should be installed automatically.
Setup a CI pipeline with GitHub actions to automize calling both the unit tests and the examples
Writing a test_<...>
method for every regression test can be a significant coding effort for a large number of regression tests, despite the code looking very similar each time.
Instead of manually writing a test_<...>
method for every regression test, it would be convenient to tell MoPyRegtest to simply turn certain elements of a Modelica library into a regression test. The resulting MoPyRegtest definition should be generated automatically.
The previous de-facto standard to package Python projects using setuptools and a setup.py is now being replaced by pyproject.toml. See the official Python packaging guide.
Move from setup.py to pyproject.toml to be future proof.
With the default installation of pip under Ubuntu 22.04, the local installation process pip3 install --user .
fails. It installs a package called UNKOWN
. This bug is already known and no other fix than installing a newer version of pip has been proposed.
A pyproject.toml for MoPyRegtest could look like
[project]
name = "MoPyRegtest" # Required
version = "0.4.0-alpha.1" # Required
description = "A Python enabled simple regression testing framework for Modelica models" # Optional
readme = "README.md" # Optional
requires-python = ">=3.8"
license = {file = "LICENSE"}
keywords = ["Modelica", "regression", "test", "automation"] # Optional
authors = [
{name = "Philipp Emanuel Stelzig"} # Optional
]
maintainers = [
{name = "Philipp Emanuel Stelzig"} # Optional
]
classifiers = [ # Optional
"Programming Language :: Python :: 3",
"License :: OSI Approved :: MIT License",
"Operating System :: Microsoft :: Windows :: Windows 10",
"Operating System :: POSIX :: Linux"
]
dependencies = [ # Optional
"numpy",
"pandas"
]
[project.urls] # Optional
"Homepage" = "https://github.com/pstelzig/mopyregtest"
"Bug Reports" = "https://github.com/pstelzig/MoPyRegtest/issues"
"Source" = "https://github.com/pstelzig/mopyregtest"
[project.scripts] # Optional
mopyregtest = "mopyregtest.cli:main"
[tool.setuptools]
package-data = {"mopyregtest" = ["templates/omc/model_import.mos.template",
"templates/omc/model_simulate.mos.template"]}
[build-system]
requires = ["setuptools>=43.0.0", "wheel"]
build-backend = "setuptools.build_meta"
In RegressionTest.compare_result
, the for loop over the entries of validated_cols
stops at the first column that exceeds the tolerance. All subsequent columns will not be checked.
for c in validated_cols:
print("Comparing column \"{}\"".format(c))
delta = metric(ref_data[["time", c]].values, result_data[["time", c]].values)
if np.abs(delta) >= tol:
raise AssertionError(
f"Values in Colum {c} of results {simulation_result} and {reference_result} differ by " \
f"{np.abs(delta)} which is larger than {tol}.")
While this is efficient for continuous integration (test must fail if any column comparison fails), it desirable from a model developer perspective to
validated_cols
checked, andvalidated_cols
for which the comparison has failedRight now a generated result for a model is compared to a reference result up user-defined precision. This is quite restrictive.
Instead, users should be able to select from
The testgen.py script is not included in the v0.3.1 release files on PyPI. Whereas the Generator
class to automatically generate regression tests with a script is included.
To use the command line tool testgen.py, a user would have to get it from the sources, e.g. by cloning the repository.
Give the user a command line tool in the release to generate regression test definitions.
pyfunnel is a collection of algorithms to compare result timeseries from hybrid simulations (continuous time, discrete event). It is used in BuildingsPy's regression tests. These algorithms are particularly well suited to compare results when there are events and discontinuities in the result timeseries.
CSV Result Compare uses similar algorithms like pyfunnel.
Currently the mopyregtest.metrics
collection has no similar built-in metric that allows for error localization in time and that is robust against events occuring at slightly different times.
Make
I think it makes more sense to fall back to an existing an proven solution instead of reproducing a similar functionality natively in MoPyRegtest.
As of commit b89de92, the mopyregtest
command line has only one functionality, namely to automatically generate regression test definitions.
Instead, mopyregtest
shall be designed as a fully functional command line tool, like e.g. the conan C++ package manager's cli
The mopyregtest
command line tool will have other usages than just test generation, like e.g.
mopyregtest generate --metric=Lp_dist --references=Model3:Model3_res.csv /path/to/gen/target MyGenTests /path/to/package Model1,Model,Model3
mopyregtest compare --metric=Lp_dist --vars=var1,var3 /path/to/csv_1.csv /path/to/csv_2.csv
or more generally
mopyregtest generate [-h] [--metric {norm_p_dist,norm_infty_dist,Lp_dist,Linfty_dist}] [--references REFERENCES] test_folder test_name package_folder models_in_package
mopyregtest compare [-h] [--metric {norm_p_dist,norm_infty_dist,Lp_dist,Linfty_dist}] [--vars VARS] csv_files_1 csv_file_2
.github/workflows
are for on Ubuntu Linux onlyopenmodelica/openmodelica:v1.21.0-minimal
is also based on UbuntuAs of v0.1.2 all output from unittest
when running MoPyRegtest defined tests are printed to the shell.
User shall be able to create a log file with test results.
If a reference result and the actual result have no columns (i.e. variable names in the result CSV files) in common and the default argument validated_cols=[]
is passed to RegressionTest.compare_result
is used, the intersection of column names from reference and actual result will be taken. The intersection in this case is just "time"
and the test will pass.
"time"
as a result variable column namevalidated_cols
and there are no common columnsvalidated_cols
contains elements, and, if so, that all its entries are present in the actual and the reference result. Otherwise throw an error.As of v0.2.2, tagging, testing and building is entirely manual and there are no releases yet, just tags.
Right now, a user would have to go to the Actions tab to see which workflows pass and which fail.
Add badges for the project status, like
The command line interface fails to generate test definition files if the Modelica model can't be translated/simulated.
To Reproduce
The automated test definition breaks for MyLibrary.M
:
package MyLibrary "Package"
model M "Model"
Real x(fixed=true, start=0);
real y; // Syntax error and number of variables not matching number of equations
equation
der(x) = -0.5*x;
end M;
end MyLibrary;
> mopyregtest generate .CI/mopyregtest/ "TestM" . MyLibrary.M
Simulating model MyLibrary.M using the simulation tools: omc
Using simulation tool omc
/bin/sh: 1: ./MyLibrary.M: not found
Traceback (most recent call last):
File "/home/USERNAME/.local/bin/mopyregtest", line 8, in <module>
sys.exit(main())
File "/home/USERNAME/.local/lib/python3.10/site-packages/mopyregtest/cli.py", line 136, in main
parse_args(sys.argv[1:])
File "/home/USERNAME/.local/lib/python3.10/site-packages/mopyregtest/cli.py", line 129, in parse_args
args.func(args)
File "/home/USERNAME/.local/lib/python3.10/site-packages/mopyregtest/cli.py", line 48, in generate
gen.generate_tests(test_folder, test_name, result_folder, references)
File "/home/USERNAME/.local/lib/python3.10/site-packages/mopyregtest/generator.py", line 237, in generate_tests
self._generate_reference(reference_folder=test_folder / "references",
File "/home/USERNAME/.local/lib/python3.10/site-packages/mopyregtest/generator.py", line 175, in _generate_reference
shutil.copyfile(pathlib.Path(tmp_res_folder) / res_file, str(pathlib.Path(reference_folder) / res_file))
File "/usr/lib/python3.10/shutil.py", line 254, in copyfile
with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/MyLibrary.Mae09u5nf/MyLibrary.M_res.csv'
System information (please complete the following information):
Return a clear error message telling the user that:
MyLibrary.M
.The script should exit gracefully instead of failing to open some file and ideally clean up all half-generated files like .CI/mopyregtest/references/MyLibrary.M_res.csv
and .CI/mopyregtest/test_testm.py
.
So something like:
> mopyregtest generate .CI/mopyregtest/ "TestM" . MyLibrary.M
Failed to generate test definition for PNlib.TA:
Failed to build model:
[/path/to/MyLibrary/package.mo:4:5-4:11:writable] Error: Class real not found in scope M.
Aborting test definition generation.
Removing temporary files.
# exit code 1
unittest
standard output.RegressionTest.compare_result
could process)For any failed regression tests, the output of the respective actual simulation run is not cleaned up (good behavior).
root
|
+ -- reference
|
+...
+ -- actual
|
+...
+ -- failed
|
+ var_failed_0
|
+ reference
|
+ actual
|
+ var_failed_1
|
+ reference
|
+ actual
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.