mzy2240/esa

Easy SimAuto (ESA): An easy-to-use Power System Analysis Automation Environment atop PowerWorld Simulator Automation Server (SimAuto)

Easy SimAuto (ESA)

Easy SimAuto (ESA) is an easy-to-use Power System Analysis Automation Platform atop PowerWorld's Simulator Automation Server (SimAuto). ESA wraps all PowerWorld SimAuto functions, supports Auxiliary scripts, provides helper functions to further simplify working with SimAuto and also turbocharges with native implementation of SOTA algorithms. Wherever possible, data is returned as Pandas DataFrames, making analysis a breeze. ESA is well tested and fully documented.

Documentation

For quick-start directions, installation instructions, API reference, examples, and more, please check out ESA's documentation.

If you have your own copy of the ESA repository, you can also view the documentation locally by navigating to the directory docs/html and opening index.html with your web browser.

If you want to use ESA or SimAuto from julia, definitely check EasySimauto.jl for a julia wrapper of ESA.

Citation

If you use ESA in any of your work, please use the citation below.

@article{ESA,
  doi = {10.21105/joss.02289},
  url = {https://doi.org/10.21105/joss.02289},
  year = {2020},
  publisher = {The Open Journal},
  volume = {5},
  number = {50},
  pages = {2289},
  author = {Brandon L. Thayer and Zeyu Mao and Yijing Liu and Katherine Davis and Thomas J. Overbye},
  title = {Easy SimAuto (ESA): A Python Package that Simplifies Interacting with PowerWorld Simulator},
  journal = {Journal of Open Source Software}
}

Installation

Please refer to ESA's documentation <https://mzy2240.github .io/ESA/html/installation.html>__ for full, detailed installation directions. In many cases, ESA can simply be installed by:

python -m pip install esa

Simulator Compatibility

Currently ESA supports PW Simulator V17, V18, V19, V20, V21, V22 and V23.

Testing Coverage

The ESA team works hard to ensure ESA is well tested, and we strive for 100% testing coverage (sometimes we cannot due to lack of specific add-ons). The table below shows the most up-to-date testing coverage data for ESA, using coverage.

ESA's testing coverage as of 2023-05-31 (Git commit: 7180cc9)

Name	Num. Statements	Missing Lines	Covered Lines	Percent Coverage
esa/__init__.py	2	0	2	100
esa/saw.py	1206	2	1204	99.8342

License

Apache License 2.0

Contributing

We welcome contributions! Please read contributing.md.

esa's Issues

Contact PowerWorld, request ESA be listed on their website

To maximize our visibility, we should see if we can get PowerWorld to link to ESA in their documentation and on their website. We can do this as one of the last tasks related to the JOSS submission.

Add area number to the power flow result

The goal is to add the area number to the power flow result, so that the user is able to filter out the result based on the area.

Finalize JOSS paper, submit to JOSS

After receiving feedback from Professor's Overbye and Davis, make adjustments to the paper as needed. Then, submit to JOSS.

Remove the "modified" Texas 2000 bus case from the repository

The modified Texas 2000 bus case is huge (30MB), and we only use it for one test case: ProcessAuxFileTestCase. We should update that test case so that it does not depend on the modified case, and then delete the modified case from the repository.

While it may not seem like it's a big deal to have it around, the primary concern is that GitHub limits Git-LFS bandwidth for repositories unless you pay for more data. We should try to preemptively avoid this situation.

@mzy2240 - would you be willing to take this on after we get this next release out of the way?

Implement and test GetParametersMultipleElementFlatOutput method

Implement and test GetSpecificFieldMaxNum method

Implement and test GetCaseHeader method

Implement and test OpenCaseType method

Image fail to show up in the matplotlib example

https://mzy2240.github.io/ESA/html/snippets.html#plot-histogram-of-line-flows-with-matplotlib

Add AUX file parsing

This would allow faster reading of case data when the same case is opened multiple times.

Documentation-only release

I've updated the text around our citations to encourage people to check back later for an up-to-date citation. As such, I'd like to update the documentation both on GitHub and out on PyPi.

@mzy2240 - would you be willing to create a new release? No change to version number, no need to run the tests, etc. All we need is rebuilt documentation and an update on PyPi (so the top-level README gets updated).

Implement and test ChangeParametersMultipleElementFlatInput method

Bump version to 1.0.0 as last step for JOSS paper

Release version 1.0.1 after all the merging and documentation dust has settled

After we've finished revising the documentation, merging branches, etc., let's release the next version of ESA. This adds some new features.

This brings up an important point: We should probably start keeping a change log/ version log where we note what's been added. I propose we add a section to the documentation and create a .rst file for tracking this.

Another important point: We need to start using git tag to add tags to the repository corresponding to our PyPi releases.

Both of the above "important points" should be documented in the RELEASE_PROCESS.rst file.

So I guess this issue covers all these points :) Let me know if you need any assistance, and please pull me in for review as needed.

I think this can be done either before or after the JOSS paper text is finalized, but should be done before we submit to JOSS (just to get things as clean as absolutely possible).

Add more tests to make the coverage 100%

@blthayer Right now the coverage is about 96%. I checked the coverage reports and noticed the uncovered lines come from the newly-added exception handling for the old version of PW. Since you are more familiar with the mock module, can you add some tests for these exceptions?

Update to use PowerWorld's "Concise Variable Names" instead of "Variable Name"

It turns out that despite being listed first in the PowerWorld export, the "Variable Name" column contains old/deprecated variable names, and the "Concise Variable Name" column represents the new/preferred variable names.

This definitely isn't critical, but it's probably better to fix this earlier rather than later. This will impact tests, examples, etc.

We should also consider if we want to support both variable names.

Finally, this will require documentation updates (which should be made no matter what to make it clear to our users what we support).

setup.py should automatically update __version__ in __init__.py

This isn't working right now, but should.

Test with Pandas 1.0.*

Pandas recently updated to version 1.0. There are some backwards incompatible changes. We should run tests with both >=0.24 and 1.0.

@mzy2240 - would you have time for this? If not, it's not urgent, and I could do it sometime in the upcoming weeks.

Determine if we can add a "note" to citations for JOSS

A couple of our citations require notes (like "accepted, to be published after delayed conference takes place"). Brandon's thesis also needs a note since the full text won't be published for a while.

Enhance logging

The SAW class creates a logger, but then we never really log anything... We should add a reasonable amount of logging.

Give option to return and accept DataFrames with a MultiIndex

In many cases it would be very useful to have DataFrames use a MultiIndex. This index would use the key fields as the different index levels. For example, if you had a DataFrame with generator information, it would be nice if you could simply index in by bus number and generator ID.

Implement and test SendToExcel method

Implement and test ListOfDevicesFlatOutput method

Implement and test GetSpecificFieldList method

Implement and test TSGetContingencyResults method

Remove all links to the SimAuto functions page

PowerWorld is going to remove the page that lists all the SimAuto functions, so we should stop linking to it. Here's the link:

https://www.powerworld.com/WebHelp/#MainDocumentation_HTML/Simulator_Automation_Server_Functions.htm%3FTocPath%3DAutomation%2520Server%2520Add-On%2520(SimAuto)%7CAutomation%2520Server%2520Functions%7C_____3

Implement and test ChangeParameters method

Implement and test WriteAuxFile method

Implement and test ListOfDevicesAsVariantStrings method

change_and_confirm_params_multiple_element assumes numeric types

The method SAW.change_and_confirm_params_multiple_element assumes numeric data types, as can be seen by:

if not np.allclose(merged[cols_in].values, merged[cols_out].values):

This should be updated to separately compare numeric and string data.

Add documentation section for projects that use ESA and motivating use cases

Update RELEASE_PROCESS.rst file

Please fill in this file with details on the release process. I've started it.

Incorporate testing coverage into documentation

https://www.sphinx-doc.org/en/master/usage/extensions/coverage.html#sphinx.ext.coverage.coveragebuilder

Testing coverage - get to 100%

Use coverage module either through PyCharm or "raw" to assess testing coverage after all methods are complete. Add tests as necessary to get coverage up to 100%.

Run all tests with different versions of Python on different machines

Run tests with Python versions 3.5, 3.6, 3.7, and 3.8. @blthayer, @mzy2240, and @SmartJingJing123 should all run the tests with all the Python versions to ensure everything is working as expected.

This should be one of the last things we do before submitting the JOSS paper.

Contact PowerWorld re: TSGetContingencyResults

PowerWorld's documentation is inconsistent. Also issue for GetCaseHeader

Publicize and/or advertise ESA

After we have an official publication listed as our citation, we should start advertising ESA. One example might include making a "Powerglobe" announcement.

Update top-level README

• Make section for documentation
• Add one-liner for pip installation

Draft statement of need

Discuss why one would use an automation tool like ESA instead of directly using PowerWorld on its own.

Remove all non-implemented methods that are not SimAuto functions

Currently, we have some methods that raise a NotImplementedError but are not SimAuto functions. Rather, they're helpers for RunScriptCommand methods. As a first step, let's completely remove all these methods. We'll add some popular RunScriptCommand methods at the end.

Review and revise updated documentation

1. Git pull
2. Build documentation
3. Use web browser to view local documentation
4. Read through everything, fix any issues
5. Propose revisions that help us meet the JOSS criteria

Please make a branch for any changes, and use a pull request to get changes reviewed. Assign @blthayer to the pull request.

Create a function like get_key_fields_for_object_type, but for required fields

To create new objects in a model you need both the key fields and the required fields.

This should be as simple as making a helper function which is nearly identical to get_key_fields_for_object_type, but takes regular expressions as input. Then, get_key_fields_for_object_type would call this helper function, and a new function, get_required_fields_for_object_type would also call it.

Create function for setting SimAuto flags/constants

There are some SimAuto constants/flags such as "CreateIfNotFound" that users may want to set. We should create a single entry point for setting these.

Make path handling consistent in all of ESA

Currently, we don't handle all file paths consistently. Let's try this in the following order:

1. Remove ALL path conversion things (like posix_path stuff), run unittests
2. If that doesn't work, change ALL file paths to be converted to Windows file paths

Implement and test GetParameters method

Unify and update documentation

• Convert README file to rst
• Split README file out into different files so it can be included in the API documentation
• Include README content in API documentation
• Clean up and update documentation

Write draft of JOSS paper

Also create the .bib file.

Bad string conversion

Code:

from esa import SAW
import pandas as pd

base_name = r"D:\PythonSession8\tx2000_base.pwb"
fname_1 = r"D:\PythonSession8\tx2000_line"

df = pd.read_csv(r"D:\PythonSession8\CandidateLines.csv")

df.rename(columns = {'From Number': 'BusNum',
                    'To Number': 'BusNum:1',
                    'Ckt': 'LineCircuit',
                    'R': 'LineR',
                    'X': 'LineX',
                    'B': 'LineC'
                    }, inplace=True)

columns = ["BusNum", "BusNum:1", "LineCircuit", "LineR", "LineX", "LineC"]

sub_df = df[columns]

saw = SAW(FileName =base_name)
# Undocumented:
saw._pwcom.CreateIfNotFound = True

saw.RunScriptCommand("EnterMode(EDIT);")

saw.change_and_confirm_params_multiple_element(
    ObjectType='branch', command_df=sub_df)

Error:

C:\Users\smartgrid.ECELBR11501.002\AppData\Local\Programs\Python\Python37\lib\site-packages\pandas\core\frame.py:3509: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  self[k1] = value[k2]
Traceback (most recent call last):
  File "c:\Users\smartgrid.ECELBR11501.002\.vscode\extensions\ms-python.python-2019.11.49689\pythonFiles\ptvsd_launcher.py", line 43, in <module>
    main(ptvsdArgs)
  File "c:\Users\smartgrid.ECELBR11501.002\.vscode\extensions\ms-python.python-2019.11.49689\pythonFiles\lib\python\old_ptvsd\ptvsd\__main__.py", line 432, in main
    run()
  File "c:\Users\smartgrid.ECELBR11501.002\.vscode\extensions\ms-python.python-2019.11.49689\pythonFiles\lib\python\old_ptvsd\ptvsd\__main__.py", line 316, in run_file
    runpy.run_path(target, run_name='__main__')
  File "C:\Users\smartgrid.ECELBR11501.002\AppData\Local\Programs\Python\Python37\lib\runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "C:\Users\smartgrid.ECELBR11501.002\AppData\Local\Programs\Python\Python37\lib\runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "C:\Users\smartgrid.ECELBR11501.002\AppData\Local\Programs\Python\Python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "d:\PythonSession8\session8_solutions\saw_alternative.py", line 28, in <module>
    ObjectType='branch', command_df=sub_df)
  File "C:\Users\smartgrid.ECELBR11501.002\AppData\Local\Programs\Python\Python37\lib\site-packages\esa\saw.py", line 164, in change_and_confirm_params_multiple_element
    ObjectType=ObjectType)
  File "C:\Users\smartgrid.ECELBR11501.002\AppData\Local\Programs\Python\Python37\lib\site-packages\esa\saw.py", line 1187, in clean_df_or_series
    obj[nn_cols] = obj[nn_cols].apply(lambda x: x.str.strip())
  File "C:\Users\smartgrid.ECELBR11501.002\AppData\Local\Programs\Python\Python37\lib\site-packages\pandas\core\frame.py", line 6928, in apply
    return op.get_result()
  File "C:\Users\smartgrid.ECELBR11501.002\AppData\Local\Programs\Python\Python37\lib\site-packages\pandas\core\apply.py", line 186, in get_result
    return self.apply_standard()
  File "C:\Users\smartgrid.ECELBR11501.002\AppData\Local\Programs\Python\Python37\lib\site-packages\pandas\core\apply.py", line 292, in apply_standard
    self.apply_series_generator()
  File "C:\Users\smartgrid.ECELBR11501.002\AppData\Local\Programs\Python\Python37\lib\site-packages\pandas\core\apply.py", line 321, in apply_series_generator
    results[i] = self.f(v)
  File "C:\Users\smartgrid.ECELBR11501.002\AppData\Local\Programs\Python\Python37\lib\site-packages\esa\saw.py", line 1187, in <lambda>
    obj[nn_cols] = obj[nn_cols].apply(lambda x: x.str.strip())
  File "C:\Users\smartgrid.ECELBR11501.002\AppData\Local\Programs\Python\Python37\lib\site-packages\pandas\core\generic.py", line 5175, in __getattr__
    return object.__getattribute__(self, name)
  File "C:\Users\smartgrid.ECELBR11501.002\AppData\Local\Programs\Python\Python37\lib\site-packages\pandas\core\accessor.py", line 175, in __get__
    accessor_obj = self._accessor(obj)
  File "C:\Users\smartgrid.ECELBR11501.002\AppData\Local\Programs\Python\Python37\lib\site-packages\pandas\core\strings.py", line 1917, in __init__
    self._inferred_dtype = self._validate(data)
  File "C:\Users\smartgrid.ECELBR11501.002\AppData\Local\Programs\Python\Python37\lib\site-packages\pandas\core\strings.py", line 1967, in _validate
    raise AttributeError("Can only use .str accessor with string " "values!")
AttributeError: ('Can only use .str accessor 

Merge 'master' into 'develop'

@mzy2240

Our master and develop branches have diverged a bit. For example, the develop branch still has the ESA version at 0.6.3. So, we should merge 'master' into 'develop'.

Caveats:

• We need to remove __version__ from __init__.py
• Potential merge conflicts with the documentation: It might be simplest to remove all the built documentation from both branches, then re-build in 'master', then merge into 'develop'.
• Other issues?

Update the minimum version requirement for numpy

Update the minimum version requirement for numpy in the setup