upb-lea/gym-electric-motor

Gym Electric Motor (GEM): An OpenAI Gym Environment for Electric Motors

gym-electric-motor's Issues

State variable 'u_sup' is insufficiently documented

The state variable 'u_sup' is given for every of the supported motor environments. However, no description of its meaning can be found in the documentation.

E.g. as found in the 'emotor-dc-extex-disc-v1' class:
State Variables: ``['omega' , 'torque', 'i_a', 'i_e', 'u_a', 'u_e', 'u_sup']``

But there is no explanation in the documentation of the DC motor:

===========================================================
Limits / Nominal Value Dictionary Entries:
-------- -----------------------------------------------------------
Entry Description
===========================================================
i_a Armature current
i_e Exciting current
omega Angular Velocity
torque Motor generated torque
u_a Armature Voltage
u_e Exciting Voltage
===========================================================

For the voltage supply, the variable 'i_sup' is defined:

i_sup(float): Supply current floating into the system.

'u_sup' therefore most probably refers to the supply voltage as provided by the power supply.
Since the supply voltage is assumed to be constant and normalized, the state vector will always show 'u_sup=1.0' (at least for all the scenarios with constant supply voltage), which may allow to omit this state variable completely.

Define an interface to pass signals to arbitrary subconverters

The doubleConverter classes allow the usage of two converters within one environment. In the backend, the signals are passed to the doubleConverter in a list [a, b], where the list is disassembled and then passed on to the two subconverters: sub0 gets [a], sub1 gets [b].
This practice only works as long as sub0 and sub1 are fed by 1D-Signals, three phase inverters can not be controlled in this fashion. The fix for this also needs to work for a planned "MultiConverter" class, where an arbitrary number of different subconverters is to be controlled.

Solution suggestions:
1)
pass a list [a, b, c, d, e], save the starting index for each subconverter, e.g.:
sub0: starting index 0, input space 1 (resulting in using signal a)
sub1: starting index 1, input space 3 (resulting in using signals b,c,d)
sub2: starting index 4, input space 1 (resulting in using signal e)
2)
pass a list of lists [[a], [b, c, d], [e]],
sub0 uses element 0: [a]
sub1 uses element 1: [b, c, d]
sub2 uses element 2: [e]

This problem showed when I was trying to set up the doubleConverter Class with three phase inverters. It was caused by the corresponding "convert"-method:

def convert(self, i_out, t):
    # Docstring in base class
    u_in = []
    for subconverter, i in zip(self._subconverters, i_out):
        u_in += subconverter.convert([i], t)
    return u_in

Extensions to graphics output

DDPG examples require unofficial keras-rl2 hotfixes

To be fixed by either

• Implementing own DDPG algo in examples folder
• Use a different neural network framework than keras-rl2
• update setup.py to keras_rl2 and annotate readme in examples on how to apply fixes to keras-rl2

Model DC-link as RC-element for time-varying behavior

Currently, all inverters are voltage-source inverter with idealized DC-link behavior i.e. it is assumed that the DC-link voltage is fixed and given. However, in realistic applications (automotive/automation) the DC-link is a capacitor fed from an external source (battery, rectifier from grid,...) and, therefore, the DC-link voltage is varying and time-dependent.

In order to make the GEM envs. a little bit more realistisch, the DC-link should be modeled as a RC-element (capacitor with internal ohmic resistance) which is fed from an ideal DC-voltage source. This results in an additional ODE which has to be transfered within GEM as part of the overall ODE system (including mechanical and electric motor ODEs).

Moreoever, this feature should be activatable and deactivatable via a corresponding parameterization

Renaming of PhysicalSystem Class

In my opinion, the name PhysicalSystem could be improved for more clarity on its purpose.

The PhysicalSystem is the part to combine all models of the subsystems (Motors, Loads, Converters etc.) into one and to execute the simulation.

Therefore I would suggest to name it "SystemModel" or "CompoundSystem".

I would also appreachiate more and better suggestions. 😃

Code Speed Investigation and Improvement

• Use a profiling tool (e.g. built-in in Pycharm) to investigate on the time consumption for the different toolbox parts for the provided examples or a dummy controller input
• Identify possible potentials for improving the code in terms of processing time
• Evaluate alternative implementations to speed up the code (e.g. on specific test branch)

Console Printer is too verbose

Currently the ConsolePrinter spams states, references and reward information step by step, but this is no useful visualization.

A more readable and descriptive concept must be elaborated.

Moreover, documentation is missing for the ConsolePrinter.

Introduce PMSM example

Based on pre-investigation for test bench use, an example on DDPG training for PMSM should be added to the repo.

Documentation stray inductance

documentation of Inductionmotor stray inductance in doctrings and online documentation does not match with dictionary keys

Change default variable order in the dq-frame

Coordinates in the dq-frame are given the "wrong" way round.

E.g. from the documentation of the Synchronous Motor:

Args:
state: The current state of the motor. [i_sq, i_sd, epsilon]
omega: The mechanical load
u_qd: The input voltages [u_sq, u_sd]

To comply with the convention and minimize error potential, it should be changed to d before q: [u_d, u_q], [i_d, i_q].

Change default load of PolynomialStaticLoad to 0

The default load for most of the environments should be changed to 0 (no load).
For the examples and environments like the DcSeries and DcShunt Motors a little Load Should be added again for better control.

Extend converter class for multi level converter

Up to now, only simple two level converters are covered by the toolbox. It would be interesting to add multilevel converters using basic building blocks in a modular fashion.

matplotlib Attributerror figure.canvas.update in motor_visualization.py

for a conda enviroment with packages installed with the

conda install

command, no error occurs.

Using a conda enviroment or enviroments created for example with venv and packages (especially matplotlib) installed with pip, the following error occurs:

self._update_figure = self._figure.canvas.update (line 216)

no such attribute as update

With the

conda install

command, additional packages are installed. Eventually some packages are missing when installing the basic packages with pip.

Rendering when episode terminates due to constraint violations

If an episode terminates due to constraint violations before the maximum number of steps is reached it has to be guaranteed that the render command covers the entire simulation time until the termination event. In particular, it could occur that the episode termination occurs very early before the render command is executed regularly the first time in a given episode. In this special case the render function should still be called to enable visualization.

Choose a consistent frame for the output variables omega and epsilon

In the vector of output variables, the angular velocity omega stands for the mechanical rotor velocity.
The angle epsilon, however, refers to the electrical rotor angle (which admittedly has advantages for many calculations that are performed in the backend).

Maybe it would be more intuitive to output both of these geometrical values in reference to the same frame (either both mechanical or both electric).

Write an example RL application using Stable Baseline3

Write an example application with the deep Q-network (DQN) implementation of Stable Baselines 3. Solve the current control problem of the permanent magnet synchronous motors (PMSM).

• https://github.com/DLR-RM/stable-baselines3
• Use mean absolute error (MAE) reward function
• Single discrete action variable: fundamental voltage vector with 7 options
• Cont. state variables

Take notes on the usability of the library and other notable findings. Prepare a small presentation of your results.

ConsolePrinter not initialized properly

Currently the ConsolePrinter is unusable. Rewriting pi_series_omega_control.py to use the ConsolePrinter as visualization instead of the MotorDashBoard leads to an error caused by the ConsolePrinter's prints relying on a missing object variable:

Traceback (most recent call last):
File "test_console.py", line 38, in
env.render()
File "/home/darius/Documents/Uni/Projekt/gym-electric-motor/gym_electric_motor/core.py", line 213, in render
self._visualization.step(self._state, self._reference, self._reward, self._reset_required)
File "/home/darius/Documents/Uni/Projekt/gym-electric-motor/gym_electric_motor/visualization/console_printer.py", line 34, in step
f'State {state * self._limits} \n'
TypeError: unsupported operand type(s) for *: 'float' and 'NoneType'

This is because self._limits is not initialized for the ConsolePrinter. This can be done with the set_physical_system function of the ConsolePrinter. I fixed this problem by adding the following line in the init of ElectricMotorEnvironment in core.py:

self._visualization.set_physical_system(self._physical_system)

I could push this fix with my solution to issue #25 if this is wanted.

build fails due to environments and visualization

Currently unit tests fail for the environment class and visualization.
To be Resolved.

see:
https://travis-ci.org/upb-lea/gym-electric-motor

It seems to be connected to QT5:
AttributeError: 'FigureCanvasAgg' object has no attribute 'update'

Develop a new "GridMotorLoad"-Class

Especially the Doubly Fed Induction Motor (DFIM) is usually operated with its stator directly connected to the power grid.
The implemented SCML (SupplyConverterMotorLoad) class contains a DC-link voltage state that does not fit into this grid-connected scenario. Moreover, grid voltage amplitude and frequency are predefined and should be untouchable for the operating controller.
Therefore, a GridMotorLoad class would allow to simplify the state space concerning these points.

The rotor of the DFIM receives a control voltage that is provided via a converter, so here the SCML class would (in principle) fit the problem. However, it has to be considered that stator and rotor belong to the same motor which is connected to the same load.

test_physical_systems.py needs comments

test_physical_systems.py has absolutely no comments. It is very difficult to get anything out of a failing assertion if there is no information about which assumption was made for that assertion. Other test files seem to provide at least some commentary though I don't know how detailed they are.

My recommendation is to either write failure messages for assertions or put some extensive comments in the code.

Pip install fails with Python 3.8

With python 3.8, pip install fails due to PyQT5's dependency to "sip".
Pip can't find a version that covers all version constraints of sip.

Torque Limit of PMSM is not universally correct

The implemented torque limit calculation of the PMSM is only correct for the assumption of surface mounted magnets (so called surface permanent magnet SM - SPMSM). In this case we have L_d == L_q.

For the case of interior permant magnets (IPMSM) we have L_d != L_q, which leads to a different torque limit.

Add GEM-Callbacks

keras-rl (and also keras itself) use callbacks to add user-specific functionality into the training process. In general, the keras-rl callbacks can also be used to modify the GEM-environments.

Nonetheless, to enable fine-granular callbacks and to gather independence from keras-rl callbacks, additional GEM-Callbacks could be useful.

[v0.2.1] Example script: "pi_series_omega_control.py" fails

Maybe a reset() is not triggered in the environment upon episode termination due to a violation. Exception is thrown in the during the control.

Traceback (most recent call last):
File "/home/praneeth/projects/gym-electric-motor/examples/pi_series_omega_control.py", line 39, in
(state, reference), reward, done, _ = env.step(action)
File "/home/praneeth/projects/gym-electric-motor/gym_electric_motor/core.py", line 228, in step
raise Exception('A reset is required before the environment can perform further steps')
Exception: A reset is required before the environment can perform further steps

Evaluate performance / usability of Stable Baseline RL Package + OpenAI Spinning up

Transfer DDPG-based PMSM current control example based on Keras-RL2 to the standard RL packages

• https://github.com/openai/spinningup
• https://github.com/hill-a/stable-baselines

Hence, we require a standard agent able to work on cont. actions and cont. states.

module not found

I'm trying to install the package but I get:
import gym_electric_motor as gem
ModuleNotFoundError: No module named 'gym_electric_motor'

Default Voltage Supply values are being overwritten in physical_systems.py

Currently the default settings in voltage supplies for the supply voltage are being overwritten by this code in physical_systems.py. The critical part is in the else-block:

       if 'u_sup' in kwargs.keys():
            u_sup = kwargs['u_sup']
        else:
            u_sup = self._electrical_motor.limits['u']
        self._supply = instantiate(vs.VoltageSupply, supply, u_nominal=u_sup, tau=tau, **kwargs)

It should be discussed if this behavior is wanted. For example the default values suggested in issue #48 for the new AC supplies are being overwritten by this code.

On_off_Controller with MotorDashboard plotted_variables='all' is not plotting anything

Running pi_series_omega_control.py with a MotorDashboard(plotted_variables='all', visu_period=1) and with an on_off_controller, is not plotting any progress on the diagrams.

Running the script with another controller or with a certain set of plotted_variables will work normally though.

Write an example RL application using Keras RL2

Write an example application with the deep Q-network (DQN) implementation of Keras RL2. Solve the current control problem of the permanent magnet synchronous motors (PMSM).

• https://github.com/wau/keras-rl2
• Use mean absolute error (MAE) reward function
• Single discrete action variable: fundamental voltage vector with 7 options
• Cont. state variables

Take notes on the usability of the library and other notable findings. Prepare a small presentation of your results.

Agenda meeting 2020-06-15

• Short presentation of your provided code increments (10 minutes max, e.g. by presentation slides or live demo)
• Administrative sprint outlook: are your individual assigned issues fully covered? What parts need / should be transfered to next sprint phase? What were the specific obstacles for you in order to solve an issue within the first sprint phase?

Add ConstraintMonitor

To enable more complex constraints than simple limit observation (e.g. the PMSMs squared current observation) a new ConstraintMonitor Class could be useful, that is independent from the RewardFunction, that currently monitores the limits. A list of (soft and hard-) constraints could be passed to the Constraint Monitor.

Hard Constraints directly terminate an episode.
(Optional:) Maybe also soft constraints could be interesting, that just degrade the reward if some quantities reach undesired operating regions.

Is has to be discussed, if the limit_violation reward (and the degraded reward in case of soft constraints) should be calculated in the reward function or in the ConstraintMonitor.

Refactoring of simple_controllers.py

The mentioned file contains basic, expert-driven feedback controllers for different motors. The file and its classes have historically evolved and require a major revision. Main issues are:

• in-code and API documentation is only very little. More detailed explanations required.
• a well-defined class structure and an allocation to the available motors is missing. For example the class "CascadedPI-Controller" is mentioned here, which can be applied de facto only to DC motors, but from the name definition it can be assumed that a general controller for different motors can be found here.
  • It would be good to derive specific controllers for individual motor/inverter/action-space classes or similar control tasks on the basis of generally valid basic controller definitions, as they already exist in some cases. In the end, it should in any case be clear which controller class can be used for which control problem (current, speed, ...) with which motor/inverter type.
• The controllers with integral action are missing anti reset-windup measures to ensure proper operation when their output is saturated.
• Feed forward compensation measures such as dq decoupling for three-phase motors or also induced voltage compensation for DC-motors is missing. Could be integrated as own classes which can be turned on and off by the user as a selectable controller features.
• Also the voltage reference conversion from dq to phase (abc/rst) coordinates is lacking a zero voltage injection / third harmonic injection in order to fully utilize the inverter voltage.

Error evaluating pmsm model

Hello team,

When I am using the 'emotor-pmsm-disc-v1' for speed control or current control, I am getting the following error:

Traceback (most recent call last):
File "test1.py", line 34, in
sigma_range=(-1,1))
File "C:\Users\Soumava\Anaconda3\envs\gem\lib\site-packages\gym\envs\registration.py", line 156, in make
return registry.make(id, **kwargs)
File "C:\Users\Soumava\Anaconda3\envs\gem\lib\site-packages\gym\envs\registration.py", line 101, in make
env = spec.make(kwargs)
File "C:\Users\Soumava\Anaconda3\envs\gem\lib\site-packages\gym\envs\registration.py", line 73, in make
env = cls(_kwargs)
File "C:\Users\Soumava\PycharmProjects\gym-electric-motor-master\gym_electric_motor\envs\gym_pmsm\perm_mag_syn_motor_env.py", line 88, in init
super().init(tau=tau, converter=converter, **kwargs)
File "C:\Users\Soumava\PycharmProjects\gym-electric-motor-master\gym_electric_motor\envs\gym_pmsm\perm_mag_syn_motor_env.py", line 21, in init
physical_system, reference_generator=reference_generator, reward_function=reward_function, **kwargs
File "C:\Users\Soumava\PycharmProjects\gym-electric-motor-master\gym_electric_motor\core.py", line 145, in init
self._reference_generator.set_modules(self.physical_system)
File "C:\Users\Soumava\PycharmProjects\gym-electric-motor-master\gym_electric_motor\reference_generators\subepisoded_reference_generator.py", line 45, in set_modules
upper_margin = (ps.nominal_state[rs] / ps.limits[rs])[0]
IndexError: index 0 is out of bounds for axis 0 with size 0

Please advise.

Thank You

Add comments to examples if default motor parameter are overwritten

In some of the provided examples the default motor parameters as defined within the toolbox are overwritten. A detailed comment to each of those examples should be added in order to explain why this is done.

Extend supply class for more than DC-voltage inputs

Goal of issue: build extended supply class to provide:

• AC 1-phase supply
• AC 3-phase suppy

The voltage source provides a fixed voltage signal with few parameters (e.g. amplitude, frequency, phase angle). Therefore, we require to insert a dummy converter class ('NoConverter') which directly throughputs the voltage supply signals to the motor. Can be used for motor simulation. . Or could be even used rectifying power solutions.

Preparation for #17

simple_controllers need an overhaul

• no consistent signatures among controllers
• alternating application of numpy and math std package
  • multi-reference control not possible and efficiency loss is a consequence

Extend motor models for additional physical effects

Extend motor model framework covering e.g.

• iron (cross-)saturation either by analytical saturation curves or numerical table values
• rotor angle-dependent inductances
• iron losses by simplified, lumped and frequency-dependent iron loss resistance
• add simple thermal models to cover varying component temperatures which will change the electrical parameter values (e.g. resistances, permanent magnet flux)

Similarly, a simple inverter loss model (conduction and switchting losses) could be added.

Visualization empty

from gym_electric_motor.reference_generators import WienerProcessReferenceGenerator
from gym_electric_motor.visualization import MotorDashboard
import gym_electric_motor as gem

env = gem.make('emotor-dc-series-disc-v1',max_episode_steps=10000,
        state_filter=['i'],
        visualization=MotorDashboard(update_period=2e-1, visu_period=1, plotted_variables=['omega', 'i', 'u']),
        converter='Disc-4QC',
        load_parameter=dict(a=0, b=.1, c=1.1, j_load=0.4),
        ode_solver='euler', solver_kwargs={},
        reference_generator=WienerProcessReferenceGenerator(reference_state='i',
                                                            sigma_range=(5e-4,
                                                                         5e-3)))
env.reset()
for i in range(1000):
    env.step(env.action_space.sample())
env.render()

Why is the resulting visualization empty?

Agenda meeting 2020-06-22

• Closing of sprint 01
• Feedback debate on sprint 01 -> good/bad things, improvement suggestions on organizational level
• Content assignment for sprint 02

Interfacing motor state initializer at beginning of each episode

Currently the motor states (e.g. motor currents, rotational speed) are always set to zero at the start of each new epsiode. In order to achieve more flexible episodes and, therefore, learning sequences it may be interesting to allow also other kinds of initializations (e.g. randomly drawn values from some distribution within the nominal operating range of a given drive) by adding a suitable interface function.

This initialization interface may cover the three following cases:

• All states at zero (current default)
• All states randomly drawn from a Gaussian, or uniform or any other distribution (including review if the resulting initialization is within the nominal drive's operating range in order to prevent initializations which are directly terminating)
• Manual access by the user to allow definition of specific initialization conditions

Refactoring of render visualization

Currently the render command is tightly fixed to plot only the relevant env. states over time including state references and constraints (if applicable). Hence, the render function should be refactored and extended in order to:

• plot also the instant reward per time step
• plot also the actions applied to the environment at each time step

Reward functions do not consider episode lengths

It might be useful to be able to define a limit_violation reward function, that is scheduled according to how many actions were already taken, especially for finite-length episodes.

A use case could be, e.g. to continually reduce limit violation punishment the older the episode gets.

To make this possible, RewardFunction must count calls to itself, keep track of them, and utilize the count during _limit_violation_reward().
On reset() this count has to be set to 0.

[improvement] register environments the recommended way

This would force some restructuring. i am not sure if its worth it, but i still wanted to make the devs aware of this fact.

https://github.com/openai/gym/blob/master/docs/creating-environments.md

Add Wrappers / StateActionProcessors for the PhysicalSystems

Wrappers to the physical systems could be useful for user specific pre- and postprocessing of the systems states and actions without the need of multiple options and functionality directly inside the PhysicalSystems classes.

For example, the squared current
i ^ 2 = i_sd ^ 2 + i_sq ^ 2
and sin(epsilon) cos(epsilon)
could be added to the systems state for a PMSM.
Furthermore, functionality like the dq-control or the normalization of the states could be outsourced to such a wrapper and turned on and off by the user easily.

In contrast to a wrapper for the environments that gym already supports, a wrapper on the PhysicalSystem-level could process states that can be rewarded, referenced, monitored and visualized.

Agenda meeting 2020-06-08

Please insert agenda bullet points to be discussed:

• General schedule within sprint

Unify arguments "converter" and "subconverters"

in case of multiple converters, both arguments "converter" and "subconverters" have to be set.
It would be more user friendly to just go with one argument "converter", which can be either a string or a list of strings, in case of multiple converters.

Extend the SpeedLoad options

The ConstantSpeedLoad class could use extension:

• at the moment speed is defined once and is kept like this in all training episodes

• when learning current control, it would be useful if speed could change from one episode to the next

  • the change can be e.g. random or user defined

• for validation of current controllers, it would be useful if speed could follow a given function

  • "SpeedProfile", can maybe be passed to the env in the form of a function: def profile(t) -> return omega at time t
    • (but I am not sure if this is the best practice)

How to generate safety critical reference lines?

WienerProcessReferenceGenerator(reference_state='i',
                                                            sigma_range=(5e-2, 5e-3)))

setting larger sigma values will result in running in a limit (limit_margin variable). i found no way to overwrite it.

If the reference process is far away from the savety margins, it is very unlikely that any controller will actually gerneate a unsave policy because it must get extremely far away from the reference process in order to brake.

All test did not pass

I ran the pytest and there were 16 errors and 1096 warnings.

I am getting the following erreor in Ref generator:
AssertionError: Reference is not equal the expected one
E assert [-0.097785086...91553687, ...] == [-0.097785086...91553687, ...]
E At index 7 diff: -0.13998424344384483 != -0.13998424344384486
E Use -v to get the full diff

Also, is the warning normal? What TensorFlow and Keras version is recommended? Please suggest.

Thank You