ricnogfer/pypsatopo

PyPSATopo is a tool that allows generating the topographical representation of any arbitrary PyPSA-based network

Description

PyPSATopo is a tool that allows generating the topographical representation of any arbitrary PyPSA-based network (thanks to the DOT language). Besides easing the understanding of a network by providing its graphical representation, the tool helps debug it given that broken links and missing buses are shown in (slightly) different shapes and colors. Technically speaking, PyPSATopo can be thought of as a reverse engineering tool for PyPSA-based networks.

To get a quick overview of the capabilities of PyPSATopo, simply launch it in a terminal as follows:

python pypsatopo.py

This will create a PyPSA-based network made of the following components:

# create dummy (PyPSA-based) network
network = pypsa.Network(name = "My Dummy Network")

# add some dummy components to dummy network
network.add("Bus", "oil")
network.add("Bus", "electricity")
network.add("Bus", "transport")
network.add("Generator", "oil", bus = "oil")
network.add("Generator", "solar", bus = "electricity")
network.add("Load", "vehicle", bus = "transport")
network.add("Store", "battery", bus = "electricity")
network.add("Link", "ICEV", bus0 = "oil", bus1 = "transport")
network.add("Link", "BEV", bus0 = "electricity", bus1 = "transport")

... as well as generate the corresponding topographical representation of the network in the SVG format:

Installation

PyPSATopo can be installed in the machine using pip, a package management system for Python. To install the tool using pip, open a terminal and execute the following:

pip install pypsatopo

PyPSATopo leverages from several components to accomplish its functionalities, namely: Python, PyPSA, Pandas and Dot (from Graphviz). Consequently, these should be installed before running PyPSATopo. While PyPSA and Pandas are automatically installed by PyPSATopo in case they are missing, Dot must be manually installed by the user (see download for additional details). Alternatively, in case of working with Conda (or Miniconda), Dot may be installed automatically by opening a terminal and executing the following:

conda install -c conda-forge python-graphviz

As a reference, PyPSATopo is known to work correctly with the following versions of the components:

• Python 3.10.8

• PyPSA 0.21.3

• Pandas 1.5.3

• Dot 2.40.1

In addition, PyPSATopo should work in any platform (i.e. operating system) as long as the components that this tool depends on are supported on the target platform. As a reference, the tool is known to work correctly in Windows, Linux and macOS.

Usage

PyPSATopo can be used in two different ways to generate the topographical representation of a PyPSA-based network, either through its application programming interface or a command-line interface.

To use PyPSATopo through its application programming interface, the following (generic) recipe should be followed:

# import PyPSATopo module
import pypsatopo

# create (PyPSA-based) network named 'my_network'
my_network = pypsa.Network()

# add components to network 'my_network'
# (...)

# generate topographical representation of network 'my_network' in the SVG format
pypsatopo.generate(my_network)

To use PyPSATopo through a command-line interface, execute the following in a terminal (where my_network.nc refers to the NetCDF file containing the PyPSA-based network):

python pypsatopo.py my_network.nc

In addition, to see all the valid parameters accepted by the tool, specify parameter --help when executing it in a terminal. In other words:

python pypsatopo.py --help

PyPSA Components

Currently, PyPSATopo supports some of the most important PyPSA components, namely: Bus, Generator, Load, Store, Link and Line. These are graphically represented by PyPSATopo as follows:

• Bus (fundamental component of the network, to which components like loads, generators and stores attach. It enforces energy conservation for all elements feeding in and out of it - i.e. like Kirchhoff’s Current Law).

• Generator (attaches to a single bus and can feed in power. It converts energy from its carrier to the carrier-type of the bus to which it is attached).

• Load (attaches to a single bus and consumes power as a PQ load).

• Store (attaches to a single bus and stores energy only (it cannot convert between energy carriers). It inherits its energy carrier from the bus to which it is attached).

• Link (component for controllable directed flows between two buses with arbitrary energy carriers. It can have an efficiency loss and a marginal cost).

• Line (represents transmission and distribution lines. It can connect either AC buses or DC buses).

Functionalities

As stated previously, PyPSATopo is a tool that allows generating the topographical representation of any arbitrary PyPSA-based network. It basically allows to reverse engineer such type of network to ease its understanding. To that end, PyPSATopo provides several functionalities to cover as many use-cases as possible. These functionalities are described below and their usage exemplified using both the application programming interface and the command-line interface.

• When generating the topographical representation of a network and in case parameter file_output is not specified, PyPSATopo saves the output either in (1) a file named topography with an extension equal to the file format if the tool is used through its application programming interface or (2) a file named as the network file with an extension equal to the file format if the tool is used through a command-line interface. To specify the file name where to save the output, set parameter file_output with the appropriate value. As an example, the following generates the topographical representation of a network and saves the output in a file named my_network.svg:

  pypsatopo.generate(my_network, file_output = "my_network.svg")

  python pypsatopo.py my_network.nc --file-output my_network.svg

• PyPSATopo is able to generate the topographical representation of a network in different formats, namely: SVG, PNG, JPG, GIF and PS. To specify the format, set parameter file_format with the appropriate value. Otherwise, in case the parameter is not set, the default format is SVG. As an example, the following generates the topographical representation of a network in the GIF format:

  pypsatopo.generate(my_network, file_format = "gif")

  python pypsatopo.py my_network.nc --file-format gif

• By default, PyPSATopo generates the topographical representation of the entire network. This might be particularly overwhelming (visually speaking) depending on the complexity of the network - see PyPSA-Eur network topographical representation to get an idea. To mitigate this, parameters focus and neighbourhood may be utilized (in combination) to focus on a particular aspect/segment of the network. The former tells PyPSATopo which bus to start visiting, while the latter tells how much neighbourhood (around the bus) should be visited (in other words, how much (indirect) components attached to the bus should be included in the representation). For instance, setting parameters focus = "DK1 0 low voltage" and neighbourhood = 2 (which focuses/starts on bus DK1 0 low voltage and includes all the components attached to it up to a maximum neighbourhood degree of 2) yields this result upon generating PyPSA-Eur network topographical representation. As an example, the following generates the topographical representation of a network focusing/starting on bus co2 atmosphere and including all the components attached to it up to a maximum neighbourhood degree of 3:

  pypsatopo.generate(my_network, focus = "co2 atmosphere", neighbourhood = 3)

  python pypsatopo.py my_network.nc --focus "co2 atmosphere" --neighbourhood 3

  Alternatively, parameters focus and neighbourhood may be set with a list of buses and respective neighbourhoods to enable focusing on several aspects/segments of the network at the same time (as opposed to just one bus and respective neighbourhood). Setting these parameters with a list of buses/neighbourhoods (instead of a scalar) enables PyPSATopo to visit and combine the output of each bus into one single topographical representation, potentially generating a disjoint union of graphs. As an example, the following generates the topographical representation of a network by focusing/starting on buses my_bus0 and my_bus1, and including all the components attached to these buses up to a maximum neighbourhood degree of 2 and 3, respectively:

  pypsatopo.generate(my_network, focus = ["my_bus0", "my_bus1"], neighbourhood = [2, 3])

  python pypsatopo.py my_network.nc --focus my_bus0 my_bus1 --neighbourhood 2 3

• Just like for links with positive efficiencies, PyPSATopo represents links with negative efficiencies with a line going from bus0 to, e.g., bus1 and an arrow at the end of the line pointing to the latter. In case of the need to invert the sense of the arrow (i.e. to point to bus0 instead) when dealing with negative efficiencies, set parameter negative_efficiency = False. As an example, the following generates the topographical representation of a network with no negative efficiencies (i.e. all links with negative efficiencies will have their arrows inverted):

  pypsatopo.generate(my_network, negative_efficiency = False)

  python pypsatopo.py my_network.nc --no-negative-efficiency

• By default, all broken links and missing buses are excluded from the topographical representation of a network. To include and show these in (slightly) different shapes and colors in the representation, set parameter broken_missing = True. As an example, the following generates the topographical representation of a network where all broken links and missing buses are included in it:

  pypsatopo.generate(my_network, broken_missing = True)

  python pypsatopo.py my_network.nc --broken-missing

• To color a certain component (namely: bus, generator, load, store or line) in function of its carrier, set parameter carrier_color with a dictionary containing key-value pairs, where key is the name of a carrier and value is a color assigned to it. Acceptable colors are defined here. As an example, the following generates the topographical representation of a network with its components colored in red, green or blue whenever their carriers are my_carrier0, my_carrier1 or my_carrier2, respectively:

  pypsatopo.generate(my_network, carrier_color = {"my_carrier0": "red", "my_carrier1": "green", "my_carrier2": "blue"})

  python pypsatopo.py my_network.nc --carrier-color my_carrier0 red my_carrier1 green my_carrier2 blue

  Alternatively, in case carrier_color is set to True (instead of a dictionary), PyPSATopo automatically assigns a new color to each distinct carrier found in the network and colors all the components associated to the carrier with this color. As an example, the following generates the topographical representation of a network with its components colored in function of their carriers:

  pypsatopo.generate(my_network, carrier_color = True)

  python pypsatopo.py my_network.nc --carrier-color

• In case fine-grained selection/visiting logic is needed, parameters bus_filter, generator_filter, load_filter, store_filter, link_filter, line_filter and carrier_filter may be utilized in combination or separately. These parameters are expected to be set with (user-defined) regular expressions. While parameters bus_filter, generator_filter, load_filter, store_filter, link_filter and line_filter tell PyPSATopo which buses, generators, loads, stores, links and lines to include/exclude, respectively, the last two also specify which links or lines may be visited (or not) upon generating the topographical representation of a network. Moreover, parameter carrier_filter may be utilized to include/exclude any component in function of its carrier. As an example, the following generates the topographical representation of a network where only the generators named wind, solar or CHP are selected (and all other generators are excluded):

  pypsatopo.generate(my_network, generator_filter = "wind|solar|CHP")

  python pypsatopo.py my_network.nc --generator-filter "wind|solar|CHP"

• By default, excluded components (due to, e.g., filtering) are not shown in the topographical representation of a network. In certain situations, however, it might be useful to understand where selected (included) components are located in the full representation (i.e. among excluded components). To show selected components in the topographical representation of a network among excluded components, set parameter context = True. While selected components are shown with the appropriate colors, excluded components are shown with faded colors (to distinguish them from the formers, visually speaking). As an example, the following generates the topographical representation of a network where only the loads containing the word agriculture in their names are selected (and all other loads are displayed with faded colors):

  pypsatopo.generate(my_network, load_filter = ".*agriculture.*", context = True)

  python pypsatopo.py my_network.nc --load-filter .*agriculture.* --context

• Given that it may take some time to process a complex network, PyPSATopo is capable of displaying log messages while processing such a network. Log messages not only facilitate understanding of the stage at which the tool is in the processing pipeline but also potential issues that the network may have. To enable PyPSATopo to display log messages, set parameter log = True. Otherwise, in case the parameter is not set, the tool behaves quietly by default (i.e. no log messages are displayed). As an example, the following displays log messages while generating the topographical representation of a network:

  pypsatopo.generate(my_network, log = True)

  python pypsatopo.py my_network.nc --log

  While parameter log tells PyPSATopo to display all log messages (independently of information or warning log messages), parameters log_info and log_warning tell PyPSATopo to display only information or warning log messages, respectively. As an example, the following displays only information log messages (and all warning log messages will not be displayed) while generating the topographical representation of a network:

  pypsatopo.generate(my_network, log_info = True)

  python pypsatopo.py my_network.nc --log-info

• While PyPSATopo strives to generate the topographical representation of a network with the most common/expected graphical features, the tool is flexible enough to let each user adjust/personalize the representation by setting PyPSATopo global variables with appropriate values. As an example, the following generates the topographical representation of a network with a background in blue (instead of transparent):

  pypsatopo.BACKGROUND_COLOR = "blue"
  pypsatopo.generate(my_network)

Support

PyPSATopo is actively developed and maintained by the Energy Systems Group at Aarhus University (Denmark). Please open a ticket here in case a bug is found or a feature is missing in this tool.

pypsatopo's Issues

Refactor PyPSATopo metrics displayed in tooltips

• Add capital cost and marginal cost information in a multi-link tooltip.
• Display the names of the buses that a multi-link connects to (@BertoGBG).

Enable PyPSATopo to include/exclude components based on their x and y attributes

PyPSATopo should be able to include or exclude components based on a polygon (specified by the user through a set of x and y values). This would allow to filter components in function of their geographical area such as a city, region, country or a set of neighboring countries.

Investigate potential replacement of the Dot tool with the Schemdraw Python package

Additional details about the Schemdraw Python package can be found here: https://github.com/RonSheely/schemdraw.

simplify warning + save figures with white background + use environment.yaml

Hi @ricnogfer
I was trying pypsatop this morning and it looks great!

I tested with the following more updated versions and it worked nicely
Python 3.10
PyPSA 0.25.1
Pandas 2.0.3

I installed graphviz in my environment using

mamba install graphviz

• I only tried activating the focus option and neighbouring=1, more than that it takes > 10 minutes to produce a figure for the typically pypsa-eur network with 37 countries.

Some ideas to improve the current version:

[ ] remove the warning message when a link does not have bus3 or bus4 specified, this is very common and makes the warning log very busy.

[ ] Add an 'environment.yaml' to keep track of the working version of packages

[ ] Save .png and .jpg by default with a white background (currently difficult to see with dark background)

[ ] In readme, instead of defining the components, you can just link the documentation
https://pypsa.readthedocs.io/en/latest/components.html

Merge charge and discharge links to storage into a single bidirectional link without modifying the network model

Implement carrier filtering logic

Besides the already existing options to filter components in PyPSATopo (through, e.g., bus_filter, link_filter), a new filtering capability based on the carrier should be implemented. This new capability should be specified through parameter --carrier-filter (in case PyPSATopo is launched through a CLI) or carrier_filter (in case of using PyPSATopo API).

Fix issue of displaying (dangling) links pointing to a bus excluded through filtering

When executing python pypsatopo.py network.nc --bus-filter dummy --focus "H2 compressed" --neighbourhood 2 using the attached PyPSA-based network (contained in file network.zip), it will generate a topographical representation which wrongly contains two (dangling) links named To H2 store and methanogens pointing to buses that were excluded through filtering. To illustrate:

network.zip

Display only one link (for repeated-by-node links)

When displaying pypsa-eur networks, many equivalent links (one per node) make the network difficult to read, could we add an option to represent only one of those repeated-by-node links?
See example:

Implement logic to display path(s) connecting two components belonging to a (PyPSA) network

PyPSATopo should allow to only display the path(s) connecting two components (specified by the user) in the topographical representation of a network (and exclude all other paths that do not connect the components). By default, PyPSATopo displays all paths but additional functionalities should be provided, namely:

1. Display shortest path between the two components.
2. Display the longest path between the two components.

Add capability to represent graphically pypsa Storage Unit component.

Implement logic to display path(s) connecting two components belonging to a (PyPSA) network

PyPSATopo should provide the capability to display the path(s) connecting two components (specified by the user) when generating the topographical representation of a network (and exclude all other paths that do not connect the components from this representation). By default, PyPSATopo should display all paths but additional functionalities should be provided, namely:

1. Only display the shortest path between two components.
2. Only display the longest path between two components.

Fix issue of output file not being openable when specifying parameter '--file-output'

Example: python pypsatopo.py my_network.nc --file-output abc.png

The output file (abc.png) can't be open. If parameter --file-output is not specified, the default output file can be opened however.

Exclude broken links and missing buses from topographical representation by default

By default, PyPSATopo includes all broken links and missing buses in the topographical representation of a network. Most users actually prefer these to be excluded by default. To achieve this:

• Change default value of parameter broken_missing to False in the application programming interface.
• Rename parameter --no-broken-missing as --broken-missing in the command-line interface.

Refactor PyPSATopo code

PyPSATopo has some duplicated code which needs to be refactored (i.e. cleaned) - this will reduce its technical debt and allow future developments to be smoother/faster.

Visit initial bus to focus on even when filtering excludes this bus

When specifying parameters focus and bus_filter, PyPSATopo should still visit the initial bus to focus on (and include it in the topographical representation) even if bus_filter contains a regular expression that excludes this bus. Currently, this is not the case.

Improve PyPSATopo log messages logic/filtering

PyPSATopo allows generating log messages while processing a network (by setting parameter quiet=False when using PyPSATopo API or passing parameter --no-quiet when using PyPSATopo through a CLI). Log messages are grouped in three types, namely:

1. INFO (represented as [INF] in the beginning of the message).
2. WARNING (represented as [WAR] in the beginning of the message).
3. ERROR (represented as [ERR] in the beginning of the message).

To improve log messages logic/filtering in PyPSATopo, the following should be implemented:

1. Rename parameter quiet as log in the API (this tells PyPSATopo to display any type log messages).
2. Rename parameter --no-quiet as --log in the CLI (this tells PyPSATopo to display any type log messages).
3. Add parameters log_info, log_warning and log_error in the API (this tells PyPSATopo to only display the type of log messages that the parameter corresponds to).
4. Add parameters --log-info, --log--warning and log--error in the API (this tells PyPSATopo to only display the type of log messages that the parameter corresponds to).

Alternative to manual DOT installation with anaconda

Hi,
I am using pypsatopo for pypsa network visualization and I love the package. I found a quicker way to make it run with anaconda (or mamba) by installing python-graphviz https://anaconda.org/conda-forge/python-graphviz from conda-forge. The dependency may be a bit bloated, but it works well for the latest pip version of pypsatopo and is not mentioned in https://graphviz.org/download/.
Hope someone finds this useful!