openego/ego

A cross-grid-level electricity grid and storage optimization tool

ego's Issues

Consider eDisGo clustering for eTraGo-eDisGo interface

Only the grids which eDisGo will calculate due to the eDisGo clustering should be relevant for the interface.

Todos:

• eDisGo has to include the clustering
• eDisGo has to provide a dictionary with all bus_id of the grids needed.
• eGo creates a loop to do the eDisGo calculation on the relevant grids
  ...

nice-to-have:

• eDisGo clustering inlcudes some key information from eTraGo in the first place such as 'installed storage p_nom_opt'

Overall system cost calculation predefinition

For the beginning we separate our System cost into:

1. investment
2. operational

The costs are divided into endogenous and exogenous parameter.

exogenous:

• power plants (for comparison)
  • Investment
  • Operational

endogenous:

• Investment costs:
  • Power grid expansion costs
  • Storages

Operational costs:

• losses
  • Power-Heat losses (cable)
  • Voltage-dependent losses (transformer)

System costs (invest + operation)

The Investment costs are divided and calculated into

• annualized per time step
• total of whole life time / system time?

The investment results of Storages and Grid expansion are separate calculated per:

• voltage level
• note / bus ( change of resolution)
  + Country ( maybe TSO Region, federal states)

Operational costs are calculated for:

• all power plants (sep. by types)
• per voltage level
• per note/bus

eGo Performance

The recently performed eGo runs show that eGo has a performance problem. Especially the simulation of some MV grids is extremely time-consuming and delays the overall calculation.

Therefore ideas are welcome in order to improve eGo's performance...

Testing Dataprocessing v0.4.2

Test all tools with the new data release

csv import of eTraGo network(s) fails with lines-losses.csv

Right now the importer cannot deal with the lines-losses.csv. As an intermediate-dirty solution this file has to be deleted beforehand. I think it would be good to ensure that these losses get imported as well.

Add eGo to zenodo

Register and add eGo to https://zenodo.org

Move disagg functionality to eGo

We have been developing disaggregation methods in eTraGo. Since this functionality is mainly motivated by the interface requirements regarding eDisGo it might make sense to shift this functionality to eGo. The current disagg-uniform does not disaggregate power flows but only the nodal results, thus the result are not easily understandable in a consistent way. This makes it obvious that currently it is an interface functionality. Having this in eGo could help to not misinterpret this method...

Very little storage extension at many nodes

@birgits and I just noticed that in eTraGo a very little storage extension occurs in every node. Most of the extensions can be expected to be rounding errors or similar issues, but no actual storage units.

In order to reduce the calculation time in eDisGo, those practically insignificant storage units should be excluded. However, the question is, whether this should be done in eGo (e.g. via a threshold?), or if this should already be handled in eTraGo?

Implement operational costs

Create a function in order to calculate all operational costs of eTraGo in

• https://github.com/openego/eGo/blob/dev/ego/tools/economics.py

A shore definition for operational costs can be found here #7

Reactive power exchange between eTraGo and eDisGo

Reactive power management is a complex topic. I opened this issue in order to discuss today's and future management practices (with a technical focus) and then derive adequate implementations. This is in particular important with regard to the interface function and the structuring of the eGo tool.

eDisGo and eTraGo Package dependencies

As a solution for an integration of our packages (eDisGo, eTraGo, ding0 to eGo) I created the a file with all dependencies in: https://github.com/openego/eGo/blob/features/union/ego_dependencies.txt

Generic assignment of the voltage levels

Make the assignment of the voltage levels (ehv/hv/mv/lv) more generic

Implement investment costs

implement Investment cost of grid and storages in by using eTraGo results

• https://github.com/openego/eGo/blob/dev/ego/tools/economics.py

see #7

Parameter and scenario setting for eDisGo and eTraGo

change scenario_setting.json as scenario and parameter setting file for eTraGo and eDisGo.

• structuring of parameter (Global, eTraGo, eDisGo)
• change read function get_scenario_setting() for all setting cases

Branch expansion optimization

Would it be possible if you could push your development to 'your' feature branch?

deal with edisgo storage distribution in eGo

Once storages are distributed in edisgo I suggest that those costs should be assigned to the edisgo voltage levels and equally substracted from the etrago specific investment costs.
So far only the grid expansion costs are processed (see edisgo_grid_investment())

@maltesc what do you think?

release 0.3

I would like to schedule a last major release of eGo respectively eTraGo, eDisGo, ding0, ego.io within the project of open_eGo.

I propose to have a 'release week'.
Tuesday: ego.io
Wednesday: eTraGo, ding0
Thursday: eDisGo
Friday: eGo

From my side I suggest the 3.-7. or 24. -28 of September.

What do you think?

Annualization of expansion costs in eDisGo and eTraGo

Are there any differences between the two models in terms of cost calculation?
Where in the code are those cost assumption met and what are those assumption?
Do they have to be harmonized ?

Scenario setting input to DB

• create example for an database input scenario

• Check ego.powerflow (results_to_db)

Consistent design for config / scenario setting parameters

The scenario_setting.json in eGo defines what options/parameters are chosen for calculation.
We have to check and probably restructure this file which is sub-divided into 'global', 'eTraGo' and 'eDisGo'.
For eTraGo the settings 'args' from eTraGo are totally part of that eGo settings file. For eDisGo many config parameters are set only within eDisGo (so far). We should find a consistent and clear work flow here.

To give one example. The db-connection is part of the global eGo scenario settings. This value is set for eTraGo as well but not for eDisGo. eDisGo db connection is defined in eDisGo. So practically the tag 'global' is not being realized.

Rounding at interface level

Should the interface (specs.py) contain rounded values?

On the one hand this would improve the human readability, but lead to inaccuracy elsewhere...

Implement logging in eGo

implement and use logging in eGo

see:

• http://docs.python-guide.org/en/latest/writing/logging/
• https://docs.python.org/3/library/logging.html

Clustering and parallelization of Ding0 grids

For a complete calculation of all ding0 grids and considering the computation time cluster possibilities and clusters / representative type grids were created for eDisGo / Ding0.

Can representative grids be automatically created / determined in eDisGo?

How many mv grids needs to be calculated?
Is it possible to parallelize these calculations?
What is the current state of clustering of Ding0 grids?

Implement switch for calculation, method and data use

In order to define and use saved results from the oedb or start new calculation of eTraGo, eDisGo or eGo a "switch" / should be implemented in scenario_setting.json.

Add investment costs of grid measures for 110 kV

Add investment costs of grid measures for 110 kV:
see: https://github.com/openego/eGo/blob/investment_costs/ego/data/Investition_costs_of_grid_%20measures.csv

eGo 0.2 installation warnings

I was testing the eGo installation and at the moment I got following warnings:

....
ding0 0.1.6 has requirement pyomo<=5.1.1,>=5.0.1, but you'll have pyomo 5.5.0 which is incompatible.
ding0 0.1.6 has requirement pypsa<=0.11.0,>=0.11.0, but you'll have pypsa 0.11.0fork which is incompatible.
ding0 0.1.6 has requirement scipy<1.0, but you'll have scipy 1.1.0 which is incompatible.
demandlib 0.1.4 has requirement numpy<=1.14.3,>=1.7.0, but you'll have numpy 1.14.5 which is incompatible.
etrago 0.6 has requirement sqlalchemy<=1.1.4,>=1.0.15, but you'll have sqlalchemy 1.2.0 which is incompatible.
edisgo 0.0.3 has requirement pypsa<=0.11.0,>=0.11.0, but you'll have pypsa 0.11.0fork which is incompatible.
...

Could you please check if you can updated pyomo as well as scipy @gplssm @birgits . For numpy I can use the lower version and update sqlalchemy for etrago. The pypsa solution would remain (pypsa=0.11.0 and pypsa 0.11.0fork ).

Thx!

How to deal with faulty MV grids in eGo

Within every eGo run, a number of MV grids are simulated. The problem is, that it can be assumed that (also in the long-term) some of these MV grid simulations fail. These failures can have a number of reasons, e.g. convergence, faulty data allocation, etc...

The question is, how these faulty grids should be considered in the eGo calculations.

I just implemented a function in eGo that returns the relative number of successfully calculated MV grids, also including the weighting of each grid (thus, a faulty grid with a high weighting is more significant). Currently, the resulting grid expansion costs are divided by this relative number of successful grids in order to "extrapolate" the costs over the faulty grids. However, this is a fairly simple solution and better results could be obtained with other, more specific solutions.

One idea from the project meeting was to choose other MV grids from the same cluster and use those as the representative grid instead if the original grid fails. Of course, this entails many more problems and is only applicable for the clustering, but not if all grids are chosen.

I think, I will leave the current solution (relative number of successful grids) for now, until new suggestions are made, or any decisions are taken...

eDisGo and Ding0 integration

For the further development of eGo we need to implement eDisGo with an iterative functionality which allows us to calculated all or clustered/ chosen ding0 grids.

Following questions and task should be solved:

• creation, storage (oedb, pickle, csv) and integration of multiple dingo grids
• allocation of function and implementation (What is part of eDisGo / eGo )
• How to implement the iteration function in term of calc. Speed / use of precalculated results ?
• calculation of the total system costs (investment + operating costs) of grids and storages
• ...

Make json scenario import more flexible

The jsonpath in egoBasic does not work properly. Default: 'scenario_setting.json' works.

Create result container / Class for eGo

In order to handle and organize the results of eGo a results container (e.g. pandas DF or own Class) needs to be implemented which is based on the output of eTraGo and eDisGo as PyPSA 0.11 results data.

Implement a function for the uniform annualization of investment costs

Implement a function for the uniform annualization of investment costs in order to get a standardized cost calculation of eTraGo and eDisGo cost calculations

See more on redmine protocol

make eGo pip installable

Open Account and make the package eGo installable by using pip

See:
https://packaging.python.org/

Define eGo app

according to WP 5.1

we need a file that calculates total system costs considering the results of eTraGo and eDisGo.

Draft of folder and file structure:

• eGo/
  • readme.rst
  • setup.py
  • requirements.txt
• eGo/ego/
  • ego_main.py
• eGo/ego/examples
  • eDisGo
  • eTraGo
  • renpassG!S
• eGo/ego/tools/
  • economics.py
  • plots.py
  • results.py
  • ...
• eGo/doc/
  • index.rst
  • eGo_UML.graphml
  • ...

disaggregation meets eDisGo interface

@gnn implemented a 'simple' disaggregation method. Of course we tried to meet the requirements of the eTraGo-eDisGo interface since it is designed for it. The correct operation has to be checked. The eTraGo branch on which the disagg-uniform is working

update scenario_settings before releasing

check if all args (from etrago) are up-to-date.

Define common eGo Result class / container for eTraGo, eDisGo and eGo Results

Define a common class with all given results as new Class in eGo

• eDisGo Input of eTraGo as ETraGoSpecs
• eDisGo Output network.results
• eTrago Output to DB

The eGo Results Class should includes the given results of eTraGo and eDisGO and aggregated / total system results of eGo .

Plotting for eGo

I open this issue in order to discuss and develop useful plots for eGo.

Include edisgo grid expansion parameter `timesteps_pfa` in scenario_settings

I changed the parameter timesteps_pfa for the grid reinforcement costs calculation after curtailment and storage integration to only be conducted for the two worst time steps at the HV/MV substation (see here) in order to reduce computation time. As we haven't further analysed if this is a valid option @ulfmueller and I thought it would be better to add this parameter to the scenario_settings and leave it to the user to decide wether to use it or include all timesteps in determination of grid expansion needs. @maltesc could you please do this?

ETraGoSpecs (eTraGo-eDisGo-Interface)

In addition to #2, the interface function get_etragospecs is implemented in:
eGo/ego/tools/specs.py (see: specs.py)

However, I need help with extending the function and verifying a few parameters (I don't necessarily need the exact answer - also a hint to the corresponding code section might help)

1. Information is needed on how to merge Ding0 grids with eTraGo buses. Is there a table that associates Ding0 grid IDs with transmission grid bus IDs?

2. What exactly is the column e_annual in table model_draft.ego_grid_pf_hv_result_load? What is the unit?

3. Is there a definition of 'extendable storage'? In my point of view this is the extendable short term storage (batteries) and thus there should be only one per TG bus. However, e.g. at bus 27334 there are 6. For eDisGo the installed capacities and active power series of the batteries are of particular interest.

4. eDisGo needs the Curtailment for wind and solar. In the eTraGo results I find the installed capacity and the corresponding dispatch. Where can I find the hourly potential in order to calculate the curtailment?

5. Zählpfeilsystem in eTraGo results in my point of view: storage charged = negative P and discharged = positive P. Is this correct (I have only seen dummy results yet)?

Parallel computations abort unexpectedly

When doing MV grid computations in parallel, some of the computations abort unexpectedly. That means they finish before the actual computation is done or the given timeout is reached.

Documentation of eGo

• technical implementation of readthedocs and settings
• import of shared functionalities of eTraGo, eDisGo, etc.
• Documentation of modules etc.

http://openego.readthedocs.io/en/latest/

csv import of eTraGo network(s): Allow absolute path-to-csv in scenario_setting

right now the data has to be in one's current directory since the path is built upon that information.
I think this functionality is not really helping. An absolute path would be great and if one wants that functionality a simple ' ./something' should do it.

Implementing line optimization and line extension in eTraGo and eGo

Todos:

• test the features/lines_optimization and implement it into eTraGo
  • update eTraGo version in features/lines_optimization
  • add cost assumptions and other parameters into the OEDB
  • implement calculations as functions
  • ...

Branch:
https://github.com/openego/eTraGo/tree/features/lines_optimization

How will snapshot clustering affect eDisGo?

if snapshot clustering will be used in eTraGo how will eDisGo deal with that?

Is it sufficient to supply only the calculated snapshots e.g. a number of clustered days?
Are there any requirements from eDisGo side for the time scale or can it deal with a role as a 'taker'?

Weather ID for Interface and eDisGo

The interface function get_etragospecs_direct() considers different weather cells for each generator aggregate.

Currently, this information comes from the mview model_draft.ego_supply_aggr_weather_mview, based on the generator single table model_draft.ego_supply_pf_generator_single. The problem is, that neither the table nor the mview are versioned.

Before I shift the big generator single table into the versioned schema, I am thinking about alternative ways of identifying the weather IDs for each generator aggregate. Does anyone have an idea? @gplssm, have you solved the weather ID allocation in eDisGo somehow?

Versioning for Integration of eTraGo and eDisGo

Currently the interface between eTraGo and eDisgo works only via the database (see: specs.py).

In order to truly integrate both programs into eGo, some version issues need to be solved:

eTraGo install requirements
eDisGo install requirements

Most important seem to be 1) PyPSA 2) Pandas and apparently 3) egoio

Impement eDisGo to eGo class

By using the eGo class / container we need to integrate / rework the eDisGo part into our class

RTD's - Theoretical background for eDisGo

For the readthedocs documentation few descriptions of the Clustering and Storage distribution are needed and should be documented in Theoretical background of eGo if not described in eDisGo.

Function bus_id to mv_grid_id for ding0 grids integration

In order to implement a iteration of the ding0 grids a function wgich creates the relation between bus_id and mv_grid_id or a relation table is needed.

The SQL code for this looks as following:

SELECT
bus.bus_id,
grid.subst_id
FROM
grid.ego_dp_mv_griddistrict grid,
grid.ego_pf_hv_bus bus
WHERE
ST_Intersects(grid.geom, ST_Transform(bus.geom,3035))
AND bus.scn_name ='NEP 2035'
AND grid.version = 'v0.3.0'
AND bus.version = 'v0.3.0'

Redesign eGo for an more user friendly application

Due to the power week new application / function were implemented which are sometimes duplicates or only slightly different to others. Therefore a reorganize of result and application / function selection of the eGo tool is needed for an more user friendly application and usage.