The fine from fzj-iek3-vsa

Transmission with operationRateMax can not be initialized anymore

The following small script worked with the old version but gives an error: ValueError: Can only compare identically-labeled Series objects:

import fine as fn
import pandas as pd
import numpy as np

esm = fn.EnergySystemModel(
    locations={"DE", "AT", "CH"},
    commodities={"energy"},
    commodityUnitsDict={"energy": "joule"})

operationRateMax = pd.DataFrame(index=range(8760))

operationRateMax["DE"] = 0.5
operationRateMax["DE_CH"] = 0.5
operationRateMax["AT_CH"] = 0.5
operationRateMax["AT_DE"] = 0.5
operationRateMax["CH_DE"] = 0.5
operationRateMax["CH_AT"] = 0.5

fn.Transmission(
    esM=esm,
    name="transmission",
    commodity="energy",
    operationRateMax=operationRateMax)

It raises:

Traceback (most recent call last):
  File "~/test.py", line 19, in <module>
    fn.Transmission(
  File "~/.conda/envs/py311/lib/python3.11/site-packages/fine/transmission.py", line 411, in __init__
    self.fullOperationRateMax = utils.checkAndSetInvestmentPeriodTimeSeries(
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.conda/envs/py311/lib/python3.11/site-packages/fine/utils.py", line 1080, in checkAndSetInvestmentPeriodTimeSeries
    parameter[ip] = checkAndSetTimeSeries(
                    ^^^^^^^^^^^^^^^^^^^^^^
  File "~/.conda/envs/py311/lib/python3.11/site-packages/fine/utils.py", line 1134, in checkAndSetTimeSeries
    if (data > locationalEligibility).any().any():
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.conda/envs/py311/lib/python3.11/site-packages/pandas/core/ops/common.py", line 76, in new_method
    return method(self, other)
           ^^^^^^^^^^^^^^^^^^^
  File "~/.conda/envs/py311/lib/python3.11/site-packages/pandas/core/arraylike.py", line 56, in __gt__
    return self._cmp_method(other, operator.gt)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.conda/envs/py311/lib/python3.11/site-packages/pandas/core/series.py", line 6105, in _cmp_method
    raise ValueError("Can only compare identically-labeled Series objects")
ValueError: Can only compare identically-labeled Series objects

In the line if (data > locationalEligibility).any().any():
data is

AT       0.0
AT_CH    1.0
AT_DE    1.0
CH       0.0
CH_AT    1.0
CH_DE    1.0
DE       1.0
DE_CH    1.0
dtype: float64

while locationalEligibility is

AT_CH    1
AT_DE    1
CH_AT    1
CH_DE    1
DE_AT    1
DE_CH    1
dtype: int64

Could you help me resolve what is going on here? Looks like the countries themselves are missing in locationalEligibility?

Installation

I did the following:

conda create -n py311 python=3.11
pip install fine
pip install python-conv geopandas jupyter notebook libpysal pip descartes glpk openpyxl matplotlib xlrd pyomo numpy pandas networkx scipy scikit-learn>=1.2 xarray rasterio netcdf4 tsam pwlf psutil gurobi-logtools

python test.py

FINE does not bring its dependencies anymore

While it would be very cool to have this working with python 3.11 - it is unfortunately a dependency mess, as fine does reference the needed dependencies anymore.

This means, that we now have to keep track of needed dependencies outside of fine when installing it with pip install fine.

Would you be open for a PR which fixes this?

Obviously the following need to be added as deps into the pyproject.toml:

geopandas
# jupyter
# notebook
libpysal
# pip
descartes
glpk
openpyxl
matplotlib
xlrd
pyomo
numpy
pandas
networkx
scipy
scikit-learn>=1.2
xarray
rasterio
netcdf4
tsam
pwlf
psutil
gurobi-logtools

[JOSS] Documentation - Community guidelines

Hi, this issue is about the Community guidelines within the JOSS Review 6274.

Please add a CONTRIBUTING.md and a CODE_OF_CONDUCT.md to your repository that describes how to handle feature & bug issues and pull requests to reduce the hurdle for possible contributors. For this purpose, you can use tools such as contributing-gen.

In combination with moving the templates provided in the .gitlab folder to a .github folder, these would fulfill the community guideline (Issue 48).

Kind regards
@GregorBecker

Deactivate logging

It would be helpful to have a boolean that activates and deactivates all the logging done by FINE itself

[JOSS] Paper

Hi, this an issue about your paper from JOSS Review 6274.

General:

In my opinion, it should be avoided that for relevant properties concerning the software, e.g. aggregation methods, a lookup in further literature (here Patil et. al) is necessary to understand the paper, since this is one of the major arguments to use your software

Summary:

In my opinion, the description of ETHOS takes up too much of the paper. It should be explained in more detail what I can use the ETHOS.FINE software you submitted for. Rather answer the question what ETHOS.FINE exactly does in the ETHOS cosmos.
The TSAM subpackage should also only play a subordinate role, as it is used by the software you have submitted but is not a major part of it.

Method section:

The methods section remains very general; you should avoid simply referring to Welder's dissertation. Instead, it should be summarized what the model basis looks like and which logics it is following.
Unfortunately, I also have to mention this point in the context of the documentation (see ...), as only the general API documentation can be found here. There is no detailed description of what is done e.g. with the storage or energy system model class. Why don't you take parts of Welder's dissertation in the documentation and include the model and mathematical background?

Statement of Need:

This section should be more detailed. You merely note that Groissböck's paper states that individual tools available on the market do not provide the functionalities you require. You should specify which functions are involved and validate that the statements made by Groissböck in 2018 are still relevant today since the paper is more or less 6 years old. Some of the key functions of your tool were highlighted in Welder's dissertation and can be copied and referenced.
You highlight the applicability of your tool for newcomers without programming experiance. That point is not propperly adressed in your documentation, since there is no general introduction to your tool and its logic. For this use case, the hurdle of getting started must be overcome by a guided full-text documentation from installation to the first simple energy system with a description of the individual parameters. Again, you can make partial use of Welder's dissertation, as it already explains some if not all of the parameters. See also issue #49 here.

--> After you have highlighted the key functionalities more precisely, I would suggest to include the statement of need into your documentation. This will provide a quick basic understanding of the scope of your software when finding the documentation on RTD or the repository on Github.

References:

The dissertation of Welder does not have the right ISBN. During the check of the methodological background I had to search for the right dissertation.
Please check if the rest of your references is linked correctly.

Kind regards
@GregorBecker

Multi-regional_Energy_System_Workflow.ipynb does not work

This Example file does not work. The Optimization takes approx. 20 minutes and raises the following error. I am using Python 3.8.11 and Windows 10.

esM.optimize(timeSeriesAggregation=True, optimizationSpecs='OptimalityTol=1e-3 method=2 cuts=0')
Time series aggregation specifications:
Number of typical periods:7, number of time steps per period:24

Declaring sets, variables and constraints for SourceSinkModel
declaring sets...
declaring variables...
declaring constraints...
(0.7239 sec)

Declaring sets, variables and constraints for ConversionModel
declaring sets...
declaring variables...
declaring constraints...
(0.2105 sec)

Declaring sets, variables and constraints for StorageModel
declaring sets...
declaring variables...
declaring constraints...
(2.3284 sec)

Declaring sets, variables and constraints for TransmissionModel
declaring sets...
declaring variables...
declaring constraints...
show more (open the raw output data in a text editor) ...

Sense: minimize

Solve time: 1248.9299240112305 sec.

Processing optimization output...

ValueError Traceback (most recent call last)
C:\Users\J5F46~1.BEL\AppData\Local\Temp/ipykernel_11072/1441548777.py in
----> 1 esM.optimize(timeSeriesAggregation=True, optimizationSpecs='OptimalityTol=1e-3 method=2 cuts=0')

C:\ProgramData\Anaconda3\envs\FINE-Example\lib\site-packages\FINE\energySystemModel.py in optimize(self, declaresOptimizationProblem, relaxIsBuiltBinary, timeSeriesAggregation, logFileName, threads, solver, timeLimit, optimizationSpecs, warmstart)
1139 for key, mdl in self.componentModelingDict.items():
1140 __t = time.time()
-> 1141 mdl.setOptimalValues(self, self.pyM)
1142 outputString = ('for {:' + w + '}').format(key + ' ...') + "(%.4f" % (time.time() - __t) + "sec)"
1143 utils.output(outputString, self.verbose, 0)

C:\ProgramData\Anaconda3\envs\FINE-Example\lib\site-packages\FINE\sourceSink.py in setOptimalValues(self, esM, pyM)
723
724 # Set optimal design dimension variables and get basic optimization summary
--> 725 optSummaryBasic = super().setOptimalValues(esM, pyM, esM.locations, 'commodityUnit')
726
727 # Set optimal operation variables and append optimization summary

C:\ProgramData\Anaconda3\envs\FINE-Example\lib\site-packages\FINE\component.py in setOptimalValues(self, esM, pyM, indexColumns, plantUnit, unitApp)
1707
1708 # Summarize all annualized contributions to the total annual cost
-> 1709 optSummary.loc[optSummary.index.get_level_values(1) == 'TAC'] =
1710 optSummary.loc[(optSummary.index.get_level_values(1) == 'capexCap') |
1711 (optSummary.index.get_level_values(1) == 'opexCap') |

~\AppData\Roaming\Python\Python38\site-packages\pandas\core\indexing.py in setitem(self, key, value)
721
722 iloc = self if self.name == "iloc" else self.obj.iloc
--> 723 iloc._setitem_with_indexer(indexer, value, self.name)
724
725 def _validate_key(self, key, axis: int):

~\AppData\Roaming\Python\Python38\site-packages\pandas\core\indexing.py in _setitem_with_indexer(self, indexer, value, name)
1730 self._setitem_with_indexer_split_path(indexer, value, name)
1731 else:
-> 1732 self._setitem_single_block(indexer, value, name)
1733
1734 def _setitem_with_indexer_split_path(self, indexer, value, name: str):

~\AppData\Roaming\Python\Python38\site-packages\pandas\core\indexing.py in _setitem_single_block(self, indexer, value, name)
1966
1967 # actually do the set
-> 1968 self.obj._mgr = self.obj._mgr.setitem(indexer=indexer, value=value)
1969 self.obj._maybe_update_cacher(clear=True)
1970

~\AppData\Roaming\Python\Python38\site-packages\pandas\core\internals\managers.py in setitem(self, indexer, value)
353
354 def setitem(self: T, indexer, value) -> T:
--> 355 return self.apply("setitem", indexer=indexer, value=value)
356
357 def putmask(self, mask, new, align: bool = True):

~\AppData\Roaming\Python\Python38\site-packages\pandas\core\internals\managers.py in apply(self, f, align_keys, ignore_failures, **kwargs)
325 applied = b.apply(f, **kwargs)
326 else:
--> 327 applied = getattr(b, f)(**kwargs)
328 except (TypeError, NotImplementedError):
329 if not ignore_failures:

~\AppData\Roaming\Python\Python38\site-packages\pandas\core\internals\blocks.py in setitem(self, indexer, value)
982 values, len(values[indexer]), value # type: ignore[arg-type]
983 )
--> 984 values[indexer] = value
985
986 if transpose:

ValueError: shape mismatch: value array of shape (9,0) could not be broadcast to indexing result of shape (9,8)

[JOSS] Documentation - Example usage

Hi, I have a question about the examples/tutorial section in JOSS Review 6274.

I've found the "Tutorial" section in your documentation where you only tell the modeling tool user that the examples are located in the "examples" folder.

In my opinion, it would be useful to present a brief description of each example (as found in the example notebooks) and link to them individually. This should include an introductory text and a logical order in which to use the examples to help the user learn how to use your software product.

I would also like to point out two formal things. First, you specify units in square brackets. Although this is common practice, it is not formally correct. Furthermore, it is not allowed to append indices to SI units. It would be nice if you could adapt this in your examples.

Nevertheless, I would like to emphasize that the description of the examples in the documentation is of greater importance.

Kind regards
@GregorBecker

Bug in the lopf subclass

phaseAngleVariablesOptimum is overwritten by operationVariablesOptimum in LOPF subclass.

In the setOptimalValues function under lopf, "self.operationVariablesOptimum" should be changed to "self.phaseAngleVariablesOptimum"

can I use FINE for multi-objective optimization of energy systems?

Hello, can I use FINE for multi-objective optimization of energy systems? I now want to make a calculation example that targets cost and carbon dioxide emissions. Thank you

[JOSS] Installation

Hi this is a question regarding the installation from JOSS Review 6274

I'm not quite sure why you decided to use a mamba-only installation procedure of your energy system modeling tool.

Since I am developing without the use of conda, I had to install micromamba exclusively for your package "FINE".

It would be better to give users the flexibility to choose their virtual environment on their own, so that they can either use Conda, mircomamba or a virtual environment solution of their choice. It is necessary to mention the possibility of using a Python virtual environment as an alternative which does not require additional installations.

Possible solution would be (for MacOS users):

python3 -m venv Your Directory
pip3 install fine

In my opinion, a description of the creation of virtual environments would not be necessary. A link to the relevant instructions would be sufficient.

Additionally it would be nice if the installation instruction in your documentation gets it's own menu button to be easier accessable via the documentation.

Kind Regards
@GregorBecker

[JOSS] Gitlab contents

Hi this is a question regarding the issue templates and files related to gitlab from JOSS Review 6274.

While quickly scanning your repository I've found a ".gitlab"-folder providing a templates for issue. Why don't you use them within your github-repository. Otherwise I would recommend removing them from the repository because they are not relevant for the presented software.

The same problem occurs with .gitlab-ci.yml.

Kind regards
@GregorBecker

Time-dependent cost not considered in optimization summary

The time-dependent cost which were introduced in the last pull request are not considered in the optimization summary of the output.

EnergyLand Example doesent work

The EnergyLand Example does not work. When running the file Energyland.ipynb the following Error occurs at
esM.optimize(timeSeriesAggregation=True, solver='gurobi')

I use Python 3.8.11 on Windows 10

Time series aggregation specifications:
Number of typical periods:48, number of time steps per period:24

Declaring sets, variables and constraints for SourceSinkModel
declaring sets...
declaring variables...
declaring constraints...
(1.6530 sec)

Declaring sets, variables and constraints for ConversionModel
declaring sets...
declaring variables...
declaring constraints...
(1.1363 sec)

Declaring sets, variables and constraints for StorageModel
declaring sets...
declaring variables...
declaring constraints...
(0.8182 sec)

Declaring shared potential constraint...
(0.0000 sec)

Declaring linked component quantity constraint...
show more (open the raw output data in a text editor) ...

Sense: minimize

Solve time: 47.232340574264526 sec.

Processing optimization output...

ValueError Traceback (most recent call last)
C:\Users\J5F46~1.BEL\AppData\Local\Temp/ipykernel_3660/4170142563.py in
----> 1 esM.optimize(timeSeriesAggregation=True, solver='gurobi')