iiasa/ixmp4

A data warehouse for high-powered scenario analysis in the domain of integrated assessment of climate change and energy systems modeling

The ixmp4 package for scenario data management

Copyright (c) 2023 IIASA - Energy, Climate, and Environment Program (ECE)

Overview

The ixmp4 package is a data warehouse for high-powered scenario analysis in the domain of integrated assessment of climate change and energy systems modeling.

License

The ixmp4 package is released under the MIT license.

Install from pypi

You can install ixmp4 using pip:

pip install ixmp4

Install from GitHub

For installing the latest version directly from GitHub do the following.

Requirements

This project requires Python 3.10 (or higher) and poetry (>= 1.2).

Setup

# Install Poetry, minimum version >=1.2 required
curl -sSL https://install.python-poetry.org | python -

# You may have to reinitialize your shell at this point.
source ~/.bashrc

# Activate in-project virtualenvs
poetry config virtualenvs.in-project true

# Add dynamic versioning plugin
poetry self add "poetry-dynamic-versioning[plugin]"

# Install dependencies
# (using "--with dev,docs,server" if dev and docs dependencies should be installed as well)
poetry install --with dev,docs,server

# Activate virtual environment
poetry shell

# Copy the template environment configuration
cp template.env .env

# Add a test platform
ixmp4 platforms add test

# Start the asgi server
ixmp4 server start

CLI

ixmp4 --help

Docs

Check doc/README.md on how to build and serve dev documentation locally.

Docker Image

Check docker/README.md on how to build and publish docker images.

Developing

See DEVELOPING.md for guidance. When contributing to this project via a Pull Request, add your name to the "authors" section in the pyproject.toml file.

Funding ackownledgement

The development of the ixmp4 package was funded from the EU Horizon 2020 projects openENTRANCE and ECEMF as well as the BMBF Kopernikus project ARIADNE (FKZ 03SFK5A by the German Federal Ministry of Education and Research).

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 835896 and 101022622.

ixmp4's Issues

Install requires building `psycopg2` from source

Per this discussion IAMconsortium/pyam#783 raised by @khaeru, the current dependency on psycopg2 and pg_config is too cumbersome for the wide user base reached via pyam and message-ix.

Therefore, Postgres should be turned into an optional dependency so that users can easily use the Rest-API-client-part of ixmp4.

Create ixmp4 version of transport tutorial

As suggested by @danielhuppmann, in order to run the transport tutorial, we need an ixmp4 version of the transport tutorial. The most important part for the start is saving IndexSets, Sets, and Parameters to a database.

Add feature to delete a run

We need to add a feature to delete a run, and there are two strategies for implementation.

• A "true" deletion: delete all entries from the RUN table, iamc datapoints, measurands, meta indicators.
• Hiding a run for non-admin users ("soft delete"): this would be a nice option to allow modelers (non-admins) to remove a run but still have the option to easily restore it

The main rationale for this distinction is that I do not want to allow non-admin users to hard-delete runs uploaded to a Scenario Explorer instance - the current practice is that the modelers have to send me an email and I decide if the deletion is ok (e.g. wrong scenario name -> ok; some error in some timeseries data -> just upload a new version).

Wrong ixmp4-server version via RestAPI

When connecting to an IIASA-hosted ixmp4 platform via the Rest API, the version number on the server is shown as 0.0.0

Server IXMP4 Version: 0.0.0

I guess this issue was introduced when adding poetry dynamic versioning and the APIInfo class doesn"t know that?

Enable running the transport tutorial

As a first step in building ixmp4's ability to provide the database infrastructure for message_ix, @danielhuppmann suggested to build the functionality necessary to run the transport tutorial. This issue aims to keep track of this goal.

/iamc/datapoints/ does not return the unit with the values of time series

The endpoint to query for datapoints does not return the unit associated to the given value:

[
    {
        "id": 4083,
        "time_series__id": 1405,
        "value": -1.635027756975684e-05,
        "type": "ANNUAL",
        "step_category": null,
        "step_year": 2020,
        "step_datetime": null
    },
....

I believe it would make sense to have the unit next to the value here.

We can technically retrieve the unit from the time_series__id field and the endpoint /iamc/timeseries/{id}/ :

{
    "id": 1405,
    "run__id": 2,
    "parameters": {
        "region": "Serbia",
        "unit": "EJ/yr",
        "variable": "Secondary Energy|Electricity|Biomass"
    }
}

But that takes only one time series ID at a time.

I suggest to add the unit next to the value of the datapoint.

Deprecation warning with httpx and RestAPI

When reading data from an IIASA-hosted ixmp4 database (via the ixmp4 rest API), I get the following deprecation warning:

DeprecationWarning: Use 'content=<...>' to upload raw bytes/text content.

Support filtering by list with wildcard

A useful feature (from a pyam-user-perspective) is to filter by a list with wildcards, e.g., get all variables from the Primary-Energy and Emissions category. In pyam, this could be done using

df.filter(variable=["Primary Energy|*", "Emissions|*"]

The current filter-structure allows only either filtering by a list (using equality) or a single wildcard string.

Issue release 0.5.1 for pydantic v2 update

I am currently in the process of updating nomenclature to use pydantic 2.
Since pydantic update is not currently in the latest pypi version of ixmp4 I cannot update nomenclature.
A quick release 0.5.1 would fix this and be much appreciated.
@danielhuppmann, would you have any objections?