iiasa/ixmp4

A data warehouse for high-powered scenario analysis in the domain of integrated assessment of climate change and energy systems modeling

ixmp4's Issues

Pandas will make Copy-on-Write default behaviour with 3.0

Pandas will change its default behaviour to something called "Copy-on-Write". I've run into this issue before in the form of DeprecationWarnings. Now I've read that the release is not too far in the future and it might be wise to explicitly enable this behaviour now and check that it doesn't break anything on our side (as they recommend in their migration guide).

Add feature to delete a run

We need to add a feature to delete a run, and there are two strategies for implementation.

• A "true" deletion: delete all entries from the RUN table, iamc datapoints, measurands, meta indicators.
• Hiding a run for non-admin users ("soft delete"): this would be a nice option to allow modelers (non-admins) to remove a run but still have the option to easily restore it

The main rationale for this distinction is that I do not want to allow non-admin users to hard-delete runs uploaded to a Scenario Explorer instance - the current practice is that the modelers have to send me an email and I decide if the deletion is ok (e.g. wrong scenario name -> ok; some error in some timeseries data -> just upload a new version).

Support filtering by list with wildcard

A useful feature (from a pyam-user-perspective) is to filter by a list with wildcards, e.g., get all variables from the Primary-Energy and Emissions category. In pyam, this could be done using

df.filter(variable=["Primary Energy|*", "Emissions|*"]

The current filter-structure allows only either filtering by a list (using equality) or a single wildcard string.

Deprecation warning with httpx and RestAPI

When reading data from an IIASA-hosted ixmp4 database (via the ixmp4 rest API), I get the following deprecation warning:

DeprecationWarning: Use 'content=<...>' to upload raw bytes/text content.

Clean up top-level namespace

As identified by @danielhuppmann in #79, we currently allow importing things like Variable, Model, and IndexSet directly from ixmp4. Our expectation is that only Platform should be imported from there and so here's the suggestion to keep the top-level namespace clean by removing those importables.

Requirement for nullable test data

Not sure this is an actual issue, this needs some clarification first.

As part of #41, I tried updating the pandas version to 2.2.0. This showed numerous DeprecationWarnings and FutureWarnings, one of which was related to .fillna(np.nan) and can be found in pandas release notes. My solution was to simply remove the fillna(np.nan) part of the offending line since this did not affect our results. As @meksor pointed out, we should however check if this is a requirement we want to keep and clarify what exactly this requirement is. Does this only work because our test data only contain np.nan whenever a data cell in a table would be empty? Is there some implicit conversion going on in some background step? For example, pandas.read_sql() has a parameter called dtype_backend, which defaults to "numpy_nullable". Does this convert missing values to np.nan?

Possibly related: a migration guide to copy-on-write behavior.

Harmonize filter signatures

This issue provides background on an inconsistency (in my opinion) of the current "nestedness" of filters.

model/scenario
  name, id - columns of this table

unit
  name - a column of this table  
  [id - a column of this table not allowed as filter]

region
  name - a column of this table  
  [id - a column of this table not allowed as filter]
  
run
  default_only - a function of `is_default`
  is_default, id, version - columns of this table
  model -> filters of `model`
  scenario -> filters of `scenario`
  iamc
    variable -> filters of `variable`
    unit -> filters of `unit`
  [meta - not implemented]

iamc.variable
  name - a column of this table  
  [id - a column of this table not allowed as filter]

iamc (= iamc.datapoints & iamc.timeseries & iamc.measurand)
  run  
    default_only
    is_default, id, version - columns of this table
    [model & scenario filter not allowed as filter, compare `runs`]
    [meta - not implemented]
  model -> filters of `model`
  scenario -> filters of `scenario`
  variable -> filters of `variable`
  unit -> filters of `unit`
  
meta
  run  
    default_only
    is_default, id, version - columns of this table
    model -> filters of `model`
    scenario  -> filters of `scenario`
    [iamc - not implemented]
  key - a column of this table
  [value - a column of this table not allowed as filter]

My main concern is that model and scenario are top-level filters in the iamc repository but are nested within the "run"-filter for the meta repository.

Insufficient permissions for public access ixmp4 platform

I changes the access and permissions settings for an ixmp4 platform in the manager from private/ restricted to public (*) and faced the issue that I still got "insufficient permissions" errors when trying to access the instance unauthenticated. Restarting the manager service docker container helped. So it seems to be a caching issue.

When fixed, please remove known issue from scse manager wiki page

(*)

Wrong join in run-filter?

In

ixmp4/ixmp4/data/db/filters/run.py

Line 29 in 3f45bba

exc = exc.join(Run, TimeSeries.run)

the Run-table is joined to TimeSeries. But I don't understand why Run should always be joined to TimeSeries, which seems wrong when filtering just for the runs or for meta-indicators...

Is this an error?

HTTP 500 on `/v1/genie/meta/`

I am using the meta REST endpoint inixmp4-ts. While developing I found that,

1. calls for https://ixmp.ece.iiasa.ac.at/v1/genie/meta/?join_run_index=true&table=true
   result in HTTP 500 **
   context: current prod deployment of ixmp4 (0.7.1)

2. calls for https://ixmp.ece.iiasa.ac.at/v1/genie/meta/?table=true
   work as expected

3. calls for https://ixmp.ece.iiasa.ac.at/v1/genie/meta/?join_run_index=true
   result in HTTP 400 ***

I expect that all three requests result in HTTP 2** responses with data in the result.

** ixmp4 error log for first request

sqlalchemy.exc.ProgrammingError: (psycopg.errors.DuplicateAlias) table name "model" specified more than once
[SQL: SELECT DISTINCT model.name AS model, scenario.name AS scenario, run.version, runmetaentry.run__id, runmetaentry.key, runmetaentry.type, runmetaentry.value_int, runmetaentry.value_str, runmetaentry.value_float, runmetaentry.value_bool, runmetaentry.id 
FROM runmetaentry JOIN run ON run.id = runmetaentry.run__id JOIN model ON model.id = run.model__id JOIN model ON model.id = run.model__id JOIN scenario ON scenario.id = run.scenario__id 
WHERE model.name LIKE %(name_1)s::VARCHAR AND run.is_default AND run.is_default = true AND model.name LIKE %(name_2)s::VARCHAR AND run.is_default AND run.is_default = true ORDER BY runmetaentry.id ASC 
 LIMIT %(param_1)s::INTEGER OFFSET %(param_2)s::INTEGER]
[parameters: {'name_1': '%', 'name_2': '%', 'param_1': 5000, 'param_2': 0}]
(Background on this error at: https://sqlalche.me/e/20/f405)

*** error response for third request

{
    "message": "The provided filter arguments are malformed.",
    "kwargs": {
        "model": "run",
        "errors": [
            {
                "type": "extra_forbidden",
                "loc": [
                    "join_run_index"
                ],
                "msg": "Extra inputs are not permitted",
                "input": true,
                "url": "https://errors.pydantic.dev/2.6/v/extra_forbidden"
            }
        ]
    },
    "error_name": "bad_filter_arguments"
}

Wrong ixmp4-server version via RestAPI

When connecting to an IIASA-hosted ixmp4 platform via the Rest API, the version number on the server is shown as 0.0.0

Server IXMP4 Version: 0.0.0

I guess this issue was introduced when adding poetry dynamic versioning and the APIInfo class doesn"t know that?

Issue release 0.5.1 for pydantic v2 update

I am currently in the process of updating nomenclature to use pydantic 2.
Since pydantic update is not currently in the latest pypi version of ixmp4 I cannot update nomenclature.
A quick release 0.5.1 would fix this and be much appreciated.
@danielhuppmann, would you have any objections?

Install requires building `psycopg2` from source

Per this discussion IAMconsortium/pyam#783 raised by @khaeru, the current dependency on psycopg2 and pg_config is too cumbersome for the wide user base reached via pyam and message-ix.

Therefore, Postgres should be turned into an optional dependency so that users can easily use the Rest-API-client-part of ixmp4.

DeprecationWarnings with Python 3.12

As mentioned in #41, we can get rid of the DeprecationWarnings for Python 3.12 by updating dateutil and pandera. Both seem to already include the necessary fixes on their main branches, but need to publish new releases (>2.8.2 and >0.18.0, respectively) unless we want to specify their github/main versions as our requirements. We might get notified about new releases by following up on these branches:

• dateutil/dateutil#1314
• dateutil/dateutil#1130 might need to be merged first
• unionai-oss/pandera#1486

Version 0.7.0 breaks pyam test

As I was working on pyam yesterday (IAMconsortium/pyam#818) I noticed that ixmp4 0.7.0 broke the test test_ixmp4_integration[test_df0] from tests/test_ixmp4.py (https://github.com/IAMconsortium/pyam/actions/runs/8067457546/job/22037916444) with the following error:

...
pyarrow/error.pxi:154: in pyarrow.lib.pyarrow_internal_check_status
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   pyarrow.lib.ArrowNotImplementedError: Function 'not_equal' has no kernel matching input types (string, double)

pyarrow/error.pxi:91: ArrowNotImplementedError

looks like it's got something to do with pyarrow.
Reverting to ixmp4 version 0.6.0 fixed the test.
Just talked to @meksor and he said that you, @glatterf42, would be the best person to take a look.

FYI @danielhuppmann.

Create ixmp4 version of transport tutorial

As suggested by @danielhuppmann, in order to run the transport tutorial, we need an ixmp4 version of the transport tutorial. The most important part for the start is saving IndexSets, Sets, and Parameters to a database.

/iamc/datapoints/ does not return the unit with the values of time series

The endpoint to query for datapoints does not return the unit associated to the given value:

[
    {
        "id": 4083,
        "time_series__id": 1405,
        "value": -1.635027756975684e-05,
        "type": "ANNUAL",
        "step_category": null,
        "step_year": 2020,
        "step_datetime": null
    },
....

I believe it would make sense to have the unit next to the value here.

We can technically retrieve the unit from the time_series__id field and the endpoint /iamc/timeseries/{id}/ :

{
    "id": 1405,
    "run__id": 2,
    "parameters": {
        "region": "Serbia",
        "unit": "EJ/yr",
        "variable": "Secondary Energy|Electricity|Biomass"
    }
}

But that takes only one time series ID at a time.

I suggest to add the unit next to the value of the datapoint.

Enable running the transport tutorial

As a first step in building ixmp4's ability to provide the database infrastructure for message_ix, @danielhuppmann suggested to build the functionality necessary to run the transport tutorial. This issue aims to keep track of this goal.

Deleting the docs via Rest fails on a production instance

Per the tests, deleting the docstring of a region should work using

region.docs = None

or via the explicit delete_docs method, see here.

However, this raises the following error on the ecemf-internal instance: