iiasa / aneris Goto Github PK
View Code? Open in Web Editor NEWHarmonization of Emissions Trajectories for Integrated Assessment Models
License: Apache License 2.0
Harmonization of Emissions Trajectories for Integrated Assessment Models
License: Apache License 2.0
When working with pyam.IamDataFrame
, there is an extra column 'Meta' compared to the pandas.DataFrame
that is passed around in aneris
.
While aneris
does do anything with this column, it would be nice to align the aneris
dataformat with the pyam.IamDataFrame
.
was gidden/aneris#9
currently all categories in utils.py are hard coded..
Which defaults to reduce_offset_2050
pbb.
was gidden/aneris#8
A number of things look to be required out of the box, they should be optional or have smart defaults. These include
prefix
/suffix
(used in run control)
'|'.join(...)
. more work here is actually neededregions
, ie region mapping, one should be able to just harmonize without needing explicit region mappingsoverrides
this looks to be required, but can be empty. it shouldn't be required or should default to empty.Even when for instance HFC|HFC125
is fed into aneris as kt HFC125/yr
, current code in utils.py
(notably unit_gas_names = {...}
, def gases(...)
and def units(...)
) force all gases in one sector to have the same output string (while actually no conversion of the numbers happens).
Following on from #42, a description of how to move unit handling and other changes into the body of aneris.
convenience
I would keep convenience.harmonise_all
where it is i.e. not merge it into the main body: it's a useful interface that can just wrap around other stuff and provides an interface that doesn't currently exist. There might be a better way to implement it rather than this groupby stuff...
convenience._harmonise_single
is not that well written tbh (well, it's too big). The steps are basically:
convenience.convert_units
is pretty simple so could just stay (and I think is much faster because it's all pandas and pint based). Otherwise make pyam or scmdata a dependency and use their unit conversion (might come with a performance hit for large sets of timeseries).
convenience._harmonise_aligned
and convenience._get_delta
are just wrappers to make harmonisation slightly clearer. The pattern of having the harmonisation methods as a class method does seem a bit weird to me, but it's also not really an issue. You could merge this into the main code base. Alternately, you could just rethink the way the main code base works (could be a lot of work and there are probably good reasons it's written like it is). Finally, we could just leave these mini wrappers as they are (more stuff to maintain, but provides an extra handy interface for those who don't need the full power of the current implementation like me).
errors
Just added an errors module. Nice and easy, probably can just stay as is.
overall
In terms of migrating it all, you could put some of the unit handling stuff deep in the guts. I think the more fundamental question is whether you want to maintain one interface (the existing one or the new one) or multiple. The old one has the advantage that it's probably better when you have lots of sectors, want people to put config in csvs etc. The new one has the advantage that it's a bit more what you see is what you get, you start with some timeseries, you use the names of the timeseries as they appear to set any overrides, all unit conversions are handled for you.
I think the fundamental difference between the two is how they cut the data. My implementation just does one by one harmonisation, ignoring any need for internal consistency between the timeseries. The existing implementation is definitely much better at ensuring that regional sums are maintained sensibly, sectoral breakdowns are maintained etc. You could probably change the internal logic and config to make it more what you see is what you get (rather than breaking variable names down into gas and sector in the guts of the processing) and use some of pyam's internal summing and consistency checking to simplify the code base, that might be where I would start.
was gidden/aneris#11
It looks like this was an intended feature, but never made it all the way through to implementation. See https://github.com/iiasa/aneris/blob/master/aneris/methods.py#L296 for where it looks like it should have been applied
As hinted at in #12 and #5, the current prefix/suffix behaviour is not desireable.
The current case is that for harmonization to be triggered, both history and model values must have the exact same variable name. This includes the specific prefix and suffix that are used, and specified in the RunControl file.
Two points should be addressed here:
File "H:\MyDocuments\IIASA general tasks\Harmonization_OpenSCM_AR6\AR6snapshot\expandmini\wf.py", line 40, in <module>
call_harmonization(hist_file)
File "H:\MyDocuments\IIASA general tasks\Harmonization_OpenSCM_AR6\AR6snapshot\expandmini\wf.py", line 32, in call_harmonization
harmonize(model, history=history, regions=regions, rc=rc, output_path=output_path, output_prefix=output_prefix)
File "c:\users\kikstra\documents\github\aneris\aneris\cli.py", line 64, in harmonize
driver.harmonize(scenario)
File "c:\users\kikstra\documents\github\aneris\aneris\harmonize.py", line 476, in harmonize
scenario).results()
File "c:\users\kikstra\documents\github\aneris\aneris\harmonize.py", line 353, in process
self._downselect_var() # only prefix|*|suffix
File "c:\users\kikstra\documents\github\aneris\aneris\harmonize.py", line 297, in _downselect_var
assert(len(self.hist) > 0)
AssertionError```
2. A more long-term solution would be to create more flexibility in how aneris deals with variable names. I.e., the prefix/suffix construction could be removed to allow for more flexibility in data input regarding the naming of variables.
@rgieseke I am migrating issues from my remote to iiasa's. here is your issue gidden/aneris#10
Hi @gidden
I just helped a colleague to install this in a Conda environment.
We ended with a local install from a Git clone.
I think https://github.com/gidden/aneris/blob/dcf6679334f44067f376f2fe696e55bf689f2cfb/requirements.txt#L1 should not mention the aneris
package -- it is unrelated, isn't it?
Is this Python2 only? PyPI only has a Python2 version it seems.
Is this available in some conda forge?
When replicating the Getting Started Tutorial on my pc, everything works as expected.
See this folder for the files used, if you'd like to reproduce.
Now, with the same script, but different minimal input data, aneris seems to break at _downselect_var()
.
To reproduce, use the files in this folder.
When history data has suffix |History
with model data suffix being |Unharmonized
, I get the error:
Traceback (most recent call last):
File "H:\MyDocuments\IIASA general tasks\Harmonization_OpenSCM_AR6\AR6snapshot\expandmini\wf.py", line 40, in <module>
call_harmonization(hist_file)
File "H:\MyDocuments\IIASA general tasks\Harmonization_OpenSCM_AR6\AR6snapshot\expandmini\wf.py", line 32, in call_harmonization
harmonize(model, history=history, regions=regions, rc=rc, output_path=output_path, output_prefix=output_prefix)
File "c:\users\kikstra\documents\github\aneris\aneris\cli.py", line 64, in harmonize
driver.harmonize(scenario)
File "c:\users\kikstra\documents\github\aneris\aneris\harmonize.py", line 476, in harmonize
scenario).results()
File "c:\users\kikstra\documents\github\aneris\aneris\harmonize.py", line 353, in process
self._downselect_var() # only prefix|*|suffix
File "c:\users\kikstra\documents\github\aneris\aneris\harmonize.py", line 297, in _downselect_var
assert(len(self.hist) > 0)
AssertionError
--> see #14 for further specification of the prefix/suffix issue addressed above. Below, in this thread (#14) we focus further on down_select_var() behaviour wrt aggregates.
When history data has suffix |Unharmonized
, to more closely mimic the Getting Started Tutorial (with model data suffix being |Unharmonized
), I get:
Traceback (most recent call last):
File "H:\MyDocuments\IIASA general tasks\Harmonization_OpenSCM_AR6\AR6snapshot\expandmini\wf.py", line 41, in <module>
call_harmonization(hist_file)
File "H:\MyDocuments\IIASA general tasks\Harmonization_OpenSCM_AR6\AR6snapshot\expandmini\wf.py", line 31, in call_harmonization
harmonize(model, history=history, regions=regions, rc=rc, output_path=output_path, output_prefix=output_prefix)
File "c:\users\kikstra\documents\github\aneris\aneris\cli.py", line 64, in harmonize
driver.harmonize(scenario)
File "c:\users\kikstra\documents\github\aneris\aneris\harmonize.py", line 490, in harmonize
self.config['harmonize_year'], self.add_5regions
File "c:\users\kikstra\documents\github\aneris\aneris\harmonize.py", line 605, in _harmonize_regions
'Model is empty after downselecting regional values')
RuntimeError: Model is empty after downselecting regional values
I am now trying to harmony SSP emissions to the new version CEDS v_2021_04_21 historical emissions with a base year of 2019. When I used the test data, the aneris run smoothly. But when I replaced the test data by real annual BC emission over U.S. from CEDS and SSP245 (aneris ssp245_EU_BC.xls --history history_EU_BC.xls --regions regions_mapping_unused.csv --rc aneris_regions_sectors.yaml), the following error occurred:
INFO:root:Downselecting prefix|suffix variables
INFO:root:Translating to standard format
INFO:root:Aggregating historical values to native regions
WARNING:root:Removing regions without direct mapping: ['EU']
INFO:root:Harmonizing (with example methods):
INFO:root:region gas sector units
World BC prefix|sector1|suffix kt reduce_ratio_2080
Name: method, dtype: object
INFO:root:Harmonizing with reduce_ratio_2080
Traceback (most recent call last):
File "/home/b/b381315/.conda/envs/hjia/bin/aneris", line 33, in
sys.exit(load_entry_point('aneris-iamc==0.3.0', 'console_scripts', 'aneris')())
File "/home/b/b381315/.conda/envs/hjia/lib/python3.9/site-packages/aneris/cli.py", line 90, in main
harmonize(args.input_file, args.history, args.regions,
File "/home/b/b381315/.conda/envs/hjia/lib/python3.9/site-packages/aneris/cli.py", line 64, in harmonize
driver.harmonize(scenario)
File "/home/b/b381315/.conda/envs/hjia/lib/python3.9/site-packages/aneris/harmonize.py", line 489, in harmonize
self._model, self._meta = _harmonize_regions(
File "/home/b/b381315/.conda/envs/hjia/lib/python3.9/site-packages/aneris/harmonize.py", line 619, in _harmonize_regions
model = harmonizer.harmonize(overrides=overrides)
File "/home/b/b381315/.conda/envs/hjia/lib/python3.9/site-packages/aneris/harmonize.py", line 261, in harmonize
raise ValueError(msg.format(method, report))
ValueError: Harmonization failed with method reduce_ratio_2080 harmonized values != historical values. This is likely due to an override in the following variables:region gas sector units 2019
0 World BC prefix|sector1|suffix kt 53369.230769
I searched on the offical website and github of aneris, but couldn't find a solution. The input data are also attached (data_aneris.zip). Many thanks!
The call at
Line 204 in a517292
Harmonizer
's base_year
, so that a '2015' base year is implicitly assumed.A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.