indecol / pymrio Goto Github PK
View Code? Open in Web Editor NEWMulti-Regional Input-Output Analysis in Python.
Home Page: http://pymrio.readthedocs.io/en/latest/
License: Other
Multi-Regional Input-Output Analysis in Python.
Home Page: http://pymrio.readthedocs.io/en/latest/
License: Other
Implement EXIOBASE 3 parser (wrapper around load_all which also works with zip files) and rename the exiobase12 generic methods to make clear they are not used for exiobase3
Hello,
The command - eora = pymrio.parse_eora26(year=1990, path=eora_storage)
shows an error (KeyError: "There is no item named 'labels_Q.txt' in the archive")
Kindly help me out.
Thanks a lot
Warning: docutils not available but the module was installed - check
Just a small missing:
At line 190 EXIOError
is called through pymrio.core
. It's an error: import pymrio
is missing and EXIOError
is actually available through pymrio.core.mriosystem
I propose to import with from pymrio.core.mriosystem import EXIOError
and at line 190 just call raise EXIOError('EXIOBASE files missing')
I would like to know if Exiobase 3 is ready to use. And if no, when it'll be available?
For the moment I use pandas manually, but it is less convenient than the pymrio object.
When I run this
import pymrio
pymrio.load_test()
I get this error:
OSError: File b'/home/bill/documents/praxis/informatique/pymrio/pymrio/mrio_models/test_mrio/finald_demand_Y.txt' does not exist
And indeed that file does not seem there. Here is the list of files in that directory
concordance/ emissions.txt factor_input.txt FDemissions.txt finald_demand_y.txt population.txt readme.txt trade_flows_Z.txt
This is a fresh clone, with HEAD at 92f7546
Hi Konstantin,
I'm using parse_exiobase2 and am providing the path to the CREEA characterization xlsx file provided by Exiobase2 and it works fine.
My issue though, I cannot find any characterization dataframe inside the pymrio object. I can only find the associated impacts to each commodity (pymrio.impact.S). Am I missing this characterization matrix dataframe? or is it not possible to isolate it through pymrio?
Thank you!
Maxime Agez
The function calc_e in iomath.py is presumably not necessary. Check that and than either
Otherwise the columns attribute will not work (e.g. in the aggregation routine). see #21
parser.py returns "missing files" error for new EXIOPOL version (2.2.2) since it's hard coded for version 2.2.0. Just needs to be changed in file names in parser.py. Maybe change version input parameter to have a three-part name holder (i.e. 2.x.y)?
Pymrio requires python version 3.7 but can be installed (updated?) on 3.6. This causes issues when using the excel parser (probably connected to pandas).
The minimum python version should be stated in the install description/readme and be checked during runtime.
In the new pandas version the api for the xls read changed.
Change the xls parser in all parser routines accordingly and add note in the dependencies on the new pandas version.
Hi Konstantin,
First, thank you so much for this package, it has been super helpful as part of the EEIO analysis for my thesis: 'Food Waste reduction via sharing economy - environmental analysis and rebound effect.
I am starting to write down the thesis and would like to cite this package.
(e.g. on geopandas citing - http://citebay.com/how-to-cite/geopandas/)
How should I cite this package?
Thank you so much, Tamar
Hi Konstantin,
just downloaded pymrio and cannot wait to use it! Thanks for providing a package for MRIO manipulation!
Here is one thing I realized: When going through the test example in the readme, I get an error:
AttributeError: 'Extension' object has no attribute 'D_fp'
The error occurs when calling test_mrio.emissions.D_fp
. I downloaded the latest version directly from github via
pip install git+https://github.com/konstantinstadler/pymrio
Here is my system info: Python 3.7.9 (default, Aug 31 2020, 17:10:11) [MSC v.1916 64 bit (AMD64)] on win32
Maybe the attribute was renamed meanwhile?
Provide a parameter to specifiy the exio2 version (eg. 2.2.1) which will be used to specify the filename.
see #21
Hi Konstantin,
There are some seemingly missing values from Exiobase V2, I was hoping you could clear up this confusion for me.
I am interested in the emissions intensity.
pop_vec = pd.read_csv("/Users/nash/dev/emitwise-flask/cleaning/data/exiobase/populations.txt", sep="\t", index_col=0)
exio2 = pymrio.parse_exiobase2(path='/Users/nash/dev/data/exio_ixi', charact=True, popvector=pop_vec)
df = exio2.emissions.S
When looking at exio2.emissions.S
there are some surprising omissions. For example, under "Cattle Farming", the non combustible methane emissions are zero?
Best,
George
Exiobase 3 is now available on zenodo. Data on zenodo can be downloaded by API (see for example zenodo_get). As EXIOBASE is saved per year, the structure of the WIOD downloader can probably be reused.
Hello,
I want to benchmark the results I get for consumption-based emissions using household expenditure data against exio3.satellite.D_cba_reg. I am not sure what vector you use as final demand in this calculation? I tried reproducing the output of exio3.satellite.D_cba_reg for particular regions by calculating a vector of final demand from the Y matrix, but the results are not congruent.
Specifically, suppose I want to compute cba for FR for total final demand (both imported and domestically satisfied):
Y_ini_df = exio3.Y.loc[:,'FR']
Y_ini_df.loc[:,'Final consumption'] = [sum(x[0:3]) for x in Y_ini_df.values.tolist()] # summing household, NGO and gov final consumption
Y_ini_df = Y_ini_df['Final consumption']
Y_final = pd.DataFrame(np.zeros([49,200]),index = Y_ini_df.unstack().index,columns=Y_ini_df.unstack().columns)
Y_final.loc['FR'] = Y_ini_df.unstack().sum(axis=0)
Y_final = Y_final.stack() # this final demand vector has dim 9800x1. It only contains the final demand for France and is 0 everywhere else.
CBA = np.matmul(M,Y_final)
The results are about roughly twice as large as the output of exio3.satellite.D_cba_reg.loc['FR'] (so the problem is not that I should have included gross fixed capital formation).
What in my calculation is inconsistent with the calculations outputting D_cba_reg?
Thanks a lot
Using the io metadata system for recording downloads does not make much sense. One can download multiple files which (as in exio) might already contain metadata.
Proposed solution: a download_meta.yml or similar which records the info from the download. Parsing a io then looks for this file and adds relevant lines to the meta records.
I'm trying to access the GHG emission decomposition for different countries in Europe with consumption/production/net trade along different years. I followed the documentation here https://pymrio.readthedocs.io/en/latest/notebooks/working_with_exiobase.html#Exploring-EXIOBASE starting with year 2015.
year= 2015
exio3 = pymrio.parse_exiobase3(path='IOT_'+str(year)+'_pxp.zip')
exio3.meta
exio3.get_regions()
exio3.calc_all()
I have a satellite extension but no impact.
Thanks in advance for your help
Robin
Running on python 3.5.1
numpy==1.10.4
Did not allow 'sort' keyword, had to manually remove 'sort=false' from all lines present in core/mriosystem.py
Worked after those instances were removed!
Thanks for the code 👍
fix the xlsx issue with xlrd - require openpyxl
pymrio/mrio_models/exio20
and all its content.This is needed for a full meta data record.
Needs to be updated after downloading (which can download multiple years). Each parser, however, works on a single year, so adding a year para should be simple
As a beginner user, I somehow feel it would be nice to have a method, other than get_regions() that would return more info about the regions: full name, member of an organisation...
I can propose a first version, and you'll decide whether you want to include it or not
Add some note that all development should first be discussed by filing an issue
Adopt the standalone script for parsing mat-ntnu-mrio structure:
I am trying to calculate consumption-based impacts with Exiobase v3.8.1 for which I followed instructions on https://pymrio.readthedocs.io/en/latest/notebooks/working_with_exiobase.html.
import pymrio
exio3_path = 'data-exiobase/IOT_2021_pxp'
exio3 = pymrio.parse_exiobase3(path=exio3_path)
# Exploring EXIOBASE
exio3.meta
exio3.get_sectors()
exio3.get_regions()
list(exio3.get_extensions())
# Calculating the system and extension results
exio3.calc_all()
# The available impact data can be checked with:
list(exio3.impact.get_rows())
The last row returns the error AttributeError: 'IOSystem' object has no attribute 'impact'
. Does anyone have a suggestion why?
I am trying to follow the Quickstart guide to exploring Exiobase2. when i call pymrio.parse_exiobase2 i get an error which i believe relates to the pymiro package itself missing a file.
the path requested is in the pymiro package in my anaconda environment C:\Users\CummingR\AppData\Local\Continuum\anaconda3\envs\RC37\lib\site-packages\pymrio\mrio_models\exio20\./misc/population.txt
however the path only exists as far as
C:\Users\CummingR\AppData\Local\Continuum\anaconda3\envs\RC37\lib\site-packages\pymrio\mrio_models
where the folder contains a pycache folder, the test_miro folder and an init.py file. Is there a different file which needs to be installed manually?
I have included the full error trace below.
exio2 = pymrio.parse_exiobase2(path=ExioFile,charact=True, popvector='exio2')
Traceback (most recent call last):
File "", line 1, in
exio2 = pymrio.parse_exiobase2(path=ExioFile,charact=True, popvector='exio2')
File "C:\Users\CummingR\AppData\Local\Continuum\anaconda3\envs\RC37\lib\site-packages\pymrio\tools\ioparser.py", line 658, in parse_exiobase2
index_col=0, sep='\t').astype(float)
File "C:\Users\CummingR\AppData\Local\Continuum\anaconda3\envs\RC37\lib\site-packages\pandas\io\parsers.py", line 685, in parser_f
return _read(filepath_or_buffer, kwds)
File "C:\Users\CummingR\AppData\Local\Continuum\anaconda3\envs\RC37\lib\site-packages\pandas\io\parsers.py", line 457, in _read
parser = TextFileReader(fp_or_buf, **kwds)
File "C:\Users\CummingR\AppData\Local\Continuum\anaconda3\envs\RC37\lib\site-packages\pandas\io\parsers.py", line 895, in init
self._make_engine(self.engine)
File "C:\Users\CummingR\AppData\Local\Continuum\anaconda3\envs\RC37\lib\site-packages\pandas\io\parsers.py", line 1135, in _make_engine
self._engine = CParserWrapper(self.f, **self.options)
File "C:\Users\CummingR\AppData\Local\Continuum\anaconda3\envs\RC37\lib\site-packages\pandas\io\parsers.py", line 1917, in init
self._reader = parsers.TextReader(src, **kwds)
File "pandas_libs\parsers.pyx", line 382, in pandas._libs.parsers.TextReader.cinit
File "pandas_libs\parsers.pyx", line 689, in pandas._libs.parsers.TextReader._setup_parser_source
FileNotFoundError: [Errno 2] File b'C:\Users\CummingR\AppData\Local\Continuum\anaconda3\envs\RC37\lib\site-packages\pymrio\mrio_models\exio20\./misc/population.txt' does not exist: b'C:\Users\CummingR\AppData\Local\Continuum\anaconda3\envs\RC37\lib\site-packages\pymrio\mrio_models\exio20\./misc/population.txt'
Exiobase docs for version3 need to be updated to account for zenodo upload and download (see also #64 )
Hi Konstantin,
thank you for your valuable work in developing this library.
I am trying to perform a shock analysis using pymrio and adopting EXIOBASE3 as database.
I was surprised when I realized that increasing the demand considerably in a certain sector implied decrease in use of certain exogenous resources (e.g. water withdrawal).
So, I tried to figure out why. I ended up comparing the baseline database with an identical version (same final demand), in order to check what was going on.
I import the same database in two different variables
baseline = pymrio.parse_exiobase3(path=exio3path)
shock = pymrio.parse_exiobase3(path=exio3path)
Then, I calculate all for both MRIO.
baseline.calc_all()
shock.calc_all()
But now, since I would like to make some changes in the shock version, I reset all to coefficients.
shock.reset_all_to_coefficients()
Of course, to run the model again, I need a final demand. So, in order to check for consistency, I assign the same final demand of the baseline.
shock.Y = baseline.Y
Now I calculate everything expecting to have the same system observable in the baseline.
shock.calc_all()
But when I look at the differences in use of factors, I observe relevant differences in some categories.
DeltaPerF = (shock.satellite.F.sum(axis=1) - baseline.satellite.F.sum(axis=1))/(baseline.satellite.F.sum(axis=1))*100
I print out the most relevant percentual differences between shock and baseline by factors.
This is the situation for the ixi database.
This the situation for the pxp database.
It looks like the F imported from the original database is somehow different from the F that one gets by F = Sx.
I checked and both S and x have no differences. Therefore, I would expect that there’s a problem in how F is recalculated after bringing everything to coefficients.
Maybe I am doing something wrong, could you help me fix it?
Thank you in advance,
Nicolò Golinucci
Provide EEBT trade matrix for every stressor
Hello Konstantin & contributors,
I have a question concerning the calculation of the green water footprint of Germany using PyMrio and Eora26.
eora_storage = ''
eora = pymrio.parse_eora26(year=2015, path=eora_storage)
eora_green = eora.Q.diag_stressor(('WFN: Total water footprint - Green','Total'), name = 'emtype2_diag')
eora.eora_green = eora_green
eora.calc_all()
source_green = eora.eora_green.D_cba.DEU.sum(axis = 1)
source_green = pd.DataFrame(source_green)
Looking at the results, only 186 out of 2914 sectors have a green water footprint. Of these 186 sectors 172 are "Agriculture" (the other 14 are mainly electricity and gas production and fishing). The way I understand national footprint calculations is that the results show the sectoral contributions to the total footprint of a country (or another specified final demand; see Kitzes 2013). If my understanding is correct, other sectors which consume products from the agriculture sectors should also have a green water footprint e.g. DEU Food & Beverages, Hotels and Restaurants, Textiles and Wearing Apparel etc.
However with the code above, this is not the case. Do you have an idea why?
I also calculated the green water footprint for Germany manually also under use of the Eora26 database (following Kitzes 2013 and multiplying F with the combined final demand of all final demand subcategories). The total water footprint is the same but the sectoral contributions to the national green water footprint are completly different.
The most contributing sectors (including supply chain) are (top - highest contribution):
I thought my results (displayed in the picture) e.g. Germany Food & Beverages, Agriculture, Hotels and Restaurants etc. are plausible as the contributions by these sectors include agricultural products from other countries via the supply chain. However, when I saw the PyMrio results I got confused.
Any idea why our results are so different? Thank you so much for your time! :)
Kitzes, Justin. 2013. “An Introduction to Environmentally-Extended Input-Output Analysis.” Resources 2 (4): 489–503. https://doi.org/10.3390/resources2040489.
Issue on line 109 in iomath.py
I changed:
recix[recix==np.inf]=0
to:
if x == 0:
recix = 0
else:
recix = 1/x
since it was returning a trying to divide by zero error that stopped computation when parsing in EXIOPOL pxp 2.2.2
Change path to include master
It seems conda does not come with a recent enough version of pandas to install pymrio. Might be worth updating the instructions?
When I try:
guillaume@ciraigLinux:~$ conda create --name io -c konstantinstadler pymrio
I get:
Fetching package metadata ...........
Solving package specifications:PackageNotFoundError: Packages missing in current channels:
- pymrio -> pandas >=0.22.0
- pymrio -> requests >=2.18
We have searched for the packages in the following channels:
- https://conda.anaconda.org/konstantinstadler/linux-64
- https://conda.anaconda.org/konstantinstadler/noarch
- https://repo.continuum.io/pkgs/free/linux-64
- https://repo.continuum.io/pkgs/free/noarch
- https://repo.continuum.io/pkgs/r/linux-64
- https://repo.continuum.io/pkgs/r/noarch
- https://repo.continuum.io/pkgs/pro/linux-64
- https://repo.continuum.io/pkgs/pro/noarch
Maps with countries coloured according to their footprint
Brilliant solution! Do you support (or plan on supporting) the hybrid (non-monetary) exiobase database?
Rename to production
Write to doc that these are F with household, avoid doublng of data
Hi,
I am trying to use the parser of exiobase3 and I get a permission error. It seems that the parser only accepts zip files? Yet, when I use the zip file directly downloaded from exiobase website it tells me KeyError: "There is no item named 'mrIot3.0.txt' in the archive", and when i use the unzipped file i get the following error:
PermissionError Traceback (most recent call last)
in ()
----> 1 io = pymrio.parse_exiobase3('C://Users/Maxime/Desktop/Thesis/Bdd/IOT_2011_pxp/')
C://Users/Maxime/Desktop/Thesis/Modules_Python/pymrio-master\pymrio\tools\ioparser.py in parse_exiobase3(zip_file, path_in_zip, version, iosystem, year, charact)
498 # read the data into a dicts as pandas.DataFrame
499 logging.info('Read exiobase3 from {}'.format(zip_file))
--> 500 zip_file = zipfile.ZipFile(zip_file)
501
502 core_data = {exio_table: pd.read_table(
C:\Users\Maxime\Anaconda3\lib\zipfile.py in init(self, file, mode, compression, allowZip64)
1088 while True:
1089 try:
-> 1090 self.fp = io.open(file, filemode)
1091 except OSError:
1092 if filemode in modeDict:
PermissionError: [Errno 13] Permission denied: 'C:\Users\Maxime\Desktop\Thesis\Bdd\IOT_2011_pxp'
In the ioparser.py,
files_exio = dict(
# exiobase 2.2.2
A='mrIot_version2.2.2.txt',
Y='mrFinalDemand_version2.2.2.txt',
S_fac='mrFactorInputs_version2.2.2.txt',
S_emissions='mrEmissions_version2.2.2.txt',
S_materials='mrMaterials_version2.2.2.txt',
S_resources='mrResources_version2.2.2.txt',
FY_emissions='mrFDEmissions_version2.2.2.txt',
FY_materials='mrFDMaterials_version2.2.2.txt',
charact = 'characterisation_CREEA_version2.2.2.xlsx'
)
but when we download the last version Exiobase 2, the name of the mrFDEmissions is mrFDEmissions_version2.2.2.txt
and not mrFDEmissions_version2.2
. An error is raised and the code aborted due to the code:
# check if source exiobase is complete
_intersect = [val for val in files_exio.values()
if val in os.listdir(path)]
if len(_intersect) != len(files_exio.values()):
raise ParserError('EXIOBASE files missing')
I proposed to add mrFDEmissions_version2.2.txt
in the dictionary to avoid the error.
rewrite index population and check script if the wrong name is used anywhere
Parsing directly from the zip file does not work (at least for EXIOBASE 3) because the path separator is different in Windows and in the zip file.
eora26 download now requires log in - check if the downloader would work after login or remove it
SY is currently used in ioutil and mentioned in the description, but not calculated in iomath. Also the save and load function need to be updated.
There seems to be two issues with the handling of Exiobase 2 by pymrio.
First, when the PxP IO system, which I call exio2p*, is reset, using the method reset_all_to_coefficients()
, the matrices A and L are missing (I haven't tested it for IxI but it might be the same).
Second, when one tries to use the method aggregate, an error is raised. If the method is called before calling .calc_all()
, the error reads:
WARNING:root:No attributes available to get sectors
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-61-441d56c89f18> in <module>()
----> 1 global_io = exio2p.aggregate(region_agg = 'global', sector_agg='total', inplace=False)
~/anaconda3/lib/python3.5/site-packages/pymrio/core/mriosystem.py in aggregate(self, region_agg, sector_agg, region_names, sector_names, inplace, pre_aggregation)
1659 region_agg = [region_agg] * len(self.get_regions())
1660 if type(sector_agg) is str:
-> 1661 sector_agg = [sector_agg] * len(self.get_sectors())
1662
1663 if ioutil.is_vector(region_agg):
TypeError: object of type 'NoneType' has no len()
whereas if ones applies .calc_all()
first, the error is:
----> 1 global_io = exio2p.aggregate(region_agg = 'global', sector_agg='total', inplace=False)
~/anaconda3/lib/python3.5/site-packages/pymrio/core/mriosystem.py in aggregate(self, region_agg, sector_agg, region_names, sector_names, inplace, pre_aggregation)
1752 data=conc.dot(self.x),
1753 index=mi_reg_sec,
-> 1754 columns=self.x.columns,
1755 )
1756 self.meta._add_modify('Aggregate industry output x')
AttributeError: 'Series' object has no attribute 'columns'
Same errors are raised if other parameters are passed in the .aggregate()
method (such as a more complicated aggregation).
*exio2p is defined as follows:
exio2p = pymrio.parse_exiobase2(path=[path to the PxP .zip], charact=True, popvector='exio2')
Hello and thank you for providing Pymrio for free and open source!
I am wondering why the full version of Eora is not supported by Pymrio? I am working with the full Eora database and I am happy to write the code for the calculations myself. It helps me to understand what I am actually doing. Still, I noticed that the full Eora is not supported by Pymrio and I am simply curios what the reason is.
Thank you and best regards
Jonas
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.