Giter Site home page Giter Site logo

aleip / eealocatorplots Goto Github PK

View Code? Open in Web Editor NEW
2.0 3.0 1.0 43.17 MB

EU-GIRP (EU-Greenhouse gas Inventory Reporting Plots): R program to generate plots from EEA-locator CRF data

License: GNU General Public License v2.0

Shell 0.45% R 40.22% Batchfile 0.01% GAMS 1.82% HTML 57.50%

eealocatorplots's Introduction

EU-GIRP (EU-Greenhouse gas Inventory Reporting Plots

R program to generate plots from EEA-locator CRF data

This program has been conceptualized and programmed by Adrian Leip European Commission, Joint Research Centre (JRC), Institute for Environment and Sustainability, Ispra (VA), Italy [email protected] to support the EU GHG inventory system.

Start of project: October 2014 First version available (EU-GIRP.v1.3: May 2015)

Last version available (EU-GIRP.v1.5: September 2015) Public repository on https://github.com/aleip/eealocatorplots.git

This folder contains the source code that are used to generate EEA plots for the EU GHG inventory (from 2015 onwards).

The following software is required to run the code:

  • R statistical software, see
  • bash scripting environment

The following files are required to start the process:

  • eealocator_20150903.txt: This is a comma-separated file containing all data. It is generated by a bash-script (eealocator_csv.bash) which prepares relevant data from a EEA-locator output (e.g. CRF_MMR_20150423.txt) into a working file. The name of the working file contains EEA-locator meta-information. This file has been generated on the basis of the following files:
    • eealocator_csv.bash: bash script extracting the data into the working file (see above). The script writes also to other files which are required for the plotting script: lists.txt and meta_data_dimensions.txt (see below). lists.txt contains all categories, measures, units, and parties that are included in the working file (in R format).
    • CRF_MMR_20150423.txt (dummy name actual eea-locator text-file output): text file generated in parallel to the EEA-locator by the EEA
  • eealocatorplots.r: main program to for the plots: EUIRplot-tool. There are several sub-programs doing different tasks. They are called eugirp_XX.r (XX=specific task). E.g.
    • eugirp_nirplots.r: part of EU-GIRP that does the plots, including graphical settings etc.
    • eugirp_attributes.txt: contains the information on the colours and patterns to be used for the barplots. can be edited to change layout of of the plots
  • curplot.r: File with information steering the EU-GIRP tool. This file has to be edited by the user and adapted to the own needs. For further information see curplot.r
  • measures_20150731.txt: contains the info how the measures are related to 'activity data' and 'emissions'. This is important as each parameter needs to be related to a corresponding activity data. Most measures though are captured automatically by the tool.

Principles of the EUIRplot-tool

There are various types of plots that are possible, which are available for two types of data. For each source category (e.g. animal type in category 'Enteric Fermentation') the following data are required: activity data (e.g. Population), emissions (by gas) and implied emission factor (by gas). They need to be indicated with the acronyms 'AD', 'EM', and 'IEF'. For other parameters (any acronym) they need to be linked to any AD (plot request in appropriate line of curplot.csv. The EUIRplot-tool distinguishes the following two types of data:

  • Summable data (ADEM): Activity data, emissions etc. This are data for which it 'makes sense' to add the values of different countries and to calculate a 'contribution' from different countries. The available plots for summable data are:

    1. 'value': In an ADEM-value plot, the sum of the values V{t} of all countries (EU18+IS) are plotted in a barplot against the years. Thus also the 'value' plot shows the trend in absolute terms. It is done$ for all Activity data and emissions. The 10 countries (number can be modified) contributing (on average over the whole period) most to the EU28+IS sum are shown in a pre-defined pattern and their share is given next to the legend indicating also the name of the countries. All other countries are lumped to the category 'Other'.
  1. 'trend': In an ADEM-trend plot, for each year t the sum of change (in absolute values) of a parameter (V{t} - V{t-1}) is plotted. It is done$ for all Activity data and emissions. For emissions, the trend is determined both by changes in AD and IEF - thus the plot shows the 10 countries (number can be modified) contributing most to the average annual change over the whole period in a pre-defined pattern. Next to the legend the contribution to the average annual change is given, as well as the contribution to the average change in the country do to the trend in AD or IEF (not yet implemented).
  • Averageable data (IEF): Implied emissions factors, other parameters and factors. This are data for which a sum does not make sense, but the EU28+IS value must be calculated as (weighted) average.

    1. **'value'**: In an **IEF-value** plot, the EU28+IS average (AD-weighted average) of the value (IEF or other factor) is plotted together with some statistical measures of the distribution across countries (boxplot): median (geometric mean), and lower and upper quartiles. Furthermore, the countries with the highest relative deviation from the EU28+IS average are plotted. In the legend, the mean, min and max value of the parameter is indicated. In practice, for each data point its relation to the EU28+IS average of the year is calculated and countries with the highest absolute difference to 1 are selected for the plot to ensure that all most important outliers are included.
    
    2. **'trend'**: In an **IEF-trend** plot, the EU28+IS average (arithmetic mean) of the inter-annual change of the value (IEF or other factor) is plotted together with some statistical measures of the distribution across countries (boxplot): median (geometric mean), and lower and upper quartiles. Selection of countries and content of legend is in analogy to the IEF-value plot. It may happen in particular for a IEF-trend plot (but in theory also for an IEF-value plot) that not all 'outliers' for a single year are covered by the 10 countries with the highest outliers; these points are then marked with another symbol not indicating which country it relates to, however these are usually relatively small inter-annual changes.
    
    3. **'countries'**: In an **IEF-countries** plot, for each country the range of values (IEF or other parameter) over the whole time period is plotted (min-max and median). 
    

eealocatorplots's People

Contributors

aleip avatar xavi-rp avatar efsolazzo avatar carmonagarcia avatar

Stargazers

Panqiao Wang avatar  avatar

Watchers

 avatar  avatar  avatar

Forkers

efsolazzo

eealocatorplots's Issues

Improve NIR-agri chapter: Sentences.

Section 5.2 Emission trends contained the following sentence:

Only emissions from follow the opposite trend, contributing to compensate the emission decrease but with a very low impact (% of agriculture total trend

  • Such a sentence doesn't make sense when there is no source category with opposite trend, thus needs to be deleted
  • It could then be substituted by a sentence saying that there was no source category with opposite trend
  • Requires a check if there is any occurrence to add a sentence or possibly write another one
  • This problem might have occurred at several places (undetected).

function simplifytestmatrix - called from eugirpA.1_eealocator.r

Lines 27ff the function simpifytestmatrix works only with data.frames. Currently called

  • allmethods<-simplifytestmatrix(as.data.frame(allmethods),"year",years2keep)
  • allinfos<-simplifytestmatrix(as.data.frame(allinfos),"year",years2keep)
  • allnotations<-simplifytestmatrix(as.data.frame(allnotations),"year",years2keep)

Share of liquid and solid manure

hi gema, thanks! i think you are right - this formulae is not correct - needs to be corrected.

From: CARMONA GARCIA Gema (JRC-ISPRA)
Sent: 05 December 2018 16:01
To: LEIP Adrian (JRC-ISPRA)
Subject: CAPRI - Inventory comparison

Hi Adrian,

I am checking a bit how things are calculated in CAPRI for comparison with the inventory data. For emissions from MMS, I think there is a problem with the equations in CAPRI_in_NIR_format.gms (folder 'gams\comparisonplots'). When allocating manure to the MMS, we have:

**Calculation of N excretion per MMS (in kt N/year)

caprinv_allagri2(party,variableUID,InvYears_) $ [UID_TO_ALLTYPE(variableUID,"Liquid") ] = SUM((RALL,MPACT) $ [MAP_IPCC_TO_RALL(party,RALL) and UID_TO_MPACT(variableUID,MPACT) and UID_TO_ALLTYPE(variableUID,"Liquid")] , DATA2(RALL,MPACT,"MANN",InvYears_) * DATA2(RALL,MPACT,"LEVL",InvYears_) * p_nemiDAT_time2(RALL,MPACT,"N","TypeShare","Liquid",InvYears_) * 0.001) ;

caprinv_allagri2(party,variableUID,InvYears_) $ [UID_TO_ALLTYPE(variableUID,"Solid") ] = SUM((RALL,MPACT) $ [MAP_IPCC_TO_RALL(party,RALL) and UID_TO_MPACT(variableUID,MPACT) and UID_TO_ALLTYPE(variableUID,"Solid")] , DATA2(RALL,MPACT,"MANN",InvYears_) * DATA2(RALL,MPACT,"LEVL",InvYears_) * p_nemiDAT_time2(RALL,MPACT,"N","TypeShare","Solid",InvYears_) * 0.001) ;

caprinv_allagri2(party,variableUID,InvYears_) $ [UID_TO_ALLTYPE(variableUID,"GRAZ") ] = SUM((RALL,MPACT) $ [MAP_IPCC_TO_RALL(party,RALL) and UID_TO_MPACT(variableUID,MPACT) and UID_TO_ALLTYPE(variableUID,"GRAZ")] , DATA2(RALL,MPACT,"MANN",InvYears_) * DATA2(RALL,MPACT,"LEVL",InvYears_) * p_nemiDAT_time2(RALL,MPACT,"N","TypeShare","GRAZ",InvYears_) * 0.001) ;

As far as I understand, this considers that Typeshare solid+liquid+grazing= 1. If this is right, it is not correct, because in CAPREG (ammo) we have Typeshare housing + grazing = 1, and housing contains liquid + solid (therefore liquid + solid = 1, they are expressed as a fraction of what is excreted other than on pastures).

Could you please have a look when you have some time and see if I am mistaken?

Thanks,

Gema

Potentially significant issues

This is calculated in Step 4 'Calculating trends and growth rates'.
It uses the agrishares versus Total emissions without LULUCF and applied a maxshare.
However, there are a few questions to be solved, such for what years should it be calculated? Is it suffficient that a category is PSI or does it matter how large the under or overestimation is? I.e if a source category is huge but the potential over/underestimation is very small, then it is not PSI - this should be included in the calculation.
Currently disactivated with 'dontdoforthemoment'.
Requires thinking not only at this point, but also when writing out the individual issues.

N application ratio check

This needs to be better explained, maybe the calculation made more transparent. The target value it NOT 1, but a value > 1, as otherwise some losses have not been considered. It can be <= 1 if bedding is considered.

file: eugirp_aggparentanimal.r

This file calculates the average values for animal types and items that are weighted (by population) .
It is called several times, once for 'swine' and 'sheep', once for 'Dairy Cattle' and 'Non-Dairy Cattle', and once for 'Cattle' (aggregating dairy and non-dairy).
The file loops of meastypes, gases, sources etc. in a quite intransparent way.

  • It needs urgently to be improved to make use of data.table features to get more transparent.
  • Also, there needs to be a test added if the calculation is OK - we have discovered cases in the past (usually at a point in time where we could not look more carefully)

Key source categories and flags for issue-files

In step 6 there Key source categories are loaded and flags calculated (to make ticking in the EMRT easier). However, this uses old code and - what is more risky - loads also old files.
The whole part needs to be carefully checked and updated, both in content and programming.

It is disactivated with logicals 'dokeycategories' and 'doaddflags'.

Also we need to see if the outlier files that are generated need all these fields or if it would be more user friendly to reduce the fields (then, they don't have to be calculated).

file eugirpA.2_meastype.r - a few lines can probably made faster

Check:

  • alldata$sector_number<-unlist(lapply(c(1:nrow(alldata)),function(x) gsub("Other \\(please specify\\)","",alldata$sector_number[x])))
  • alldata$category<-unlist(lapply(c(1:nrow(alldata)),function(x) gsub("livestock","Livestock",alldata$category[x])))
  • alldata$sector_number<-unlist(lapply(c(1:nrow(alldata)),function(x) gsub(alldata$category[x],"",alldata$sector_number[x])))

function: prepareplot()

This function in file eugirp_funnirplots.r prepares the data for one plot.
It is coded to work with different datasources for comparison plots, and value or ief plots, calculating ranges etc.
The coding is very intransparent with little documentation and likely contains a lot of code that is not required (any more) or is done in a complicated way. It has been adapted to work with data.table for ADEM but not yet for other modes and not for multiple data sources.

Aggregation of IEF for Non-Dairy Cattle and Poultry

  • Introduce check of final IEF = EM/AD once all IEFs aggregates are calculated.
    In May 2020, for Non-Dairy Cattle and Poultry the EU - IEF obtained from AD and EM deviated slighlty from the one calculated bottom-up from MS IEFs. We haven't checked yet the reason, probably one MS that had an animal type not considered in the aggregation (or otherwise wrongly considered).
    We need a safety check that stops the script when this occurs.

IEF-range plots

Check if the IPCC default ranges are all correct - and correct if necessary!

Improve NIR-agri chapter - policy

  • Update text on CAP with new reform
  • Include Green Deal and Farm to Fork --> targets!
  • Update action plans for NVZs. Use information from the trend issues
  • Update on industrial emissions directive?
  • Integrated Nutrient Management Plan?
  • Other examples retrieved from trend issues.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.