usepa/emf

Emissions Modeling Framework (EMF)

Emissions Modeling Framework

The Emissions Modeling Framework (EMF) is a client-server system designed to store information related to emissions modeling, with integrated quality control processes. The EMF can drive and track emissions modeling processes, providing organization for model run settings, input datasets, and generated outputs. The Control Strategy Tool (CoST) is a component of the EMF used for estimating the emission reductions and economic costs associated with different control scenarios.

To download the latest version of the EMF and CoST, visit https://www.cmascenter.org/cost/. The CMAS Center website also provides documentation on installing and using the EMF.

emf's Issues

screen reader: set accessible name and description for focusable labels

Calling JLabel.setLabelFor(component) when the component is a label doesn't seem to update the accessible name. Update the custom class Label to directly set the accessible name and description.

window focus fix doesn't work in Java 8

Steps to reproduce:

Run the EMF Client on Java 8
Use the Dataset Manager to select a dataset and view its properties
Close the Dataset Properties View window
The Dataset Manager window doesn't have focus (no window has focus). This means keyboard navigation doesn't work. Selecting the window from the Window menu gives focus back to the window.

Commit f76c567 switched the JInternalFrame used within the desktop manager, which fixed this problem on Java 11.

JAWS isn't reading read-only label values on properties window

Currently, the Label object gets accessibleName set to the field's label (e.g. Creator) and accessibleDescription set to the actual value (e.g. username).

Try concatenating the two values as the accessibleName.

Dataset properties editor/viewer enhancement

Have a question: For the sources tab, there is a name filter field in the source code, but not showing up in GUI. Do we want this field to be used?

improvements to obsolete sector handling

From Chris Allen:

Related to obsolete sectors hanging around in cases, we could add these potential EMF enhancements to our “wish list”. These features would make this sort of thing easier to manage:

• There is a list of sectors in the Summary tab, but if you remove a sector from that list, it does not remove jobs/inputs/parameters associated with that sector. It would be nice if removing a sector from the Summary tab also removed everything else in the case associated with that sector.
• It would also be nice to have an easier way to rename a sector. For example, automatically take all jobs/inputs/parameters associated with “ptfire_mxca” and change to “ptfire_othna”. Right now renaming a sector is a bit tedious because we have to edit each job/input/parameter individually.

keyboard navigation: use focusable labels for dataset properties view Summary tab

Labels in the status section already use custom Label objects instead of JLabel. Switch JLabels in the overview and time period sections to also use Label.

dataset manager: change how pull-down list of dataset types handles typed-in letters

For the Show Datasets of Type pull-down in the Dataset Manager window, typing a letter immediately selects the first dataset type starting with that letter. See if this can be changed to just go to that letter in the pull-down and not select the item.

check controlled inventory deletion

What happens if you delete a control strategy, including any controlled inventories, and that inventory is used as input to another control strategy? Is the controlled inventory deleted or not?

keyboard navigation: access to main menus

If the currently open window has a button shortcut which is the same as the main menu (F, M, W, T, or H), you can't access the menu.

add example counties dataset to CMAS package

For the CoST training, the system needs a counties list dataset loaded. This should be dataset type "List of Counties (CSV)".

Bug: Least Cost Strategy can't handle larger point key Ids and scc code

Least Cost strategy is failing when SCC code larger than 10 characters (i.e., from a FF10 inventory allows 12 character scc codes) tries to get loaded into the ORL Merged Inventory format.

CoST: Project Future Year Inventory - Missing Design Capacity

Design Capacity might not getting passed thru when controlled inventory is created. Double check if this is an issue.

Test EMF under Java 9 (HiDPI)

The current EMF client doesn't scale when run on HiDPI displays on Windows. Java 9 adds support for automatic pixel doubling. The client needs to be tested on Java 9 to see if there are any issues that come up (possibly some deprecated functions that have been removed).

Eventually, the server component should get tested running on Java 9 as well.

internal_sources.table_name gets truncated

Postgres table names can only be 64 characters long, but internal_sources.table_name doesn't have any restriction. This could lead to a name collision in the database.

Disable new dataset version if no final versions

If a dataset has no versions marked as final, the New Version button should be disabled in the Dataset Versions Editor window. This situation can come up when a module creates a new dataset because the version the module creates isn't marked as final.

EMF: What to do with existing contents of exports folder from old tomcat 6 webapps folder

rename fields in Strategy Detailed Result dataset type

Rename the inventory source data fields to match the FF10 format:

FIPS -> REGION_CD
PLANTID -> FACILITY_ID
POINTID -> UNIT_ID
STACKID -> REL_POINT_ID
SEGMENT -> PROCESS_ID

Additionally, rename effective cost-per-ton and emission reduction fields to match new controlled fields (i.e. Ctl_Ann_Cost_per_Ton and Ctl_Emis_Reduction).

Ann_Cost_per_Ton -> Eff_Ann_Cost_per_Ton
Emis_Reduction -> Eff_Emis_Reduction

update precision for control strategy fields

control_strategies.total_cost and control_strategies.reduction are currently reals, which only gives 6 digits of precision. Look at updating to double precision. Any issues with Hibernate layer?

Same issue for strategy_results.total_cost and strategy_results.total_reduction.

keyboard navigation items

This issue is for noting any places where UI can't be controlled solely with the keyboard.

• Clicking on a column name in a table sorts the table by that column. Not sure if this is currently accessible via keyboard.

• [DONE] When viewing the results of a QA step, the File > Close menu option is grayed out. The only way to close the window is to click the Close box.

• [DONE] Missing button shortcuts in View QA Step window (View Results, Export)

• [Created separate issues #46] If the currently open window has a button shortcut which is the same as the main menu (F, M, W, T, or H), you can't access the menu.

fix case-sensitivity for reference search

In ReferencesDAO.getReferences, the search term is set to lower case but the LIKE restriction is case-sensisitive. This means matching for words that are capitalized in the database won't work (i.e. MACTEC). Switch to Restrictions.ilike for case-insensitive matching.

consolidate EMF documentation

EPA installation guide
https://www.cmascenter.org/emf/install/

Main EMF user's guide (includes MARAMA installation instructions)
https://www.cmascenter.org/emf/internal/guide.html

CoST user's guide (includes all-in-one installation instructions)
https://www.cmascenter.org/cost/documentation/2.12/html/

SQL Basics for the EMF
https://www.cmascenter.org/emf/internal/sql_basics.html

Temporal Allocation Using the EMF
https://www.cmascenter.org/emf/internal/temporal_allocation.html

Inventory Projection Using the EMF
https://www.cmascenter.org/emf/internal/inventory_projection.html

SMOKE-MOVES and the EMF (has info about case management)
https://www.cmascenter.org/emf/internal/smoke_moves/

create Plant Closure Extended dataset type

Add support for new dataset type in CoST inventory projection.

look at making mandatory dataset fields not null

Automatically add not null constraint to flexible format columns marked as mandatory?

Missing ComponentUtility class

The user optimization updates refer to gov.epa.emissions.framework.client.util.ComponentUtility which didn't get added in the commit.

MARAMA connection issues

MARAMA reported some of its clients are encountering connection issues with the EMF.

I did see some exceptions in the various tomcat logs that warrant further research.

Document "built-in" flex format dataset types

With the new flex format editing option, it might be good to document which dataset types users need to be careful editing. For example, types used in control strategies or temporal allocation where the column names are referred to in stored procedures or code.

Bug when using the Dataset Manager Advanced Dataset Search "Filter" functionality

EPA's emf doesn't have this issue.

To recreate issue:

1. Login into MARAMA's emf
2. Go to the Adanced Dataset Search window of Dataset Manager
3. Choose "Flat File 2010 Point" dataset type
4. Type in filter: SCC in ('10200101', '10200104')
5. Run search.

Here is the exception from MARAMA's EMF tomcat server log file:

28 May 2014 18:09:36,077 ERROR DataServiceImpl:1650 - Could not find similar datasets.
java.sql.SQLException: Error with Data Value Filter, incorrect filter SQL syntax: SCC in ('10200101', '10200104')
at gov.epa.emissions.framework.services.data.DatasetDAO.filter(DatasetDAO.java:1825)
at gov.epa.emissions.framework.services.data.DatasetDAO.findSimilarDatasets(DatasetDAO.java:1794)
at gov.epa.emissions.framework.services.data.DataServiceImpl.findDatasets(DataServiceImpl.java:1647)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.axis.providers.java.RPCProvider.invokeMethod(RPCProvider.java:397)
at org.apache.axis.providers.java.RPCProvider.processMessage(RPCProvider.java:186)
at org.apache.axis.providers.java.JavaProvider.invoke(JavaProvider.java:323)
at org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
at org.apache.axis.handlers.soap.SOAPService.invoke(SOAPService.java:454)
at org.apache.axis.server.AxisServer.invoke(AxisServer.java:281)
at org.apache.axis.transport.http.AxisServlet.doPost(AxisServlet.java:699)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
at org.apache.axis.transport.http.AxisServletBase.service(AxisServletBase.java:327)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:409)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1044)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.doRun(AprEndpoint.java:2441)
at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:2430)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

CoST: Reconcile what SCCs are used in the CMDB SCC Selector Window versus the core reference.scc table

Reconcile what SCC codes that are used betweem CMDB SCC Selector Window (this uses a local file /config/ref/delimited/scc.txt) versus the core reference.scc table.

Maybe we should start pulling the SCCs from the database and not a local file?

update cost year menu in Control Measure Manager

Update the cost year menu in the Control Measure Manager so that the list of years is dynamic rather than static. The cost year menu when editing a control strategy already pulls from the reference.gdplev table starting with 2000 onward. The Control Measure Manager menu can work the same way.

screen reader: make text in dialog boxes accessible

Not sure if it's a Java problem or an NVDA issue, but dialogs that use the JOptionPane types QUESTION_MESSAGE or PLAIN_MESSAGE (no icon) don't get their text read. Looking at where JOptionPane.showConfirmDialog() is used, most are warning messages that need the user to confirm (e.g. Are you sure you want to delete this item?), so converting to type WARNING_MESSAGE is appropriate.

• Replace JOptionPane.QUESTION_MESSAGE with ERROR_MESSAGE, WARNING_MESSAGE, or INFORMATION_MESSAGE as appropriate
• Check for calls to JOptionPane.showConfirmDialog() that don't set the type and add explicit types

test client installer for TLS v1.2

The system that hosts cmascenter.org has been upgraded to require TLS version 1.2 for HTTPS traffic. Since the EMF client installer downloads files from cmascenter.org, double-check that it works with the update.

inventory filter tool tip

The tool tip contains ANN_EMIS which needs to be updated to use ANN_VALUE for the all-in-one package. See lizadams#25

associate labels with components

A lot of the EMF UI code uses a helper function (SpringLayoutGenerator.addLabelWidgetPair) that calls setLabelFor on the various JLabels and components as they get created. There are a few places that directly create labels and components without calling setLabelFor. These spots either need to switch to using addLabelWidgetPair or call setLabelFor directly.

admin

• RegisterUserProfilePanel
• SelectDTypePanel
• EditableUserProfilePanel (note - ViewableUserProfilePanel calls addLabelWidgetPair)

case management

• CaseDescriptionWindow
• CaseManagerWindow
• CompareCaseWindow
• ImportCasePanel
• RegionEditorPanel

cost

• MeasureReferenceAddWindow
• MeasureReferenceWindow

data

• PaginationPanel
• DatasetsBrowserWindow
• DatasetTypesManagerWindow

exim

• ImportInputPanel

login

• [DONE] LoginWindow

[DONE] InstallWindow (EMF Client Installer)

classification for dataset types

With the introduction of modules and the need for more custom dataset types, consider options for categorizing and protecting dataset types.

• Distinguish between user-created types and system types. Allow users to edit their own types without admin privileges.
• Add tagging or other categories to the types, then use those types in the User Preferences panel to let users hide categories (i.e. dataset types for modules).

Advanced Search data filter doesn't work with shared table dataset types

For example, the dataset type Plant Closures (CSV) stores all of its datasets in a single database table. The SQL query constructed by the Advanced Dataset Search needs to limit itself to the correct dataset id.

From DatasetDAO.filter()

String sqlStr = "SELECT * FROM " + this.qualifiedEmissionTableName(ds) + " WHERE " + dataValueFilter + " LIMIT 1;";

unable to purge control measures associated with a strategy

Trying to remove temporary "Copy of ..." measures from EPA server. Changed all to temp sector "asm_backup". Imported empty CSVs and set sector to Purge. Error when trying to finalize strategies.

gov.epa.emissions.framework.services.EmfException: Cannot update without owning lock
	at gov.epa.emissions.framework.services.persistence.LockingScheme.doUpdate(LockingScheme.java:90)
	at gov.epa.emissions.framework.services.persistence.LockingScheme.renewLockOnUpdate(LockingScheme.java:97)
	at gov.epa.emissions.framework.services.cost.ControlStrategyDAO.updateWithLock(ControlStrategyDAO.java:187)
	at gov.epa.emissions.framework.services.cost.ControlStrategyDAO.finalizeControlStrategy(ControlStrategyDAO.java:536)
	at gov.epa.emissions.framework.services.cost.controlmeasure.io.CMImportTask.run(CMImportTask.java:118)
	at gov.epa.emissions.framework.services.GCEnforcerTask.run(GCEnforcerTask.java:23)
	at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Thread.java:745)

detect non-ASCII characters in control measure data

Look at options for detecting during import and updates through UI.

Discuss unmaintained control strategy types

Annotate Inventory and Apply Measures In Series haven't been updated to support FF10 inventories, only ORL.

Annotate, Apply in Series, and Project Future Year Inventory use out-of-date cost equations implementation.

Dataset Manager exports containing old records

This concerns exports of datasets directly from the Dataset Manager (as opposed
to from an EMF case).

When using certain types of row filters, I am finding that the resulting
exported dataset sometimes contains records from older versions of the same
dataset that have since been edited or deleted.

Example: dataset nonroad_ff10_california_2011neidraft_augmented_VOC, version 7.
When exporting using this row filter:

scc like '226%' or scc like '228%'

then records from older versions of the dataset appear in the results. For
example, we get duplicated PM records:

"US","06115",,,,"2282010005","X","EXH__PM10",1.43814,...
"US","06115",,,,"2282010005","X","EXH__PM2_5",1.37229,...
"US","06115",,,,"2282010005","X","EXH__PM10-PRI",1.43814,...
"US","06115",,,,"2282010005","X","EXH__PM25-PRI",1.37229,...

PM10/PM2_5 were renamed PM10-PRI/PM25-PRI in Version 2 of this dataset, and so
Version 7 only has PM10-PRI and PM25-PRI, not PM10 or PM2_5. This is a double
count.
When exporting the same dataset/version without that row filter (or with a less
complicated row filter), we only get the PM10-PRI and PM25-PRI records and not
the PM10/PM2_5 records.

Two other notes about the Dataset Manager exporter:

• The "overwrite files if they exist" option does not work. If selected, and
  the file exists, the resulting status message is "FILE EXISTS" and the file is
  not overwritten.
• The "Export" button becomes disabled after clicking it once, so if one wishes
  to re-export with a different row filter or to a different directory, one has
  to close the window and re-enter everything again. We would prefer the "Export"
  button remain active after clicking it once.

dataset type manager: no focus after using quick filter

After using the filter option in the Dataset Type Manager window, no UI element has focus, so keyboard navigation doesn't work anymore. Note that other places that use this same UI like the Control Strategy Manager and the Control Program Manager don't have this problem.

Repo is missing deploy/db directory

Probably want to prune some old backup files and emf_upgrade_beta directory. If we're not changing the structure, I'll add the deploy/db/cost/ directory so I can check in the CoST updates.

screen reader: set accessible description for rows in main datasets table

Currently, each cell has the dataset's name as the tooltip. This automatically gets used as the accessible description and then gets read out for every cell. See if the accessible description can be set to something else, or blank. Don't change the tooltip behavior.

screen reader: make sort-filter-select status label accessible

The status label is the info line displayed at the bottom of the sort-filter-select tables that lists the total number of rows and columns in the table, the number of selected items, and any filtering or sorting applied.

35 rows : 12 columns: 0 Selected [Filter: None, Sort: None]

Can't view QA Step Result -- dataset table name too long?

Couldn't view QA Step result, even though it seems to have finished (not a refresh issue, I tried to view with no success):

Dataset = Cntld_NEast_NOx_nonpoint_2025_MaxEmsRdxn_70ppb
QA Step = compare w/base

Notes:

1. CSC has renamed dataset, which in turns changed the table name to a different name as shown in the error below.
2. The original table name (Stored in EMF) is greater than 64 characters, maybe this causes the error thrown below since the table name is too long. Look at code that names table name and make sure it doesn't exceed 64 characters.
3. Most likely this is table length issue, since renaming dataset changed the table name to smaller than 64 characters and its functioning as expected now.

Some supporting entries from sage's catalina.out:

java.sql.SQLException: No columns found in table 'Cntld_NEast_NOx_nonpoint_2025_MaxEmsRdxn_70ppb_20140509102515436'
at gov.epa.emissions.commons.db.TableMetaData.getColumns(TableMetaData.java:46)
at gov.epa.emissions.commons.db.TableMetaData.getColumnMap(TableMetaData.java:53)
at gov.epa.emissions.framework.services.qa.comparedatasets.SQLCompareDatasetsProgramQuery.getDatasetColumnMap(SQLCompareDatasetsProgramQuery.java:789)
at gov.epa.emissions.framework.services.qa.comparedatasets.SQLCompareDatasetsProgramQuery.createProgramQuery(SQLCompareDatasetsProgramQuery.java:336)
at gov.epa.emissions.framework.services.qa.comparedatasets.SQLCompareDatasetsProgramQueryRunner.query(SQLCompareDatasetsProgramQueryRunner.java:24)
at gov.epa.emissions.framework.services.qa.comparedatasets.SQLCompareDatasetsProgramQueryRunner.run(SQLCompareDatasetsProgramQueryRunner.java:45)
at gov.epa.emissions.framework.services.qa.RunQAStepTask.runSteps(RunQAStepTask.java:94)
at gov.epa.emissions.framework.services.qa.RunQAStepTask.run(RunQAStepTask.java:74)
at gov.epa.emissions.framework.services.qa.RunQAStep.run(RunQAStep.java:35)
at gov.epa.emissions.framework.services.GCEnforcerTask.run(GCEnforcerTask.java:23)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Thread.java:722)

QA Step 'compare w/base' query:
Ran QA step, compare w/base, in 0 secs

integrate Apache Ivy into master branch

Darin's commit 7d48de3 sets up the Ivy configuration files with the existing libraries used by the EMF.

review widget enabling after refresh

In the Dataset Type Manager, the refresh code re-enables all components once the data has been fetched. When the buttons are initially set up, Edit, New, and Delete are disabled for non-admins, but they get re-enabled automatically.

We should review other places that use this setup, to see if there are other instances of access not be restricted as intended.

fix spreadsheet screenshots in CoST User's Guide

https://www.cmascenter.org/cost/documentation/3.2/CoST%20User's%20Guide/#exercise-4.-examining-co-benefits-of-a-control-strategy

Multiple QA argument editing windows

Regarding Google Docs task "Issue editing QA Step argument with multiple QA Step Editors open at the same time." and commit f4a8fd5 "QA argument set window with unique names". Should users be able to open multiple Edit QA Argument windows at the same time?

remove unused files from client installation

Looks like none of the files from the config directory (ref/delimited/ or preferences/) are needed by the EMF client, with the exception of config/ref/delimited/scc.txt. Issue #4 suggests changing things so even this one file isn't needed. For now, all the other files can be removed from the client installation site.

issues with exporting after installing standalone package

From Robin/Charlie:

If you try to export, for example, the At-A-Glance report it fails because it tries to write it to a non-existent folder (C:\Program Files\Apache Software Foundation\Tomcat 8.5\webapps\exports\admin). If you add that directory yourself it will write it to that directory, but then it doesn’t know that it should be copied (or more ideally moved) to what would be the EMF Output Files (or maybe EMF_Data) directory on our machines. If you edit the EMFPrefs.txt file in the directory containing EMFClient.bat to include a line that says:
local.tmp.dir=(whatever directory you want the files to end up)
then the file will be copied there after being exported to the C:\Program Files\Apache Software Foundation\Tomcat 8.5\webapps\exports\admin directory. Editing the EMFPrefs.txt file is not mentioned in the documentation.

When Charlie tried to export the control measures on his laptop, it failed. It creates the _summary.csv file in the exports\admin directory but fails to write anything to it, so he didn't even get to the point where he would know if it eventually ends up in the right place. Charlie is still trying to figure out how to make this work.