in-core/pyincore

pyIncore is a component of IN-CORE. It is a python package consisting of two primary components: 1) a set of service classes to interact with the IN-CORE web services, and 2) IN-CORE analyses . The pyIncore allows users to apply various hazards to infrastructure in selected areas, propagating the effect of physical infrastructure damage and loss of functionality to social and economic impacts.

pyincore

pyIncore is a component of IN-CORE. It is a python package consisting of two primary components: 1) a set of service classes to interact with the IN-CORE web services described below, and 2) IN-CORE analyses. The pyIncore allows users to apply various hazards to infrastructure in selected areas, propagating the effect of physical infrastructure damage and loss of functionality to social and economic impacts.

Installation with conda

Installing pyincore with Conda is officially supported by IN-CORE development team.

To add conda-forge channel to your environment, run

conda config –-add channels conda-forge

To install pyincore package, run

conda install -c in-core pyincore

To update pyIncore, run

conda update -c in-core pyincore

You can find detail information at the Installation section at IN-CORE manual.

Installation with pip

Installing pyincore with pip is NOT supported by IN-CORE development team. Please use pip for installing pyincore at your discretion.

Installing pyincore with pip is only tested on linux environment.

Prerequisite

• GDAL C library must be installed to install pyincore. (for Ubuntu, gdal-bin and libgdal-dev)
• ipopt executable must be installed to run some analyses such as seaside CGE, joplin CGE, etc.
• For developers, pre-install must be installed. If not, run brew install pre-commit or pip install pre-commit.

To install pyincore package, run

pip install pyincore

Testing and Running

Please read the Testing and Running section at IN-CORE manual.

Documentation

pyIncore documentation can be found at https://incore.ncsa.illinois.edu/doc/incore/pyincore.html

pyIncore technical reference (API) can be found at https://incore.ncsa.illinois.edu/doc/pyincore/.

Acknowledgement

This work herein was supported by the National Institute of Standards and Technology (NIST) (Award No. 70NANB15H044). This support is gratefully acknowledged. The views expressed in this work are those of the authors and do not necessarily reflect the views of NIST.

pyincore's Issues

Refactor damage ratio datasets to use consistent column names and updated utility classes

Buildings and Roads use Mean Damage Factor, Bridges uses Best Mean Damage Ratio, we should use Mean Damage Factor for both and add a column that maps to our DS_ damage states so the values can be referenced by DS_ keys.

We can also remove the type row (java.lang.String) since it's unnecessary.

implement new get_building_period() method

now period and num_stories is a part of expression existing in the fragility curve;

rewrote get_building_period() and need to implement that in various place

dmg_probability = selected_fragility_set.calculate_limit_state_refactored_w_conversion(
hval_dict, **building_args, period=xxx)

 

in building damage this period =xxx seems to be not needed, cuz it's already a part of the equation of the fragility curves

make sure i test removing it

 

using the 5b47b34e337d4a36290754a0 as example 

description	"default building period"
fullName	null
expression	"(0.097) * math.pow(num_stories * (13.0), 0.624)"

 

Plan refactoring of pytests

We need to add pytests for analyses, figure out what we need to assert generally and individually and make individual tasks for them. Also, we should not test against the services, we should mock the responses.

Also, we should make a document about test-driven development so that anyone who wants to contribute to pyIncore can follow it.

Pipeline damage with repair rate only computes total repairs when including liquefaction

Pipeline damage with repair rate should compute total repairs regardless of including liquefaction. Currently, the logic to compute total repairs is inside the conditional if liquefaction is being computed.

wrong output data metadatata

update documentation models

water pipeline restoration model

see https://opensource.ncsa.illinois.edu/confluence/display/INCORE1/Water+Facility+and+Pipeline+restoration+analysis

Add Joplin building functionality recovery time model

Implement only the original, empirical part

Rename building economic loss output to something more general

Building economic loss computes loss from mean damage which can be generally used for mean structural, non-structural and content damage to compute loss from each of those outputs; however, the output for economic loss is saved as "strloss" and "strlossdev".

remove deprecated methods from pyIncore

remove deprecated methods (@deprecated) that are annotated in pyIncore.

remove the following scripts:

fragilitycurve.py

conditionalstandardfragilitycurve.py

customexpressionfragilitycurve.py

parametricfragilitycurve.py

periodbuildingfragilitycurve.py

periodstandardfragilitycurve.py

standardfragilitycurve.py

rename fragilitycurverefactored.py to fragilitycurve.py

rename fragilitycurveset.py's method calculate_limit_state_refactored_w_conversion to calculate_limit_state

make tasks to improve calculate_mean_damage

Many more improvements can be made to "calculate_mean_damage()" method so it is not very rigid.
https://opensource.ncsa.illinois.edu/bitbucket/projects/INCORE1/repos/pyincore/browse/pyincore/utils/analysisutil.py#95
Some are: # remove the 2nd line of the csv that says "java.lang.string,..." etc. incore doesn't need it anymore and is causing us to write workarounds on incore-ui and pyincore

Remove references to Slight, moderate etc. references in the CSVs and also in this method.
The code still refers to the csv lines by indexes, and it will break if the no of the damage states change(to say 3 or 6). It should work with DS values as keys instead of indexes
Standardize the column names. Some use "Best Mean Damage Ratio" and others "Mean Damage Ratio". These are conditionally hardcoded
Bottom line is, this method is very fragile and needs some refactoring after discussion with someone who understands the science and terminology behind it.

add vacant household category to pd race/ethnicity output

Add 0 HU no race/ethnicity vacant data in population dislocation json utility

• add no race/ethnicity vacant to the race_categories
• count huapd["hua_re"] = 0 rows
• add to json

fix docker build for docs

Automatic build for pyincore docker is having an error due to the sphinx path. This should be fixed.

review and implement seaside restoration model

review Hesam's seaside restoration

convert hua and pd outputs to json

Lisa's notebook does not run pyincore analysis. There is a csv file with retrofits Wangoutput_2020-05-28_sheet1.csv which is merged with IN-CORE_2ev3_SetupJoplin_FourInventories_2019-08-06_HUinventory.csv. There are two lines named Population dislocation which is compared to HUA dislocation. That might be the only code to be run in pyIncore.

need to pass the dataset type as a parameter to from_dataframe method

@classmethod
def from_dataframe(cls, dataframe, name):
"""Get Dataset from Panda's DataFrame.

Args:
dataframe (obj): Panda's DataFrame.
name (str): filename.

Returns:
obj: Dataset from file.

"""
dataframe.to_csv(name, index=False)
return Dataset.from_file(name, "csv")

Some improvements and bug fixes when handling error codes and unmapped inventory in pyincore analyses

Issue1: When there are inventory items not mapped to a DFR3, the csv result of the analysis generates an empty row with just "guid" field. Provide feedback to user why there is empty row - may be a debug message?

Issues2: 
if the first row of the output has an error or is unmapped to a fragility, the CSV generation fails. - This can be fixed by using pandas instead of "csv" package. Or continue using "csv", iterate through entire list to find the longest dict and use it to generate the headers (little expensive than pandas option)
example, this works:

{{"guid","LS_0","LS_1","LS_2","LS_3","DS_0","DS_1","DS_2","DS_3","DS_4","haz_expose"
"00919ec2-5c9d-4269-8ddb-e8d2a0d960d4","0.8608868844","0.6585604091","0.5295291166","0.3289412247","0.1391131156","0.2023264753","0.1290312925","0.2005878919","0.3289412247","yes"
"efd6e76c-cf38-4265-be16-b5c6b53843b2","","","","","","","","","","error"
"5f1a56e1-07ec-4e4d-a6a4-ee5146f4da56","0.7273581592","0.5920946565","0.471725649","0.2573799957","0.2726418408","0.1352635027","0.1203690075","0.2143456533","0.2573799957","yes"}}

baseanalysis fails to generate an output like below when the first row doesn't have LS/DS values:

{{"guid","LS_0","LS_1","LS_2","LS_3","DS_0","DS_1","DS_2","DS_3","DS_4", "haz_expose"
"00919ec2-5c9d-4269-8ddb-e8d2a0d960d4","","","","","","","","","","error"
"efd6e76c-cf38-4265-be16-b5c6b53843b2","0.7273581592","0.5920946565","0.471725649","0.2573799957","0.2726418408","0.1352635027","0.1203690075","0.2143456533","0.2573799957", "yes"
"5f1a56e1-07ec-4e4d-a6a4-ee5146f4da56","0.7273581592","0.5920946565","0.471725649","0.2573799957","0.2726418408","0.1352635027","0.1203690075","0.2143456533","0.2573799957", "yes"}}
 

Discussion item:

if there were errors calculating damage for a few inventory items in the analysis, the result will contain some rows with empty values for LS_x & DS_x. Even though the analysis itself works fine, when an output like this chanined to MCS or mean damage it will fail because it expects to find LS/DS values. We should discuss how to handle chained analyses in this case.

Pandas warning - default dtype for empty series will be 'object' instead of 'float64' in a future version

joplin_cge.ipynb and seaside_cge.ipynb under incore-docs/notebooks has warning from pandas

FutureWarning: The default dtype for empty Series will be 'object' instead of 'float64' in a future version. Specify a dtype explicitly to silence this warning.

the dump of warning message is in attached.

pandas-warning.log

port optimization code

Pyomo Code README.pdf
Multiple Objective Model_Joplin Flow Chart.pdf
Multiple objective Model Dataset informaton .pdf
Pyomo Code README_final.pdf
Details of damage estimation and retrofit cost.pdf

Housing recovery model analysis

The housing recovery model using Census appraisal data, Galveston testbed notebook example.
Notes: https://opensource.ncsa.illinois.edu/confluence/display/INCORE1/Housing+recovery+analysis

update repair and restoration calculations to use expression based curves

Once the repair curves in database are migrated to new expression based format, we must update the calculation so it evaluates the expression instead of performing own calculation based on current alpha and beta values

Links to service change: IN-CORE/incore-services#48

Numpy warning - RuntimeWarning Mean of Empty slice

Residential_building_recovery.ipynb has runtime warnings from numpy

residentialbuildingrecovery.py:158: RuntimeWarning: Mean of empty slice
  mean = np.nanmean(group_hhinc_values)
/numpy/lib/nanfunctions.py:1879: RuntimeWarning: Degrees of freedom <= 0 for slice.
  var = nanvar(a, axis=axis, dtype=dtype, out=out, ddof=ddof,

Building economic loss multipliers are percents

The building economic loss uses occupancy multipliers from Ergo/Hazus that are percents and they need to be converted to decimals before used to determine economic loss.

water facility restoration model

details see: https://opensource.ncsa.illinois.edu/confluence/display/INCORE1/Water+Facility+and+Pipeline+restoration+analysis

Provide an object that constructs the get_spec JSON in a generalized manner

A factory object is required to regularize and simplify get_spec across all analyses. The current implementation hampers standardization across multiple files in a systematic manner.

create school and hospital resilience analysis

Create an analysis for school and hospital resilience analysis. This will be a refactoring and modifying the code provided by Emad Hassan. Check and test the code and make it as pyincore analysis

shapely deprecation error with tornadoepn analysis

Faced on Gowtham's Macbook with local pyincore 1.1.0 installation (shapely 1.8.0). Run the "tornado epn" notebook from incore manual to reproduce.

/Users/vnarah2/mambaforge/envs/viz140rc1/lib/python3.8/site-packages/pyincore/utils/geoutil.py:168: 
ShapelyDeprecationWarning: Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0.
 Use the `geoms` property to access the constituent parts of a multi-part geometry.
  for line in line_segment:
/Users/vnarah2/mambaforge/envs/viz140rc1/lib/python3.8/site-packages/pyincore/analyses/tornadoepndamage/tornadoepndamage.py:267: 
ShapelyDeprecationWarning: Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the `geo

add github action to publish to conda

We need to automatize publishing our builds to Anaconda.

Add business closure model analysis

Business operation analysis is based on Mohammad Aghababaei cease operation of businesses model in Lumberton, NC, after 2016 Hurricane Matthew.

Need to add percentage of change in the utils to convert CGE output to json for playbook

https://github.com/IN-CORE/pyincore/blob/develop/pyincore/utils/cgeoutputprocess.py#L15
need to add percentage of change

remove calculate_custom_limit_state_w_conversion() and it's usage in epfdamage's liquefaction usage

Also, test if EPF damage's liquefaction code works. If so, create a test script to test it.

More details on EPF change: #22 (comment)

Add Lisa's corrections to residential building recovery

Perform changes to latest optimized code base to match scientific intent as corrected by Lisa Wang.

• Correct computation of lognormal distribution parameters
• Have correct indexing of dataframes

fix pep8 issues

Update building econ loss to include non structural and content damage

Currently, buildingeconloss only supports structural loss. With additional mean damage inputs for acceleration sensitive, drift sensitive and content mean damages, total loss including structural and non-structural can be computed.

Also, update buliding_loss jupyter notebook example to demonstrate this analysis.

add input dataset to joplin building functionality recovery time model

Add an optional dataset to the model with individual Target functionality levels (column). The idea is to set default target levels from the parameter (required) and use custom levels for subset of buildings (guid) provided in the input dataset (optional)

Pandas warning - The frame.append method is deprecated

residential_building_recovery.ipynb has warning:

FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.

electric power facility model

details see: https://opensource.ncsa.illinois.edu/confluence/display/INCORE1/Water+Facility+and+Pipeline+restoration+analysis and https://opensource.ncsa.illinois.edu/confluence/display/INCORE1/Power+Facility+restoraiton+analysis

Create conda recipe

Create a conda build recipe

hotfix: heatmap not properly filtered

instead of saying ['dislocated'].notnull(), it should be just ['dislocated']

fix total population dislocation is Null

this needs to be pop_disl_tot

it is causing the HUA_total_dislocation.json used in playbook to have null values for population dislocation; hence break the playbook

{"household_dislocation_in_total": {"dislocated": {"population": null, "percent_of_population": null}, "not_dislocated": {"population": null, "percent_of_population": null}, "total": {"population": null, "percent_of_population": null}}, "population_dislocation_in_total": {"dislocated": {"population": 10772, "percent_of_population": 19.197661777548074}, "not_dislocated": {"population": 45339, "percent_of_population": 80.80233822245194}, "total": {"population": 56111, "percent_of_population": 100}}}

Implement INDP model from Hesam

INDP stands for Interdependent network design problem.

Had an initial meeting and code transfer: code in box folder https://uofi.app.box.com/folder/150397462573

fix styling issues

fix styling to comply with pep-8 guidelines

show the api response messages that the services return

When pyincore receives an error message from api, it just prints the error code (500 etc.) and ignores the message included. Below is an example of what pyincore shows to user

ERROR - client.py:return_http_response() - HTTPError: The server returned a 500 failure response code. You can find information about HTTP response status codes here: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status

raise self._exception
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://incore-tst.ncsa.illinois.edu/hazard/api/earthquakes/5dfa4058b9219c934b64d495/values

Add check to mean damage analysis to verify damage keys match inventory type and remove unsupported types

The Mean damage spec lists the supported damage_ratio types as building, bridges and roadways; however, the spec list of inventory types supported does not match. Any unsupported types should be removed until mean damage factors are provided for those types.

In addition, we can add a check that damage_interval_keys should be of length 4 for buildings/roadways and 5 for bridges; any other value we can throw an unsupported exception.

Add github action to run unit tests

Here is the pytest line from the old build server:

pytest --cache-clear -s pyincore/test_client.py pyincore/test_dataservice.py pyincore/test_dfr3service.py pyincore/test_spaceservice.py pyincore/test_hazardservice.py --junitxml=test-reports/results.xml

A couple of things here.

1. We need to probably setup a conda environment because requirements.txt has a reference to ipopt
2. We need to use the test user to connect to the services to run these tests

Create github action for automatic docker build and publish for the documentation

Create an automatic docker build and publish github action for the documentation container

pyincore util to convert population dislocation output for heatmap

rename master branch to main

update fragility specific functions so it deals with the service deprecations and using common DFR3Curve classes

Fix the broken pytests and pyincore analyses that use fragilities.

Links to service change: IN-CORE/incore-services#48

Building economic loss does not include occupancy type multipliers

Building economic loss should include a table of occupancy multipliers for structural, non-structural and content damage broken out by occupancy type (e.g. RES1, RES2, etc) so economic loss can be computed for structural and non structural damage.