npr99 / intersect-community-data Goto Github PK

View Code? Open in Web Editor NEW

18.0 5.0 14.0 299.82 MB

Python functions to obtain and clean data required for the version 2 Housing Unit Allocation. Workflow uses Census API.

Jupyter Notebook 99.32% Python 0.68%

census-api synthetic-population pyincore

intersect-community-data's Issues

Update LODES job record generation and run for multiple counties

Integrate Amin's LODES data code

https://github.com/npr99/intersect-community-data/tree/main/pyncoda/99_SandboxCode/SandboxAE

On my VS Code running code for
Employment Inventory.ipynb

copied code from Amin's sandbox into my work folder on VS Code. The gitnore allows work folders to be ingored

build conda package

Started slack channel on IN-CORE
https://in-core.slack.com/archives/C03MLKDV4ER

Get "Used By" section on repo

requirements.txt

https://github.com/Kanaries/pygwalker/tree/main

https://docs.github.com/en/code-security/supply-chain-security/understanding-your-software-supply-chain/about-the-dependency-graph#supported-package-ecosystems

I noticed in pygwalker github page that they have a used by section. Also the package has a nice readme page.

It looks like a package that uses another package needs to have a requirments.txt file - then github can identify the package that depends on another package and will make a used by section on the repository.

Consider adding Homeless

https://data.census.gov/table?q=foster+child&g=0100000US_0500000US37155_1600000US3739700&tid=ACSST5Y2019.S0901

https://data.census.gov/cedsci/table?q=foster%20child&g=0100000US_0500000US37155_1600000US3739700&tid=ACSST5Y2019.S0901

Homeless students
share housing with other persons due to loss of housing or economic hardship, live in hotels or motels, trailer parks, or campgrounds due to lack of alternative arrangements, those awaiting foster care placement, living in substandard housing, and children of migrant workers (CRS, 2018)

good list of characteristics - some of these are in the housing unit and person record inventory. Shared housing, trailer parks - could be identified in the data. Foster care might be something to summarize - the census bureau has counts of people in foster care.

I will see if I can flag shared housing. The person record inventory has household structure - could be based on number of adults to children. I would assume that shared housing would have multiple adults (3-4+) and multiple children (3+)

According to the National School Boards Association, during the 2018-2019 school year, 77% of homeless students lived in shared housing, 12% lived in shelters, transitional housing, or were awaiting foster care, 7% lived in hotels or motels, and 4% were unsheltered (Cai, 2021).

Cai, Jinghong. (2021). Homeless Students in Public Schools Across America: Down but Not Out. National School Boards Association.
Congressional Research Service (CRS). (2018). Homelessness: Targeted Federal Programs. CRS Report.

Work on entity relationship figure

I have started a entity relationship figure in LUCID chart here is the link:
https://lucid.app/lucidchart/aaa0a9d7-087e-4a23-8e75-67feb6efb684/edit?viewport_loc=19%2C179%2C3762%2C1893%2C0_0&invitationId=inv_9dc49768-f696-4dcd-990f-a92a5d52a56b

explore sex by age - looks like missing values and clusters

Notice that histogram shows clustering of ages around every 10 years. Also looks like there is a gap between 80 and 85

change main folder pyhui

pyhui stands for python housing unit inventory... need to change the name but have not decided on the future package name

options
pyicd - python intersect community data
pyincoda
pyintcomdat
pyintersectcd
pyntrsctcmntydt

pyixcommunitydata

dtype issue

/workspaces/intersect-community-data/pyncoda/CommunitySourceData/api_census_gov/acg_02a_add_categorical_char.py:216: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '['48' '48' '48' ... '48' '48' '48']' has dtype incompatible with int64, please explicitly cast to a compatible dtype first.
add_geovarid_df.loc[:,geo_level_lower] =
/workspaces/intersect-community-data/pyncoda/CommunitySourceData/api_census_gov/acg_02a_add_categorical_char.py:216: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '['167' '167' '167' ... '167' '167' '167']' has dtype incompatible with int64, please explicitly cast to a compatible dtype first.
add_geovarid_df.loc[:,geo_level_lower] =
/workspaces/intersect-community-data/pyncoda/CommunitySourceData/api_census_gov/acg_02a_add_categorical_char.py:216: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '['720100' '720100' '720100' ... '726200' '726200' '726200']' has dtype incompatible with int64, please explicitly cast to a compatible dtype first.
add_geovarid_df.loc[:,geo_level_lower] =
/workspaces/intersect-community-data/pyncoda/CommunitySourceData/api_census_gov/acg_02a_add_categorical_char.py:216: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '['48' '48' '48' ... '48' '48' '48']' has dtype incompatible with int64, please explicitly cast to a compatible dtype first.
add_geovarid_df.loc[:,geo_level_lower] =
/workspaces/intersect-community-data/pyncoda/CommunitySourceData/api_census_gov/acg_02a_add_categorical_char.py:216: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '['167' '167' '167' ... '167' '167' '167']' has dtype incompatible with int64, please explicitly cast to a compatible dtype first.
add_geovarid_df.loc[:,geo_level_lower] =
/workspaces/intersect-community-data/pyncoda/CommunitySourceData/api_census_gov/acg_02a_add_categorical_char.py:216: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '['720100' '720100' '720100' ... '725800' '725800' '725800']' has dtype incompatible with int64, please explicitly cast to a compatible dtype first.
add_geovarid_df.loc[:,geo_level_lower] =
/workspaces/intersect-community-data/pyncoda/CommunitySourceData/api_census_gov/acg_02a_add_categorical_char.py:781: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise an error in a future version of pandas. Value '0.5' has dtype incompatible with int64, please explicitly cast to a compatible dtype first.
set_flag_df.loc[conditions, self.flaggeo_var] = .5
/workspaces/intersect-community-data/pyncoda/CommunitySourceData/api_census_gov/acg_02a_add_categorical_char.py:781: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise an error in a future version of pandas. Value '0.5' has dtype incompatible with int64, please explicitly cast to a compatible dtype first.
set_flag_df.loc[conditions, self.flaggeo_var] = .5
/workspaces/intersect-community-data/pyncoda/CommunitySourceData/api_census_gov/acg_02a_add_categorical_char.py:804: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise an error in a future version of pandas. Value '7.5' has dtype incompatible with int64, please explicitly cast to a compatible dtype first.
set_flag_df.loc[conditions, self.flaggeo_var] = self.round+.5

Research Idea

We could use ipums to see what the general likelihood is that households have 0 to x number of vulnerabilities
Also could perform PCF to see how vulnerabilities relate to each other and which ones are highlighly correlated, the way the different categories weight could help guide the synthetic population variables to include

Modify housing unit inventory income model

Modify housing unit inventory income model to include race, ethnicity, age of householder, and tenure

some of the code in population inventory private github

output data folders

code seems to be creating multiple outputdata folders

why are there 2 data-axel folders?

see if the 2 folders can be combined

found bug - expected total seems to decrease

Issue moved over from: https://github.com/npr99/Labor_Market_Allocation/issues/1

The expected total should be the same across loops of MCMC SA - noticed with block 371559610003026 (
MCMC SA settings:
random_accept_threshold = 0.1
start_reduction_threshold = 0.1
max_reduction_threshold = 0.4
seed = 133234)
That the expected total decreases from 1205 to 1184 to 1159 to 1147....
This should not happen. It is possible that the random select function has a bug.

Labor_Market_Allocation/pyincoredata_addons/SourceData/lehd_ces_census_gov/lodes_mcmcsa_loops.py

Lines 84 to 99 in 731d71a

Should probably make length_expected using the wac job list not the random selection. But the random selection should always selected the same number - so the error is probably in the random selection process.

Labor_Market_Allocation/pyincoredata_addons/SourceData/lehd_ces_census_gov/lodes_mcmcsa_util.py

Lines 350 to 358 in 731d71a

if k > 2:
seed_k = seedk + k
df = rand_select_jobs(df, seed_k)
# Check if number of jobs selected for each od pair
# matches the expected number of jobs

 # New function here 

 total_mcmcsa[k,1] = calculate_total_fitness(\

Note line 356 in lodes_mcmcsa_util.py references same issue

Try using unique job id
It might be that jobidac is not merging correctly and therefore dropping obs

workflow for prechui

getting ready to publish example files for the person record files, with student and staff examples.
prechui_v0-2-0_37155_2010_rs9876.csv
prec_v0-2-0_Lumberton_NC_2010_rs9876_schoolstaff.csv
prec_v0-2-0_Lumberton_NC_2010_rs9876_students.csv

I need to work on getting this workflow up-to-date on github. Right now it is not clear who these files could be replicated.
The code was from 2022-03-02

Add HUA Application Section to Annotated Bibliography

Google scholar search Housing Unit Allocation
Add general articles that use HUA to the annotated bibliography

https://scholar.google.com/scholar?hl=en&as_sdt=0%2C23&as_ylo=2019&inst=14379318592444324147&q=%22housing+unit+allocation%22+Rosenheim&btnG=

For each citation provide a short description of how the article uses HUA data

Run code in codespace

GitHub has a new option to run code in a virtual environment called codespace. This sounds like a good idea but - when I initially launch the codespace the environment is not setup. I had expected the environment to be based on the .yml file. But it looks like I need to make a .dev file.

Following these instructions:
https://docs.github.com/en/codespaces/setting-up-your-project-for-codespaces/adding-a-dev-container-configuration/setting-up-your-python-project-for-codespaces

future warnings to resolve

Here's a summary of the issues along with their contexts:

Deprecated Downcasting on .fillna, .ffill, .bfill:

Files/Lines:
acg_01a_BaseInventory.py: lines 741, 803, 856
acg_02a_add_categorical_char.py: lines 470 (multiple)
Details: Using .fillna, .ffill, .bfill for downcasting object dtype arrays is deprecated. Suggested to use result.infer_objects(copy=False) or set the option pd.set_option('future.no_silent_downcasting', True).
Callable in GroupBy Sum:

Files/Lines:
acg_01a_BaseInventory.py: lines 1014 (multiple)
Details: The callable currently uses DataFrameGroupBy.sum. Future versions will use the callable directly. To retain current behavior, pass the string "sum" instead.
Setting Incompatible Dtype:

Files/Lines:
acg_02a_add_categorical_char.py: line 760 (multiple)
ncoda_00e_geoutilities.py: line 294 (multiple)
Details: Setting an item with an incompatible dtype is deprecated. For example, setting 0.5 in an int64 column or setting 2637860 and 'Hesperia' in a float64 column. Explicit casting to compatible dtypes is required.
Invalid Value Encountered in Computation:

Files/Lines:
expressions.py: line 73
Details: Runtime warnings due to invalid values encountered during less_equal and greater operations.
Deprecated DataFrame.applymap:

Files/Lines:
ncoda_07a_generate_hui.py: line 170
Details: DataFrame.applymap has been deprecated. Use DataFrame.map instead.
In summary, these issues are primarily related to deprecated pandas methods and operations, which need updating to avoid future compatibility problems.

Validate poverty results

U.S. Poverty Rates Differ by Age and County

https://www.census.gov/library/stories/2022/12/poverty-rates-by-age-county-region.html?utm_campaign=20221208msacos2ccstonl&utm_medium=email&utm_source=govdelivery

Can ncoda output replicate poverty by age?

Idea for visualizing data flow

https://github.com/transitland/transitland-processing-animation

Github repo has python code for animating points on a map over time

Run HUA with non incore building inventory

working on workflow to run the Housing Unit Inventory on a non incore dataset
For example - in southeast texas we have National Address Data Inventory and Parcel data. What do I need to do to make the hua work with this type of file

Get "Used By

requirements.txt

https://github.com/Kanaries/pygwalker/tree/main

https://docs.github.com/en/code-security/supply-chain-security/understanding-your-software-supply-chain/about-the-dependency-graph#supported-package-ecosystems

I noticed in pygwalker github page that they have a used by section. Also the package has a nice readme page.

Add education to HUI

Add job record file and establishment intersection

code in population inventory private github

Add vehicle ownership to HUI

look up ACS tables

Add disability type to person record file

directory design - use community name

intersect-community-data/pyncoda/ncoda_00b_directory_design.py

Line 11 in f7503b7

def directory_design(state_county_name,

current code uses the county name as the primary folder for the directory structure
consider using the community name instead and put all files inside the same folder for the county loop

rework joblist to prec merge

once @aminenderami has his joblist that combines the wac-od-rac files. Here is a possible workflow for how to combine the joblist with the person record files.

@npr99 did this for Lumberton School Staff.

Example data published on DesignSafe
https://www.designsafe-ci.org/data/browser/public/designsafe.storage.published//PRJ-2961v3/PersonRecordFiles_Lumberton_2022-03-02?doi=10.17603%2Fds2-jwf6-s535

prec_v0-2-0_Lumberton_NC_2010_rs9876_schoolstaff.pdf

adding code to NPR sandbox

found code in
G:\Shared drives\HRRC_IN-CORE\Tasks\M5.2-01 Pop inventory\github_com\npr99\Population_Inventory\pyincore_data_addons

Archived code in this folder:
pyncoda\99_SandboxCode\SandboxNPR\prechui_lodes_code

The file
ncoda_07fv1_HUA_PREC_NSI.ipynb

generates the HUI and PREC files (note need to add code that combines the HUI and PREC files. This code is NPR Sandbox but has not been updated.

Issue with family variable in Oceana County

running code for oceana county
found that there is an issue with the family variable having missing values.
The code did not run the first time - error at the point of making the figure for family income.
The code ran fine the second time through.

Also an error with building available table. Might be an issue with the data not having anyone in missing buildings.

Add Housing Unit Allocation

add code (and update code) to run housing unit allocation in icd

apply map for removing .0

/workspaces/intersect-community-data/pyncoda/ncoda_07a_generate_hui.py:170: FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.
hui_incore_df_fixed = hui_incore_df.applymap(lambda cell: int(cell) if str(cell).endswith('.0') else cell)

pop_results_tables

/workspaces/intersect-community-data/pyncoda/ncoda_04c_poptableresults.py:400: FutureWarning: The provided callable <function sum at 0x764c0c1a0ee0> is currently using DataFrameGroupBy.sum. In a future version of pandas, the provided callable will be used directly. To keep current behavior pass the string "sum" instead.
for col in table.columns:
/workspaces/intersect-community-data/pyncoda/ncoda_04c_poptableresults.py:400: FutureWarning: The provided callable <function sum at 0x764c0c1a0ee0> is currently using DataFrameGroupBy.sum. In a future version of pandas, the provided callable will be used directly. To keep current behavior pass the string "sum" instead.
for col in table.columns:
/workspaces/intersect-community-data/pyncoda/ncoda_04c_poptableresults.py:400: FutureWarning: The provided callable <function sum at 0x764c0c1a0ee0> is currently using DataFrameGroupBy.sum. In a future version of pandas, the provided callable will be used directly. To keep current behavior pass the string "sum" instead.
for col in table.columns:

building inventory generation

Create OpenStreetMap plus Microsoft Building footprint building inventory generation

Add Address Point Estimation

code in
G:\Shared drives\HRRC_IN-CORE\Text\Rahimi et al 2022 - Galveston Recovery Models

dtype for point_gdf

/workspaces/intersect-community-data/pyncoda/ncoda_00e_geoutilities.py:294: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise an error in a future version of pandas. Value '4110850' has dtype incompatible with float64, please explicitly cast to a compatible dtype first.
copy_point_gdf.loc[precise_matches.index,geolevel+col] = polygon[col]
/workspaces/intersect-community-data/pyncoda/ncoda_00e_geoutilities.py:294: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise an error in a future version of pandas. Value 'Cannon Beach' has dtype incompatible with float64, please explicitly cast to a compatible dtype first.
copy_point_gdf.loc[precise_matches.index,geolevel+col] = polygon[col]

Add student record file and school allocation

code in population inventory private github

project overview not found

File /workspaces/intersect-community-data/pyncoda/ncoda_06c_Codebook.py:100, in codebook.add_projectoverview(self, pdf)

FileNotFoundError: [Errno 2] No such file or directory: 'pyncoda\ncoda_00a_projectoverview.md'

error that this file is missing - but it is in the pyncoda folder
pyncoda/ncoda_00a_projectoverview.md

Add to Annotated Bibliography

Amini, M., Jeon, H., Sanderson, D. R., Cox, D. T., Barbosa, A. R., & Cutler, H. (2023). Integrated Engineering–Economic Analysis for Multihazard Damage and Loss Assessment. Journal of Infrastructure Systems, 29(4), 04023031.

https://ascelibrary.org/doi/pdf/10.1061/JITSE4.ISENG-2229

references HUA but may not uses CGE data. Think about including it.