npr99/intersect-community-data

Python functions to obtain and clean data required for the version 2 Housing Unit Allocation. Workflow uses Census API.

intersect-community-data's Issues

Add vehicle ownership to HUI

look up ACS tables

Add Address Point Estimation

code in
G:\Shared drives\HRRC_IN-CORE\Text\Rahimi et al 2022 - Galveston Recovery Models

Add job record file and establishment intersection

code in population inventory private github

Add HUA Application Section to Annotated Bibliography

Google scholar search Housing Unit Allocation
Add general articles that use HUA to the annotated bibliography

https://scholar.google.com/scholar?hl=en&as_sdt=0%2C23&as_ylo=2019&inst=14379318592444324147&q=%22housing+unit+allocation%22+Rosenheim&btnG=

For each citation provide a short description of how the article uses HUA data

found bug - expected total seems to decrease

Issue moved over from: https://github.com/npr99/Labor_Market_Allocation/issues/1

The expected total should be the same across loops of MCMC SA - noticed with block 371559610003026 (
MCMC SA settings:
random_accept_threshold = 0.1
start_reduction_threshold = 0.1
max_reduction_threshold = 0.4
seed = 133234)
That the expected total decreases from 1205 to 1184 to 1159 to 1147....
This should not happen. It is possible that the random select function has a bug.

Labor_Market_Allocation/pyincoredata_addons/SourceData/lehd_ces_census_gov/lodes_mcmcsa_loops.py

Lines 84 to 99 in 731d71a

Should probably make length_expected using the wac job list not the random selection. But the random selection should always selected the same number - so the error is probably in the random selection process.

Labor_Market_Allocation/pyincoredata_addons/SourceData/lehd_ces_census_gov/lodes_mcmcsa_util.py

Lines 350 to 358 in 731d71a

if k > 2:
seed_k = seedk + k
df = rand_select_jobs(df, seed_k)
# Check if number of jobs selected for each od pair
# matches the expected number of jobs

 # New function here 

 total_mcmcsa[k,1] = calculate_total_fitness(\

Note line 356 in lodes_mcmcsa_util.py references same issue

Try using unique job id
It might be that jobidac is not merging correctly and therefore dropping obs

Validate poverty results

U.S. Poverty Rates Differ by Age and County

https://www.census.gov/library/stories/2022/12/poverty-rates-by-age-county-region.html?utm_campaign=20221208msacos2ccstonl&utm_medium=email&utm_source=govdelivery

Can ncoda output replicate poverty by age?

Modify housing unit inventory income model

Modify housing unit inventory income model to include race, ethnicity, age of householder, and tenure

some of the code in population inventory private github

rework joblist to prec merge

once @aminenderami has his joblist that combines the wac-od-rac files. Here is a possible workflow for how to combine the joblist with the person record files.

@npr99 did this for Lumberton School Staff.

Example data published on DesignSafe
https://www.designsafe-ci.org/data/browser/public/designsafe.storage.published//PRJ-2961v3/PersonRecordFiles_Lumberton_2022-03-02?doi=10.17603%2Fds2-jwf6-s535

prec_v0-2-0_Lumberton_NC_2010_rs9876_schoolstaff.pdf

adding code to NPR sandbox

found code in
G:\Shared drives\HRRC_IN-CORE\Tasks\M5.2-01 Pop inventory\github_com\npr99\Population_Inventory\pyincore_data_addons

Archived code in this folder:
pyncoda\99_SandboxCode\SandboxNPR\prechui_lodes_code

The file
ncoda_07fv1_HUA_PREC_NSI.ipynb

generates the HUI and PREC files (note need to add code that combines the HUI and PREC files. This code is NPR Sandbox but has not been updated.

Research Idea

We could use ipums to see what the general likelihood is that households have 0 to x number of vulnerabilities
Also could perform PCF to see how vulnerabilities relate to each other and which ones are highlighly correlated, the way the different categories weight could help guide the synthetic population variables to include

Get "Used By

requirements.txt

https://github.com/Kanaries/pygwalker/tree/main

https://docs.github.com/en/code-security/supply-chain-security/understanding-your-software-supply-chain/about-the-dependency-graph#supported-package-ecosystems

I noticed in pygwalker github page that they have a used by section. Also the package has a nice readme page.

It looks like a package that uses another package needs to have a requirments.txt file - then github can identify the package that depends on another package and will make a used by section on the repository.

Work on entity relationship figure

I have started a entity relationship figure in LUCID chart here is the link:
https://lucid.app/lucidchart/aaa0a9d7-087e-4a23-8e75-67feb6efb684/edit?viewport_loc=19%2C179%2C3762%2C1893%2C0_0&invitationId=inv_9dc49768-f696-4dcd-990f-a92a5d52a56b

explore sex by age - looks like missing values and clusters

Notice that histogram shows clustering of ages around every 10 years. Also looks like there is a gap between 80 and 85

Run HUA with non incore building inventory

working on workflow to run the Housing Unit Inventory on a non incore dataset
For example - in southeast texas we have National Address Data Inventory and Parcel data. What do I need to do to make the hua work with this type of file

Integrate Amin's LODES data code

https://github.com/npr99/intersect-community-data/tree/main/pyncoda/99_SandboxCode/SandboxAE

On my VS Code running code for
Employment Inventory.ipynb

copied code from Amin's sandbox into my work folder on VS Code. The gitnore allows work folders to be ingored

building inventory generation

Create OpenStreetMap plus Microsoft Building footprint building inventory generation

Get "Used By" section on repo

requirements.txt

https://github.com/Kanaries/pygwalker/tree/main

https://docs.github.com/en/code-security/supply-chain-security/understanding-your-software-supply-chain/about-the-dependency-graph#supported-package-ecosystems

I noticed in pygwalker github page that they have a used by section. Also the package has a nice readme page.

It looks like a package that uses another package needs to have a requirments.txt file - then github can identify the package that depends on another package and will make a used by section on the repository.

Issue with family variable in Oceana County

running code for oceana county
found that there is an issue with the family variable having missing values.
The code did not run the first time - error at the point of making the figure for family income.
The code ran fine the second time through.

Also an error with building available table. Might be an issue with the data not having anyone in missing buildings.

Add student record file and school allocation

code in population inventory private github

Consider adding Homeless

https://data.census.gov/table?q=foster+child&g=0100000US_0500000US37155_1600000US3739700&tid=ACSST5Y2019.S0901

https://data.census.gov/cedsci/table?q=foster%20child&g=0100000US_0500000US37155_1600000US3739700&tid=ACSST5Y2019.S0901

Homeless students
share housing with other persons due to loss of housing or economic hardship, live in hotels or motels, trailer parks, or campgrounds due to lack of alternative arrangements, those awaiting foster care placement, living in substandard housing, and children of migrant workers (CRS, 2018)

good list of characteristics - some of these are in the housing unit and person record inventory. Shared housing, trailer parks - could be identified in the data. Foster care might be something to summarize - the census bureau has counts of people in foster care.

I will see if I can flag shared housing. The person record inventory has household structure - could be based on number of adults to children. I would assume that shared housing would have multiple adults (3-4+) and multiple children (3+)

According to the National School Boards Association, during the 2018-2019 school year, 77% of homeless students lived in shared housing, 12% lived in shelters, transitional housing, or were awaiting foster care, 7% lived in hotels or motels, and 4% were unsheltered (Cai, 2021).

Cai, Jinghong. (2021). Homeless Students in Public Schools Across America: Down but Not Out. National School Boards Association.
Congressional Research Service (CRS). (2018). Homelessness: Targeted Federal Programs. CRS Report.

change main folder pyhui

pyhui stands for python housing unit inventory... need to change the name but have not decided on the future package name

options
pyicd - python intersect community data
pyincoda
pyintcomdat
pyintersectcd
pyntrsctcmntydt

pyixcommunitydata

Idea for visualizing data flow

https://github.com/transitland/transitland-processing-animation

Github repo has python code for animating points on a map over time

Add education to HUI

build conda package

Started slack channel on IN-CORE
https://in-core.slack.com/archives/C03MLKDV4ER

Add person record file generation

code in population inventory private github

Add Housing Unit Allocation

add code (and update code) to run housing unit allocation in icd

workflow for prechui

getting ready to publish example files for the person record files, with student and staff examples.
prechui_v0-2-0_37155_2010_rs9876.csv
prec_v0-2-0_Lumberton_NC_2010_rs9876_schoolstaff.csv
prec_v0-2-0_Lumberton_NC_2010_rs9876_students.csv

I need to work on getting this workflow up-to-date on github. Right now it is not clear who these files could be replicated.
The code was from 2022-03-02

Run code in codespace

GitHub has a new option to run code in a virtual environment called codespace. This sounds like a good idea but - when I initially launch the codespace the environment is not setup. I had expected the environment to be based on the .yml file. But it looks like I need to make a .dev file.

Following these instructions:
https://docs.github.com/en/codespaces/setting-up-your-project-for-codespaces/adding-a-dev-container-configuration/setting-up-your-python-project-for-codespaces

Add to Annotated Bibliography

Amini, M., Jeon, H., Sanderson, D. R., Cox, D. T., Barbosa, A. R., & Cutler, H. (2023). Integrated Engineering–Economic Analysis for Multihazard Damage and Loss Assessment. Journal of Infrastructure Systems, 29(4), 04023031.

https://ascelibrary.org/doi/pdf/10.1061/JITSE4.ISENG-2229

references HUA but may not uses CGE data. Think about including it.

Add disability type to person record file

Update LODES job record generation and run for multiple counties

directory design - use community name

intersect-community-data/pyncoda/ncoda_00b_directory_design.py

Line 11 in f7503b7

def directory_design(state_county_name,

current code uses the county name as the primary folder for the directory structure
consider using the community name instead and put all files inside the same folder for the county loop

why are there 2 data-axel folders?

see if the 2 folders can be combined