The datausa-tracker from datausa

Dartmouth Data

Ingest Dartmouth Atlas of Health Data
Create Schema

Geographies name update

Todo:

Rename places that are duplicate within a state. Use a script.
use namelsad for counties, manually edited on server
use namelsad for counties, change in acs-core mondrian schema_shared.py

As part of the pluck "bug" in Mondrian rest, found out that there's lots of duplicates names that weren't handled well in data and in Mondrian.

Report for the weird Mondrian Rest bug:What's happening is a combination of an implicit Mondrian cube server behavior, combined with lack of error handling in Mondrian Rest.In the counties dim, there are counties in the same state which have the same short name, e.g. "Roanoke City" and "Roanoke County" become "Roanoke". Both have different ID (geoid). However, Mondrian doesn't like duplicate names, so it decides that there's only one dimension level member with name "Roanoke", so it simply takes the last last Roanoke it sees.This translates into an error in Mondrian Rest because the ID key for the first "Roanoke" still exists, even though the associated dimension level member has been silently squashed.When formatting the default json to something else, MRest uses a field called cell_keys, which holds all the ID keys. MRest then maps each ID key to a dimension level member, but because of Mondrian's behavior, that dim level member no longer exists for the duplicate name. Ruby happily assigns a NilClass instead of the expected mapping. Then in the next step, metadata is plucked out of each dim level member, and when it comes time to do this with the NilClass element, an exception occurs.Solution:

step one, need to make sure there's no duplicate names in dimensions, even if ID keys are distinct. (I'll have something up tomorrow morning, it's an easy fix)
step two, to help recognize this in the future, I'm going to catch that exception at the pluck and give a nice error message explaining that the cause might be the above.
cc @jspeiser @dave
(I've done step one locally, that's how I know it's the issue. But I need to coordinate a few things to make sure it's a permanent fix tomorrow morning)

county, place and puma names

IPEDS Admissions Data

Ingest
Schema

IPEDS University Student Financial Aid

Data
Schema

add annotation to hide exact MOE values by default in pums cubes

the exact values for PUMS MOEs can be time consuming to compute so by default these should be hidden unless explicitly requested

ingest university coordinates to geospatial database

Opiods data

ingest
schema
test

IPEDS Expenses Cube

data
schema

generate university similarity data

IPEDS Living SFA YUA

Ingest
Cube Schema

IPEDS Endowment Quintiles Cube

Setup geoservice-api

Stand-up basic geoservice-api for computing related geographies.

PUMS NAICS to BEA I/O Code Mapping

Need to provide a mapping which given a PUMS NAICS code will provide the closest matching IO Code

ACS

Normal tables

health insurance agg

B27002 - B27009 - csv done, needs upload and schema frag

race/ethnicity

B17001A - still running...
B19013A - still running...

Checkmark means schema and data in place; there may still be bugs though!

Update all to 2016 (have to add acs/ to fetch url)
for schema generator, add moe to measures

IPEDS Financials Cube

data
schema
test

SFA Aid by Income Level

Data
Schema

Create University dimension table

IPEDS Graduation Rates by Timeframe

BLS Growth Data

BLS Industry Growth
BLS Occupation Growth

Defaults Data

Convert O*NET dataset to cube architecture

Ingest ONET Data
Create Schema

add error_for_measure annotations to pums

PUMS SOC to BLS SOC crosswalk update

Several BLS SOC codes in the new ONET data are missing mappings to similar PUMS codes. Need to update the crosswalk.

Census-geo: incorporate county & place renames into app

for now we've included ad-hoc renames of namelsad to satisfy Mondrian's name uniqueness requirements for places but moving forward we'll likely want to do this at the shapefile ingest phase

County Health Rankings Data

Ingest
Schema
Build map for years/source

(Note from @hwchen... map is built for years/source, but check back in because I don't know final format)

New Health Data

Add RCA to PUMS Cubes

Population RCAs
Wage RCAs

IPEDS Instructional Salaries

Data
Schema

Federal Spending Data

Ingest
Schema
Update with Financial Assistance data

Tuition by CIP

We will need a separate cube for tuition by CIP

data
schema

BLS CES Data

Ingest
Schema

IPEDS Graduation Rates Cube

Create CIP dimension table

BEA I/O Data

Ingest use data
IO Code Hierarchy
Create schema

IPEDS Retention Rates Cube

data
schema
test

health insurance ACS tables for deloitte

Process B207010 using https://www.kff.org/other/state-indicator/total-population/?currentTimeframe=0&sortModel=%7B%22colId%22:%22Location%22,%22sort%22:%22asc%22%7D to do some manual roll-up

B25048
B25052- can we do as percentages?
B28002- listed as a number--can we convert to percentages?
B25092
C25074
B27010 if we can roll it up
B27004- any way we can convert to percentages?
B27007- can we convert to percentages?
B27006- can we convert to percentages?

[x] all etl done, just waiting on customizing schema

Economic Census Data

IPEDS Tuition Tables

Create Pipeline
Ingest
Schema

disable hideMemberIf parentName for industries/occupations

from the deeper levels disable hideMemberIf="IfParentsName"

Freight Data

Ingest
Schema

ACS language 2013-2014

etl to csv done, just need to upload and update table.

Census-geo; puma name updates

Add name column to pumas geospatial table that converts NYC-Manhattan Community District 8--Upper East Side PUMA to Upper East Side PUMA

update postgres pumas table
update scripts
update monetdb

IPEDS Non Instructional Salaries

Create Pipeline
Ingest
Schema

IPEDS Degrees Awarded Cube

data
schema
test

connect university coordinates to geoservice api

IPEDS Enrollment Data Cube

ingest
schema

create git repo for datausa-mondrian and update monetdb jdbc driver

https://dev.monetdb.org/downloads/Java/

IPEDS IC Living Expenses

acs null values

in a nutshell, as of 2016 there are now a few "magic" values that equal null: -666666666, -222222222, -999999999, -555555555, -333333333 ... but make we'll need to make sure to null those values out in the dataframe so that sums dont get messed up ... you might want to use na_values when loading the dataframe in pandas or df.replace

Unfortunately, looks like I missed this one. So all datasets will have to be rerun, or manually fixed in the db.

datausa / datausa-tracker Goto Github PK

datausa-tracker's People

Watchers

datausa-tracker's Issues

Normal tables

health insurance agg

race/ethnicity

Recommend Projects

Recommend Topics

Recommend Org