urbansim/urbansim_parcels

New version of urbansim_defaults for compatibility with new developer model

urbansim_parcels's Issues

Prices for feasibility analysis should be restricted to new buildings

The parcels DataFrame that is input into the SqFtProForma object requires price columns - one for each use that is analyzed. This is totally flexible; urbansim_parcels has a simple placeholder calculation, and the San Diego model uses something a little more sophisticated.

Keeping in mind that it's very region specific, we should still change the default behavior to only look at new buildings. This is because prices for new buildings are the only reasonable comparator to potential new developments.

Filter input into pro forma model by market research heuristics

This is simply extending the work done so far to allow user-defined functions to modify profit calculations and development selection.

We should also provide the option to filter inputs into the feasibility model based on other characteristics, like occupancy. This might actually already be provided by the "filter_parcels" option in the pro forma model, in which case we just need to implement it with occupancy.

format of feasibility table different on different proposals_to_keep

The proposals_to_keep parameter is set in the proforma.yaml configs file. It can take the value 1 or more than 1.
This parameter affects how the feasibility table is built. The lookup_by_form function builds the feasibility table using the forms passed to the function (e.g. "residential") and the allowed function to check the feasibility of each form for each parcel.
Then, before returning the feasibility table, the lookup_results dictionary is used to create the feasibility data frame, but there are two ways of creating it, one path is when the proposals_to_keep is > 1 and the other path is when proposals_to_keep is 1. These are the lines.
When the proposals_to_keep is more than 1, the feasibility table gets all the correct columns, plus the "form" column filled with the corresponding form ('residential', 'office', etc). A one-level set of columns.
The issue comes when the proposals_to_keep is equal to 1, the data frame that is created get multiindex columns with the (form/column). This creates an issue, later on when running the developer and calling the feasibility table and calling a specific column, in this case "max_profit_far", because the column is there but in level 1. So the correct way to call that column would be ('residential', 'max_profit_far').
Solution:
I think that the feasibility table, with one or more proposals to keep should have the same format to be used in the latter functions such as the developers. Therefore I suggest changing these lines, to have a one-level index of columns and a column called "form".

Full transition model looses the index name and saves orca table with more than local columns

I would like to improve two things in the utils.full_transition model.

1. It happens that the new dataframe that results from this operation:

new, added_hh_idx, new_linked = model.transition(
        hh, year, linked_tables=linked_tables)

which is then saved as the new orca table, looses the index name, for example 'household_id'
So the solution I propose is to add this line:
new.index.name = hh.index.name
right after this line, to make sure the index name gets preserved.

2. The agents table is called with the local columns and the settings.get('add_columns', []):
   hh = agents.to_frame(agents.local_columns + settings.get('add_columns', [])).
   But then, when the new table is saved as an orca table, orca.add_table(agents.name, new), all the columns specified in the to_frame() are passed as local_columns which is a change that the model should not do. So I propose that when registering the new data frame as an orca table, we pass the local_columns as well, like this:
   orca.add_table(agents.name, new[agents.local_columns])

How to set up properly?

Hi,

We are experimenting with this repo, and we got rather confused on how to set it up. Hear is what we tried:

• git clone developer
• python setup.py install in the new developer folder
• git clone urbansim_parcels
• run jupyter notebook
• and try to run Simulation

We get no module name models. We tried to install the urbansim_parcels with python setup.py install witch did not help.The new urbansim_parcels module had nothing useful.

How do we get started with thees examples?

Add linked tables option to Simple Transition Model

Right now, the utils.simple_transition model does not allow for linked tables to be changed as the transition model changes the main table. We use this specifically when the households are transitioning, the persons table should change as well. Though we have that option in the full_transition model in a parameter called linked_tables, we don't have the same possibility when running a simple transition model with a growth rate.

Add ProForma object callback, and add **kwargs to other callbacks

Just a simple callback in this line, for easier monkey patching, something like:

pf = update_sqftproforma_callback(pf)

Also, add **kwargs to callbacks, like here and [here]

(https://github.com/urbansim/urbansim_parcels/blob/master/urbansim_parcels/utils.py#L732).

(Suggestion by @hanase)

Provide information about bytes types in HDF5 DataFrames in Python 3

Some of the pandas DataFrames within HDF5 files with urbansim-related data (tables of parcels, buildings, etc) have index names (i.e. df.index.name) that are saved as binary data (bytes type) rather than strings. Since bytes and str is handled the same way in Python 2.7, it wasn't a problem. In Python 3, they're treated separately.

We've decided not to add code to orca table registration functions that convert these names. Instead, folks using Python 3 with data that has these quirks should just update their datasets. When we release the version of UrbanSim that has python 3 compatibility, we should include a note about this in the release notes.

We can include a code snippet to do this conversion in HDF5 files, something like:

import numpy
import os
import pandas as pd

new_store = pd.HDFStore('path/to/newstore.h5', mode='w')

with pd.HDFStore('path/to/oldstore.h5') as store:
    for tablename in store.keys():
        table = store[tablename]
        if table.index.name and type(table.index.name) == numpy.bytes_:
            table.index.name = table.index.name.decode()
        new_store.put(tablename, table, format='t')
new_store.close()

Add development limits functionality into utils.py() a-la bayarea_urbansim

MTC uses jurisdiction-specific, form-specific development limits in its developer models, defined in a master config .yaml and referenced in their development model steps (see here for an example). In the spirit of keeping models.py nice and clean, and moving model specifications to config .yaml's and utils.py, it might be a good idea to provide this functionality for MTC and other MPO's out of the box by way of augmenting utils.run_developer() and maybe adding a dev_settings .yaml config.

Needs a general audit

The current state of this repo is a messy combination of urbansim_defaults and sanfran_urbansim. We need to go through variables.py, models.py, utils.py, datasources.py, and settings.yaml pretty extensively and make sure everything:

• Fits together correctly for the SF example (which it does now)
• Is ready to use for more complex models (e.g. with accessibility variables)

It might be helpful to add more examples to the repo, perhaps borrowed from bayarea_urbansim.

Incorporate recent changes into the master branch

I believe the ej-test branch is being used in some operational models. Let's open a PR to merge those changes into master, with some documentation in the PR writeup of what's been modified.

• PR of ej-test -> master with documentation in the writeup

Let's also begin including dev version numbers that increment with each PR -- this helps us track which version of the code is being used in a model.

• version = 0.1.dev1 in setup.py
• add __version__ = '0.1.dev1' to urbansim_parcels/__init__.py

And we can add a changelog while we're at it!

• create a CHANGELOG.md

See choicemodels or another recently updated UDST library as an example.

Can i assign this to @alephcero and/or @PyMap, with help if needed from @janowicz for figuring out what should be included in the PR and what the changes represent?

Aggregated parcels (as opposed to split) not supported in current data structures

The current pipeline and sites tables don't support the ability for one site to be on multiple parcels. Given that aggregating parcels into sites is something we want to support in the future, we need to consider potential changes to these core structures.

Refine custom selection criteria in SF/SD models

The placeholder functions that provide custom criteria for development selection (example here) should be refined. Some ideas:

• Add in a filter of an absolute minimum profit (in addition to a minimum profit per sqft)
• Add in a threshold profit to cost ratio

Network-based queries (rather than zone-based) for real estate models

Currently, our the SD and SF models calculate prices and occupancies using the already-existing "zone" geography (these are traffic analysis zones as I understand it). So, for a given parcel, its rent (in $/sqft) is equivalent to that use's average price for the zone that it is in.

We can do this in a more sophisticated way, by using Pandana to aggregate prices and occupancies (per use) to nodes on the network. For example, get the average price for residential uses within 500m network distance from this node. Then we can assign node prices to parcels, and same with occupancies.

Replace SF example with San Diego example

Similar to SEMCOG's urbansim, we should set up a copy of San Diego's urbansim implementation with public data that works with our new developer model. This will be increasingly important as we test out new features.

Add additional example(s) with accessibility variables

The sf_example is helpful but limited because it includes no network data / accessibility variables. We should figure out a more fully featured example model; perhaps downloading a network from OSMnet can be an early step.

Incorporate orca_test

We should incorporate orca_test into the San Francisco and San Diego simulation runs, both as an actual test but also to provide an applied example for orca_test.

Add a license

Let's add a BSD 3-clause open source license to the repo.

Framework for market research model

As a modeler, I want to incorporate basic market research into the real estate steps.

This story is aimed at building a simple framework for market research in the parcel model. This will depend on actual model classes/components in the developer repo, which will be developed simultaneously. The parts in urbansim_parcels that will be update are:

• Add data collection step before feasibility step: collect information about vacancy, prices for all buildings, tag as year
• Add major model step for market research, including:
  • Calculate absorption for buildings built in 2019
  • Calculate absorption trends between 2016 and 2019
  • Calculate "hot" neighborhood trends based on development activity between 2017 and 2019
  • Calculate price trends between 2017 and 2020
  • Determine which buildings are candidates for feasibility analysis (send to pro forma)
• Filter input into pro forma model
• Add columns to inputs for pro forma model based on market research
• Revise developer model functionality to allow for "market based" selection

This first pass should build each of these in a simple, "placeholder" manner that works on a data roundtrip.

MTC model

It could make sense to throw in a modified version of the latest Bay Area urbansim, in order to test things like rent vs. own and multifamily vs. single family in the real estate workflow.