aequilibrae/tradesman

A package for building AequilibraE transport models from open data sources

AequilibraE

AequilibraE is a fully-featured Open-Source transportation modeling package and the first comprehensive package of its kind for the Python ecosystem, and is released under an extremely permissive and business-friendly license.

It is developed as general-purpose modeling software and imposes very little underlying structure on models built upon it. This flexibility also extends to the ability of using all its core algorithms without an actual AequilibraE model by simply building very simple memory objects from Pandas DataFrames, and NumPY arrays, making it the perfect candidate for use-cases where transport is one component of a bigger and more general planning or otherwise analytical modeling pipeline.

Different than in traditional packages, AequilibraE's network is stored in SQLite/Spatialite, a widely supported open format, and its editing capabilities are built into its data layer through a series of spatial database triggers, which allows network editing to be done on Any GIS package supporting SpatiaLite, through a dedicated Python API or directly from an SQL console while maintaining full geographical consistency between links and nodes, as well as data integrity and consistency with other model tables.

AequilibraE provides full support for OMX matrices, which can be used as input for any AequilibraE procedure, and makes its outputs, particularly skim matrices readily available to other modeling activities.

AequilibraE includes multi-class user-equilibrium assignment with full support for class-specific networks, value-of-time and generalized cost functions, and includes a range of equilibration algorithms, including MSA, the traditional Frank-Wolfe as well as the state-of-the-art Bi-conjugate Frank-Wolfe.

AequilibraE's support for public transport includes a GTFS importer that can map-match routes into the model network and an optimized version of the traditional "Optimal-Strategies" transit assignment, and full support in the data model for other schedule-based assignments to be implemented in the future.

State-of-the-art computational performance and full multi-threading can be expected from all key algorithms in AequilibraE, from cache-optimized IPF, to path-computation based on sophisticated data structures and cascading network loading, which all ensure that AequilibraE performs at par with the best commercial packages current available on the market.

AequilibraE has also a Graphical Interface for the popular GIS package QGIS, which gives access to most AequilibraE procedures and includes a wide range of visualization tools, such as flow maps, desire and delaunay lines, scenario comparison, matrix visualization, etc. This GUI, called QAequilibraE, is currently available in English, French and Portuguese and more languages are continuously being added, which is another substantial point of difference from commercial packages.

Finally, AequilibraE is developed 100% in the open and incorporates software-development best practices for testing and documentation. AequilibraE's testing includes all major operating systems (Windows, Linux and MacOS) and all currently supported versions of Python. AequilibraE is also supported on ARM-based cloud computation nodes, making cloud deployments substantially less expensive.

Comprehensive documentation

AequilibraE documentation built with Sphinx

What is available only in QGIS

Some common resources for transportation modeling are inherently visual, and therefore they make more sense if available within a GIS platform. For that reason, many resources are available only from AequilibraE's QGIS plugin, which uses AequilibraE as its computational workhorse and also provides GUIs for most of AequilibraE's tools. Said tool is developed independently and a little delayed with relationship to the Python package, and more details can be found in its GitHub repository.

tradesman's Issues

Replace pycountry usage for project about

Since we have added country_name and country_code to the project database about, it is possible to remove some pycountry imports.

Some files to change are:

• population_file_address.py
• pop_by_sex_and_age.py
• microsoft_building_footprint.py
• delete_links_and_nodes.py

Rename function parameters to guarantee logic

Some function parameters such as model_place are receiving country_name or country_code when they shouldn't. Hence these parameters must be renamed to guarantee logic.

Add package to Anaconda

Need to create recipe and figure out how to add the package to Anaconda

Improve Bing imports

As for March 2023, our current pipeline consists in loading Microsoft Bing URLs and checking if the model place is within a country or is a country with any available data. If it has, then we download all country data. In case the model place is a country, we're good, but if it's not, then we're downloading several unused files. Thus the improvement here would be in only downloading the necessary data (and making the URL loading faster).

build_zoning fails when we do not save hex_bins to the database

File /workspaces/tradesman/notebooks/../tradesman/model.py:128, in Tradesman.build_zoning(self, hexbin_size, max_zone_pop, min_zone_pop, save_hexbins, overwrite)
126 if sum(self._project.conn.execute("Select count(*) from Zones").fetchone()) > 0:
127 return
--> 128 zone_builder(self._project, hexbin_size, max_zone_pop, min_zone_pop, save_hexbins)

File /workspaces/tradesman/notebooks/../tradesman/model_creation/zoning/zone_building.py:44, in zone_builder(project, hexbin_size, max_zone_pop, min_zone_pop, save_hexbins)
42 zones_with_pop = zones_with_population(project, zones_with_locations)
43 print("\n after zones_with_pop", strftime("%Y-%m-%d %H:%M:%S", gmtime()))
---> 44 project.conn.execute("DELETE FROM hex_pop;")
45 project.conn.execute("DELETE FROM zones;")
46 project.conn.commit()

OperationalError: no such table: hex_pop

Create pipeline/structure for producing models

In the near future, we will have to update models continuously. We need a workflow for doing that.

Initially, it would include:

• A requirements.txt file with everything needed to produce models (Tradesman and the #1 requirement)
• A list of countries/places to build models for
• A set of scripts to make this happen

Documentation for PopulationSim

There are too many features that are hard to use without examples. We could use a set of Jupyter Notebooks showing the features

1. Notebook with just a straightforward application of the tool
2. Notebook where the user can use "all" the features

Deleting of features outside country borders is problematic

There are two major problems

• Takes a virtually infinite amount of time to run
• Deletes links that are just outside the border, including bridges across bays and other coastal infrastructure

Update documentation

It must reflect reflect the package itself and not the road_analytics pipeline.

Improve population raster imports

There are two different tasks here:

• Allow the user to use his/hers own population raster files, so if the user has updated files from one country he/she can use it instead of using data from our sources.
• If the user wants to use one of the existing World Pop or Meta files, allow the download of multiple files at the same time. For bigger countries whose population raster files are quite large, it could be really time-consuming waiting for each one of the files to be downloaded.

Make Tradesman capable of building models for regions smaller than a country

It would be ideal to be able to build models automatically for Cities/states rather than entire countries

Creation of synthetic population given a population pyramid

There are four main items to be addressed here:

• Creation of a large set of microsample to feed into the population synthesizer for the US (PUMS/PUMA)
• Documentation of the micro-sample format (Filtered down to the minimum set of fields required) for users
• Research into the availability of micro-samples outside the USA
• Implementation of PopulationSim to our model structure
• Creation of the model tables to hold the synthetic population (two table specifications in SQL format alongside the others that are in the package already)
• Thorough documentation

Set save raw_population to False

Currently, we save the raw_population into the project_database. It is time-consuming (and memory-consuming) depending on the size of the model area, and it is only used to build zones.

It would be ideal to save raw_population according to user preference, and consider saving this information in the database not being mandatory.

Improve amenities and buildings import

We import building data from OSM and Microsoft Bing. Amenities are only downloaded from OSM.

Currently, we can save data in the database, but we cannot retrieve it when opening the model.

• Fix building import from Bing (URLs, download, and file manipulation)
• Be able to save OSM data in cache and retrieve it
• Write tests

Create gallery of Jupyter notebook examples

1. Example of creation of a model from scratch (possible to do for a tiny country?)
2. Example (or multiple examples) of different cool visualizations and analysis with a larger model (that is pre-built and stored on Github with releases)

The reference what this will look like is: https://aequilibrae.com/python/latest/_auto_examples/index.html

Rationalize and test import of population

The links for importing WorldPop are well formatted, so we should be able to use pycountry to get the country's code and format the way it is done below and get rid of the file all_raster_pop_age_and_sex_source.csv

https://data.worldpop.org/GIS/Population/Global_2000_2020/2020/ARM/arm_ppp_2020.tif
https://data.worldpop.org/GIS/AgeSex_structures/Global_2000_2020/2020/ARM/arm_m_80_2020.tif

The information for WorldPop that we see on all_raster_pop_source.csv can also go.

We should also investigate the possibility of doing the same for META data (does not seem easy or even possible)

• Remove all_raster_pop_age_and_sex_source.csv
• Create file path automatically for total population for WorldPop data
• Create file path automatically for the population pyramid for WorldPop data
• Create file path automatically for population data from META

Improve Tradesman logging

As for today, Tradesman only benefits partially from AequilibraE's logging tool. Especially when handling large areas, it would be useful for the user (and for debugging) to have a better logging tool to store warnings and project configuration/info.

Incorporate creation of a model from an OSM file instead

Workflow assumes one would download OSM file from GeoFabrik or equivalent and use OSM2GMNS to create a GMNS network that would be imported into the model instead of using AequilibraE's OSM importer

Set overwrite in population by sex and age

Unlike the overall population import, if we change the zoning system and try to rewrite the population by sex and age, Tradesman will return an error. The suggestion here is to allow the deletion and the file overwriting for the population pyramid.

Switch back to GADM subdivisions

Change from geoboundaries from GADM. Let's keep both options if possible

Switch boundary imports

We are currently getting political boundaries from GADM, and we should be getting them from GeoBoundaries in order to leverage their API

https://www.geoboundaries.org/api.html