codethechange/mobility_pipeline

Mobility Data Pipeline for UNICEF

mobility_pipeline's Issues

Write Code to Validate Input Datafiles

We should have code that at minimum spot-checks the data files to give some assurance they are in the expected format. We should validate:

• Mobility data file
• Country shapefile
• Towers data file
• Voronoi file

Shapely type hinting

MyPy does not appear to support type hinting shapely, so for now we will tell mypy to ignore shapely with # type: ignore at the end of the import statement. Ideally, we will somehow add type hinting support for shapely by providing stub files. Related to #1.

Numpy type hinting

MyPy does not appear to support type hinting numpy, so for now we will tell mypy to ignore numpy with # type: ignore at the end of the import statement. Ideally, we will somehow add type hinting support for numpy by providing stub files. Here are some useful links:

• numpy/numpy#7370
• https://stackoverflow.com/questions/35673895/type-hinting-annotation-pep-484-for-numpy-ndarray/35688036#35688036
• https://github.com/machinalis/mypy-data/tree/master/numpy-mypy

Write matrix multiplication function

Write a function to make the admin-to-admin matrix from:

• tower-to-tower mobility matrix
• admin-to-tower overlap matrix
• tower-to-admin overlap matrix

The main purpose of this function is for clarity and documentation, not code reuse. Matrix multiplication can be done with numpy. Assume all input matrices are numpy matrices.

Optimize overall computation of admin-to-admin matrix

Introductory Documentation

Write introductory documentation for our project to introduce developers to our codebase. Write in restructured text.

Example:

• Rendered: https://culturemesh-android.readthedocs.io/en/latest/codeStructure.html
• Raw: https://raw.githubusercontent.com/culturemesh/CultureMeshAndroid/master/docs/source/codeStructure.rst

The documentation should include:

• What the code is capable of doing
• General organization and layout of the code
• Terms and technical paradigm
• Where to get the datasets
• How to use utility code (e.g. plotting)
• How to use the program

Toggle Layer Visibility Province-Specifically

Toggle the visibility of an existing layer only when a particular province is clicked.

Make Admin-to-Tower and Tower-to-Admin Matrices

See https://medium.com/@mikefabrikant/cell-towers-chiefdoms-and-anonymized-call-detail-records-a-guide-to-creating-a-mobility-matrix-d2d5c1bafb68

Find what Layer IDs Are

Add console.log() statements to the setVisibleLayers action to figure out what the IDs are. Specifically, we are looking for how the IDs relate to the contents of the JSON file.

Research how to optimize the computation of the admin-to-admin matrix from the tower-to-tower matrix

Potential ways:

• Vectorize with numpy
• Process tables with pandas

Load Country Shapefiles

Helpful Link: https://gis.stackexchange.com/questions/113799/how-to-read-a-shapefile-in-python

Shapefile: https://gadm.org/download_country_v3.html

Fix Filepaths and Import Paths

Loading data from filepaths and importing modules should work no matter how the script is run (i.e. from terminal or through PyCharm, from inside or outside the directory)

Unit Tests for All of `lib`

All of the lib module should be covered by unit tests. Check at codecov.

Toggle Layer Visibility on Click

Pick some layer that is already in the JSON and make it visible when any admin region is clicked. This might be possible by adding the layer ID to the list of currently visible layers (probably stored in state) and passing the new list to setVisibleLayers.

Compute Polygon Overlap

Write a function to compute the overlap between one polygon and a list of polygons

Parameters:

• enclosing: A MultiPolygon
• polygons: A list of MultiPolygon objects, each of which may overlap with enclosing

Return: A list of percentages such that the percentage at index i is the percent area of polygons[i] that is within enclosing.

Allow Saving Tower-to-Admin and Admin-to-Tower Matrices

It will be common for a user to want to compute admin-to-admin matrices for many days, but the current code requires that they re-compute the admin-to-tower and tower-to-admin matrices each time. This is the most inefficient part, and it only needs to happen once per country. The program should save off those files and let the user specify (or auto-detect) when they have already been generated.

Create one script to run once per country, which would:

• Takes command-line args: identifier shapefile_path_prefix voronoi_path
  • shapefile_path_prefix: The .shp and .dbf files should be at [shapefile_path_prefix].shp and [shapefile_path_prefix].dbf
    • If user specifies full path to .shp or .dbf files, should strip extension
• Convert shapefile to JSON
  • Save as [identifier]-shape.json
• Compute tower-to-admin
  • Save as [identifier]-tower-to-admin.csv
• Compute admin-to-tower
  • Save as [identifier]-admin-to-tower.csv

And a second script to run once for each day, which would:

• Takes command-line args: identifier tower_prefix mobility_path
• Reads in tower-to-admin and admin-to-tower using identifier and searching DATA_PATH for the filenames defined above
• Compute admin-to-admin, save as [identifier]-admin-to-admin.csv

All paths relative to the DATA_PATH constant, except for command-line args.

Finally, create a wrapper script that caches the outputs from the first script