bigandsweet/a-comparative-study-of-loss-functions-traffic-predictions-in-regular-and-congestion-scenarios

Source Code for the paper "A Comparative Study of Loss Functions: Traffic Predictions in Regular and Congestion Scenarios"

A Comparative Study of Loss Functions: Traffic Predictions in Regular and Congestion Scenarios

This repository reproduces the results reported in A Comparative Study of Loss Functions: Traffic Predictions in Regular and Congestion Scenarios.

Requirements

Dependency can be installed using the following command:

pip install -r requirements.txt

Retraining all the models

Under the four folders of the form $MODEL(-quantile) (D2STGNN, D2STGNN-quantile, Graph-WaveNet, Graph-WaveNet-quantile), we provide the adapted source code for each model to reproduce the results in this paper.

• You can follow the instructions in the README.md under each of these four folders to retrain the models. After training, the best model checkpoints in training will be saved under archive, and the predicted traffic speed in the test set will be saved under result.
• Quantile loss requires a slightly different model output structure than the other loss functions. -quantile means that the we made changes to the code in order to adapt to the Quantile Loss implemented in Quantifying Uncertainty in Deep Spatiotemporal Forecasting, KDD 2021.
• All the other loss functions (mae, mse, mae-focal, mse-focal, bmse1, bmse9, huber, kirtosis, Gumbel) are implementeed under $MODEL. For Graph-WaveNet, the loss functions are implemented under util.py; for D2STGNN, the loss functions are implemented under models/losses.py.

Dataset

Download the METR-LA and PEMS-BAY folders. Both folders contain

• two subfolders GraphWaveNet and D2STGNN. These two folders contain
  • the best model checkpoints in training, under models folder; and
  • all the predicted traffic speed in the test set in $MODEL-$LOSS.pkl.
  • These can also be generated by retraining the models, as described in the last section.
• Source data files: METR-LA/metr-la.h5, METR-LA/graph_sensor_locations.csv, PEMS-BAY/pems-bay.h5, PEMS-BAY/graph_sensor_locations_bay.csv.
• change_points.pkl, change_point_intervals.pkl and congested.pkl. These can also be generated following the instructions below.

Metric Computation

To run our metric computation, we need change_point_intervals.pkl and congested.pkl for the METR-LA and PEMS-BAY datasets.

The change_points.pkl contains all the identified change points in the time series of the historical traffic speed of each of the sensor location. The change_point_intervals.pkl incorporates two-time steps preceding and following each identified change point. To regenerated all the change points in METR-LA and PEMS-BAY datasets, run

python find_change_points.py --path $DATA_PATH

Replace $DATA_PATH by METR-LA or PEMS-BAY. For example, you can run

python find_change_points.py --path METR-LA

Please note that due to the computational cost of offline changepoint identification algorithms, this can take a few hours to run.

The congested.pkl contains all the sensor locations with significant bimodal pattern in the histogram of the historical traffic speed. To regenerate this file for METR-LA and PEMS-BAY datasets, run

python bimodal.py --path $DATA_PATH

Replace $DATA_PATH by METR-LA or PEMS-BAY. For example, you can run

python bimodal.py --path METR-LA

If you would like to visualize all the bimodal distributions, as well as an interactive map as shown in Figure 2 of the paper, you can run bimodal-LA.ipynb or bimodal-bay.ipynb.

Finally, run

python metrics.py --model GraphWaveNet --dataset METR-LA
python metrics.py --model D2STGNN --dataset METR-LA
python metrics.py --model GraphWaveNet --dataset PEMS-BAY
python metrics.py --model D2STGNN --dataset PEMS-BAY

to generate all the tables in the paper.