pratham16cse/aggforecaster

Code for "Coherent Probabilistic Aggregate Queries on Long-horizon Forecasts", IJCAI 2022

Coherent Probabilistic Aggregate Queries on Long-horizon Forecasts

This is the code produced as part of the paper Coherent Probabilistic Aggregate Queries on Long-horizon Forecasts

Coherent Probabilistic Aggregate Queries on Long-horizon Forecasts.

Prathamesh Deshpande and Sunita Sarawagi. IJCAI 2022. arXiv:2111.03394v1.

Package Dependencies / Requirements

• Python 3.7.9 (recommended).
• All the requirements are specified in requirements.txt.

Run

./script.sh

How to work with Command Line Arguments?

• If an optional argument is not passed, it's value will be extracted from configuration specified in the file main.py (based on dataset_name, model_name).
• If a valid argument value is passed through command line arguments, the code will use it further. That is, it will ignore the value assigned in the configuration.

Command Line Arguments Information

Argument name	Type	Valid Assignments	Default
dataset_name	str	azure, ett, etthourly, Solar, taxi30min, Traffic911	positional argument
saved_models_dir	str	-	None
output_dir	str	-	None
N_input	int	>0	-1
N_output	int	>0	-1
epochs	int	>0	-1
normalize	str	same, zscore_per_series, gaussian_copula, log	None
learning_rate	float	>0	-1.0
hidden_size	int	>0	-1
num_grulstm_layers	int	>0	-1
batch_size	int	>0	-1
v_dim	int	>0	-1
t2v_type	str	local, idx, mdh_lincomb, mdh_parti	None
K_list	[int,...,int ]	[>0,...,>0 ]	[]
device	str	-	None

Datasets

All the datasets can be found here.

Add the dataset files/directories in data directory before running the code.

Output files

Targets and Forecasts

Following output files are stored in the <output_dir>/<dataset_name>/ directory.

File name	Description
inputs.npy	Test input values, size: number of time-series x N_input
targets.npy	Test target/ground-truth values, size: number of time-series x N_output
<model_name>_pred_mu.npy	Mean forecast values. The size of the matrix is number of time-series x number of time-steps
<model_name>_pred_std.npy	Standard-deviation of forecast values. The size of the matrix is number of time-series x number of time-steps

Metrics

All the evaluation metrics on test data are stored in <output_dir>/results_<dataset_name>.json in the following format:

{
  <model_name1>: 
    {
      'crps':<crps>,
      'mae':<mae>,
      'mse':<mse>,
      'smape':<smape>,
      'dtw':<dtw>,
      'tdi':<tdi>,
    }
  <model_name2>: 
    {
      'crps':<crps>,
      'mae':<mae>,
      'mse':<mse>,
      'smape':<smape>,
      'dtw':<dtw>,
      'tdi':<tdi>,
    }
    .
    .
    .
}

Here <model_name1>, <model_name2>, ... are different models under consideration.

aggforecaster's Issues

TODOS

• Plotting parameter
• json parser
• Re-structure main file, and others if needed
• save epoch to resume the training
• Add 911 and taxi datasets, but only in the context of counts
• Features??

Add M5 dataset

Inspect seq2seqnll_dualtpp model for bugs

In the current state, the model gives flat predictions.

However, seq2seqnll_optls model returns correct predictions.

Taxi data possibilities

1. Consider all zones irrespective of number of pickups -- If a zone does not have a pickup in an hour, the value will be zero
2. Consider only zones that have at least one pickup per hour
3. Only consider top k zones according to number of pickups over the duration

• If required, get data for more months as well

Sine data variance bugfix

Forecast method

As per my under standing of your paper ('Long Range Probabilistic Forecasting in Time-Series using High Order Statistics'), you are taking multiple (2) averages of the original series and then the informer model is being trained independently on these averages and the original series as well. I have a couple of questions here.

1. How many informer models are being trained?...i.e if we have 3 time series (2 aggregate and 1 original) does 1 informer model get trained on these 3 time series independently or does each time series get its own informer model ?

2. Once the model trained, a new multivariate gaussian is being defined with mean and covariance. How exactly is this mean being calculated ?

Train base model in teacher-forcing mode

• The model makes sequential predictions during training also.
• This might be the reason for poor performance of DILATE and MSE on Taxi dataset as forecast horizon at the bottom level is sufficiently large.
• Add teacher-forcing mode of training in order to better learn the parameters of the base model at bottom level.

Set DILATE parameter values for datasets that are present in the DILATE paper

Remove redundant models for bottom level

If two different inference models share a bottom level model, do not retrain, use the existing bottom-level model.

Fit Slope+(Sum or Average) instead of Slope+Intercept

Parser for Datasets of Gaussian Copula Paper

• In both train.json and test.json, each line is a time-series.
• In test.json, each series can be treated as an independent sequence.
• Context length needs to be limited for both train and test.
• For wiki, find out why initial ~9000 train and test sequences overlap.
• For selecting only 2000 sequences for wiki, how to ensure same 2000 sequences get selected from both train and test?
• Create this pre-processing for Solar, Wiki, and Exchange-rate.

Incrementally optimize for each bin during inference.

Adjust Model Complexity according to value of K

• Explore the heuristics to reduce model complexity as value of K increases.
• If the aggregate itself tracks multiple values, increase the complexity accordingly?

During training, use dilate_loss for early stopping

• When base model is DILATE, compute dilate_loss(..) on val data during training.
• A separate metric for early stopping for each model based on sutaibility.

Normalization

• Try normalization only at bottom level
• Find out how to handle normalization across levels

Variance as a function of distance from forecast begin

• Plot variance along with mean predictions
• Explore variance as a function of how far are we predicting in the future.

Learning Rate Tuning

• Fix a basic learning rate tuning method that can be applied across all base models
• Can try learning rate schedule also?

DILATE runs into NaN issue with Taxi dataset

On taxi dataset, at step 31 of the training epoch, loss.backward() assigns NaN gradients to all the parameters in the network

Include datasets used in other papers such as Tensor Train RNNs

Use K as a hyper-parameter

• Understand how to perform hyper-parameter turning in pytorch.
• Get multiple solutions using multiple values of K.
• Select the K that performs best on validation data.

Add LSTNet Baseline

Need better way to set stride while creating batches for training

Possible options:

• Manually set stride for each dataset such that the model can see maximum possible patterns of input-output sequences in the training data.
• Add stochasticity in the stride. For example, if stride is s, shift the window by U(s-r,s+r) where r is width for stochasticity

Errors running script.sh

This project looks exciting, but Im having trouble running the project. I don't see a requirements.txt or environment.yml, so I did my best to create one.

It looks like there might be some missing files from the project too:

in base_model.py

from models import transformer_manual_attn, transformer_dual_attn
ImportError: cannot import name 'transformer_manual_attn' from 'models' (unknown location)

Thanks for the help

Least Square Aggregation

• Create first level of aggregation
• Model and overall code should support multiple values in the time-series
• Create new inference model for least square based aggregation
• Write optimizer for least square consistency during inference
• Modify real dataset processing such that each sequence is 3 dimensional at level 1
• Multiple aggregation support in the main.py file (sum, leastsquare)

Multi-aggregate Training and Inference

• Add multiple aggregations with different values of K.
• Design Inference for such aggregations.
• Try using K as a hyper-parameter?