crickwu/dl4epi

Deep Learning for Epidemiological Predictions

Deep Learning for Epidemiological Predictions

Paper

Deep Learning for Epidemiological Predictions - Yuexin Wu et. al. SIGIR 2018

The overall structure is composed of 3 parts: a CNN for capturing correlation between signals, a RNN for linking up the dependencies in the temporal dimension and the residual links for fast training and overfitting prevention. We carefully restrain the parameter space, making the total model have a similar size as autogression models.

Dependencies

Python == 2.7, Pytorch == 0.2.0, numpy

Update: the current code can be executed with Python >= 3 and Pytorch >= 0.4.0

How to Run

Preprocessing: run bash ./sh/mklog.sh to create empty log save folders.

Simple Example

python main.py --normalize 1 --epochs 2000 --data ./data/us_hhs/data.txt --sim_mat ./data/us_hhs/ind_mat.txt --model CNNRNN_Res \
--dropout 0.5 --ratio 0.01 --residual_window 4 --save_dir save --save_name cnnrnn_res.hhs.w-16.h-1.ratio.0.01.hw-4.pt \
--horizon 1 --window 16 --gpu 0 --metric 0

Experiment for a Single Dataset/Method

For CNNRNN_Res, CNNRNN, VAR_mask:

Update: new method VAR_mask is added. This baseline is the same with VAR except that the correlation is only considered when two signals are adjacent (designed by ind.txt),

bash ./sh/grid_<model>.sh <data_path> <adj_mat_path> <log_info> <gpu_number> <normalization>

e.g.

bash ./sh/grid_CNNRNN_Res.sh ./data/us_hhs/data.txt ./data/us_hhs/ind_mat.txt hhs 0 1

For VAR, GAR, AR:

bash ./sh/grid_<model>.sh <data_path> <log_info> <gpu_number> <normalization>

e.g.

bash ./sh/grid_VAR.sh ./data/us_hhs/data.txt hhs 0 1

Use python log_parse.py to parse the results in log\.

Full Experiment in Paper

NOTICE: This may take LONG time. Consider running single experiments first to estimate time.

bash ./sh/run_all.sh
python log_parse.py

Option Explanation

For main.py

normalize: normalization options
	0: no normalization
	1: signal/row-wise normalization
	2: global/matrix-wise normalization

More information can be found in main.py --help.

Log Format

rse: Root-mean-square error
rae: Absolute error
correlation: Pearson correlation score

Notice

The results in the paper are produced by pytorch-0.2.0, which seems to have some unstable numerical issues and is likely to produce NaN in some cases. If that happens, you may try rerun the code or switch to a more stable pytorch version (e.g. 0.4.0) for more robust prediction.

Citation

@inproceedings{wu2018deep,
  title={Deep Learning for Epidemiological Predictions},
  author={Wu, Yuexin and Yang, Yiming and Nishiura, Hiroshi and Saitoh, Masaya},
  booktitle={The 41st International ACM SIGIR Conference on Research \& Development in Information Retrieval},
  pages={1085--1088},
  year={2018},
  organization={ACM}
}

dl4epi's Issues

how do you constrcut the ind_mat.txt

hello I do not understand how do you construct ind_mat , can you explain ? Thanks