sindhura97/strats

STraTS: Self-Supervised Transformer for Sparse and Irregularly Sampled Multivariate Clinical Time-Series

This repo contains an official re-implementation of STraTS in pytorch.
Paper: https://arxiv.org/pdf/2107.14293.pdf

Important changes

• We included implementations for the following models: STraTS, GRU-D, InterpNet, GRU, TCN, SaND
• For STraTS, we removed LayerNorm and replaced ReLU activations in the FFN with GELU, as this improved the performance.
• We used mostly similar hyperparameters for both the datasets, and used the same hidden dimension of 64 for all models.
• Taking inspiration from GRU-D, which uses the same dropout mask at each input time-step, resulting in masking out some variables, we also drop a fraction of variables from the input while training STraTS.
• These changes, along with some possible differences arising from Pytorch's inbuilt modules, make some of the results deviate from the original numbers published in the paper.

Conda env setup

conda create -n strats python=3.10.9
source activate strats
pip install pytz pandas tqdm matplotlib 
pip install torch==2.0.0+cu117 torchvision==0.15.1+cu117 torchaudio==2.0.1 --index-url https://download.pytorch.org/whl/cu117
pip install transformers==4.35.2
pip install scikit-learn==1.2.2

Dataset preprocessing

Download PhysioNet2012 dataset from https://physionet.org/content/challenge-2012/1.0.0/.
Download MIMIC-III from https://physionet.org/content/mimiciii/1.4/,
Update "RAW_DATA_PATH" variable in the preprocessing scripts and run them.

python preprocess_physionet_2012.py
python preprocess_mimic_iii_large.py

Training and evaluation

The shell script run_main.sh contains the commands for training and evaluating each of the supported models.

chmod +x run_main.sh
./run_main.sh

Results from run_main.sh

Citation

If you found this work useful, please cite our paper:

@article{tipirneni2022self,
  title={Self-supervised transformer for sparse and irregularly sampled multivariate clinical time-series},
  author={Tipirneni, Sindhu and Reddy, Chandan K},
  journal={ACM Transactions on Knowledge Discovery from Data (TKDD)},
  volume={16},
  number={6},
  pages={1--17},
  year={2022},
  publisher={ACM New York, NY}
}

strats's Issues

Model interpretability missing from new Pytorch implemenation

Contribution scores seem to be missing from the latest version of the repo. I-STraTS is built but there is no reference to how to generate the scores shown in the paper. How would one go about obtaining these scores?

Missing information in strats_notebook.ipynb (fore_max_len)

Under the comment "Find max_len." there is tn explanation or programatic path to follow to know what this value should be. The number 880 is hardcoded in. What should this value be if trying to apply this code to a different project?

means = demo.mean(axis=0, keepdims=True)
stds = demo.std(axis=0, keepdims=True)
stds = (stds==0)*1 + (stds!=0)*stds
demo = (demo-means)/stds
# Get variable indices.
varis = sorted(list(set(data.variable)))
V = len(varis)
var_to_ind = inv_list(varis, start=1)
data['vind'] = data.variable.map(var_to_ind)
data = data[['ts_ind', 'vind', 'hour', 'value']].sort_values(by=['ts_ind', 'vind', 'hour'])
# Find max_len.
fore_max_len = 880

Problem during executing pretrain on forecasting

Always run into incompatible shape error. Error occurs when executing the line

e_loss += fore_model.train_on_batch([ip[ind] for ip in fore_train_ip], fore_train_op[ind])

Potential bug

Hi, I just tried reproducing your results in strats_notebook.ipynb and I got an AttributeError when running:

model, fore_model = build_strats(D, fore_max_len, V, d, N, he, dropout, forecast=True)

Apparently there is a problem with the Transform layer.

Add a LICENSE

To actually allow others to use, adapt, modify the code of STraTS, a LICENSE file should be added to the repository.

For more details on licence-less repositories, see https://choosealicense.com/no-permission/ . Adding a license will allow the authors to retain copyright while controlling what can be done with their work.

I suggest the permissive open-source licenses APACHE 2.0 or MIT.

A problem about the model interpretability

I use this model to predict the crash label in a driving safety modeling dateset. Using this code, I can output the predicted labels, but cannot get the model interpretability results. I wonder if this function 'build_strats_interp()' in the origin codes can work? Thanks a lot!