khirotaka/sand

[Implementation example] Attend and Diagnose: Clinical Time Series Analysis Using Attention Models

SAnD

This has been archived.

AAAI 2018 Attend and Diagnose: Clinical Time Series Analysis Using Attention Models

Warning This code is UNOFFICIAL.

Paper: Attend and Diagnose: Clinical Time Series Analysis Using Attention Models

If you want to run this code, you need download some dataset and write experimenting code.

from comet_ml import Experiment
from SAnD.core.model import SAnD
from SAnD.utils.trainer import NeuralNetworkClassifier

model = SAnD( ... )
clf = NeuralNetworkClassifier( ... )
clf.fit( ... )

Installation

git clone https://github.com/khirotaka/SAnD.git

Requirements

• Python 3.6
• Comet.ml
• PyTorch v1.1.0 or later

Simple Usage

Here's a brief overview of how you can use this project to help you solve the classification task.

Download this project

First, create an empty directory.
In this example, I'll call it "playground".
Run the git init & git submodule add command to register SAnD project as a submodule.

$ mkdir playground/
$ cd playground/
$ git init
$ git submodule add https://github.com/khirotaka/SAnD.git

Now you're ready to use SAnD in your project.

Preparing the Dataset

Prepare the data set of your choice.
Remember that the input dimension to the SAnD model is basically three dimensions of [N, seq_len, features].

This example shows how to use torch.randn() as a pseudo dataset.

from comet_ml import Experiment

import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import TensorDataset, DataLoader

from SAnD.core.model import SAnD
from SAnD.utils.trainer import NeuralNetworkClassifier


x_train = torch.randn(1024, 256, 23)    # [N, seq_len, features]
x_val = torch.randn(128, 256, 23)       # [N, seq_len, features]
x_test =  torch.randn(512, 256, 23)     # [N, seq_len, features]

y_train = torch.randint(0, 9, (1024, ))
y_val = torch.randint(0, 9, (128, ))
y_test = torch.randint(0, 9, (512, ))


train_ds = TensorDataset(x_train, y_train)
val_ds = TensorDataset(x_val, y_val)
test_ds = TensorDataset(x_test, y_test)

train_loader = DataLoader(train_ds, batch_size=128)
val_loader = DataLoader(val_ds, batch_size=128)
test_loader = DataLoader(test_ds, batch_size=128)

Note:
In my experience, I have a feeling that SAnD is better at problems with a large number of features.

Training SAnD model using Trainer

Finally, train the SAnD model using the included NeuralNetworkClassifier.
Of course, you can also have them use a well-known training tool such as PyTorch Lightning.
The included NeuralNetworkClassifier depends on the comet.ml's logging service.

in_feature = 23
seq_len = 256
n_heads = 32
factor = 32
num_class = 10
num_layers = 6

clf = NeuralNetworkClassifier(
    SAnD(in_feature, seq_len, n_heads, factor, num_class, num_layers),
    nn.CrossEntropyLoss(),
    optim.Adam, optimizer_config={"lr": 1e-5, "betas": (0.9, 0.98), "eps": 4e-09, "weight_decay": 5e-4},
    experiment=Experiment()
)

# training network
clf.fit(
    {"train": train_loader,
     "val": val_loader},
    epochs=200
)

# evaluating
clf.evaluate(test_loader)

# save
clf.save_to_file("save_params/")

For the actual task, choose the appropriate hyperparameters for your model and optimizer.

Regression Task

There are two ways to use SAnD in a regression task.

1. Specify the number of output dimensions in num_class.
2. Inherit class SAnD and overwrite ClassificationModule with RegressionModule.

I would like to introduce a second point.

from SAnD.core.model import SAnD
from SAnD.core.modules import RegressionModule


class RegSAnD(SAnD):
    def __init__(self, *args, **kwargs):
        super(RegSAnD, self).__init__(*args, **kwargs)
        d_model = kwargs.get("d_model")
        factor = kwargs.get("factor")
        output_size = kwargs.get("n_class")    # output_size

        self.clf = RegressionModule(d_model, factor, output_size)


model = RegSAnD(
    input_features=..., seq_len=..., n_heads=..., factor=...,
    n_class=..., n_layers=...
)

The contents of both ClassificationModule and RegressionModule are almost the same, so the 1st is recommended.

Please let me know when my code has been used to bring products or research results to the world.
It's very encouraging :)

Author

Hirotaka Kawashima (川島 寛隆)

License

Copyright (c) 2019 Hirotaka Kawashima
Released under the MIT license

sand's Issues

why do you apply sqrt(d_model) in positional encoding and not in encoder block after attention?

First of all thank you for sharing your code!

I noticed that you apply the math.sqrt(self.d_model) in the forward method of the positional encoding, but from what I can see that is only applied once via EncoderLayerForSAnD, but in the paper it indicates that this scaling should be applied for each attention layer?

I could definitely be misunderstanding!

Improved Documentation (for Regression)

Hi, first of all thank you for taking the time to create this implementation. I have yet to find another. I'm using this model for a project, particularly for a regression problem.

I was wondering whether the regression module is fully complete, and if you would be able to add some documentation (how to properly instantiate it, description of each parameter etc)? I see that the core SAnD nn.module hardcodes the ClassificationModule instead of the RegressionModule, so I am wondering whether the model will work correctly if I simply replace it with RegressionModule and provide output_size instead of n_class?

It would also be great if the README contained a few brief, but complete examples for instantiating a SAnD model (for classification and regression). In its current form, it is unclear how the trainer and model work together, or how one might use the SAnD nn.module with other frameworks to train. Thanks!

[ToDo] Regression

ToDo RegressionModule

PositionalEncoder

In the paper they talk about: "The encoding is performed by mapping time step t to the same randomized lookup table during
both training and prediction."

In your code it seems more like using sinusoidal functions (and the one in the paper Attention is all You Need).

Is this Ok? Why not using then the randomized lookup solution?

How to deal with regression task?

Hi Khirotaka,

Thanks for proving the code.
I am a little confused about how to convert the default classification task to the regression one.

As you said in the README file, there are two steps:
Specify the number of output dimensions in num_class.
Inherit class SAnD and overwrite ClassificationModule with RegressionModule.

Am I did correct in the following steps:

1. Cuz it's a regression problem, num_class should be set to 1.
2. Also, I just simply convert ClassificationModule to RegressionModule in the model.py file.

Thanks for your time!

masked self-attention

Again a question about implementation vs paper. I haven't found in the code any implementation of what they call "masked self-attention".

Hope your comments,

Question regarding the input dataset

Hi,

I'm working on my thesis, and the aim is to study the progress of acute kidney injury up to several time steps ahead.

My plan is to use your code as the basis, and then add some things, like the ability to predict the progress of the disease while adding a level of uncertainty on every prediction, in order to know which predictions can be legit and trustworthy.

Having that, my first question is:
Because not all itemids (features) have the same number of measurements, I believe you have to rearrange the events in time steps in order to fit in the input dimension you mentioned: [N, seq_len, features].
If so, did you produce time steps for every hour? Every 2 hours?

The second question is:
Did you run any specific benchmark to produce your dataset?

Best regards,
Pedro Domingues