m-nolan/mrae

Multiblock RNN Autoencoder

mrae.py

Michael Nolan

Multiblock Recurrent Autoencoder implementation in Pytorch

mrae's Issues

Fix `device` assignment - not yet implemented. Required for GPU training.

mrae/mrae/MRAE.py

Line 20 in 1951d46

max_grad_norm=5.0, rand_samp=True, device='cpu'):

The device input to the model is entirely for show. This needs to be applied to all parts of the model. Also, every array created within the model needs to be assigned to the appropriate memory location.

`mrae.fit()` training loop needs early stopping mechanism

mrae/mrae/MRAE.py

Line 111 in a32fd63

for epoch_idx in tqdm(range(max_epochs)):

The current training loop runs and optimizes the model to minimize training loss. This has no stop point beyond the max iterations argument.

This needs an early stopping mechanism, either a moving average of the loss delta, some train/valid loss crossover, etc.

Test with the sample dataset.
Add a test with random data (?) to show that it works to stop training if it isn't learning anything.

Refactor `run_example.py`

mrae/run_example.py

Line 10 in 1951d46

current_dir = os.path.dirname(__file__)

The directory management and hparam.yaml file management should be refactored to utils.py.

The rest of the script should be refactored into a def main() method that's called from the if __name__ == "__main__": ... block at the end of the file. Maybe the "create_mrae_from_hyperparams()" method, etc, could be moved to utils.py as well.

Memory issue when CUDA training

mrae/mrae/MRAE.py

Line 267 in 7f6743c

mrae_out = self(input)

Something breaks when switching to CUDA. Memory blows up by training epoch 10, max.

Save current code version to run directory

mrae/mrae/MRAE.py

Line 114 in 1951d46

def _initialize_opt(self,save_dir,overwrite):

I don't plan to change the model architecture at this point, other than perhaps to move the training loop out of the MRAE model and into a separate training run manager class. Regardless, having the current code state (i.e. git commit hash, etc) saved into the run directory along with the model checkpoints is a good idea. Either adding this at the run initialization or at the beginning of the training script would work, I think.

Save train/valid loss values

mrae/mrae/MRAE.py

Line 204 in 1951d46

epoch_valid_loss = self.compute_epoch_loss(

The train/valid objective and loss outputs should be saved for each epoch. Maybe some stats over the batch as well?

You would want to be able to write to this again after resuming a previous run, so something simple like a .csv file would work fine.

ex:
epoch_idx,train_total_loss,train_output_loss,train_block_loss_0,train_block_loss_1,...,valid_total_loss,valid_output_loss,...