Stanford CS224N Natural Language Processing with Deep Learning
We present our results in this paper.
mbounded-dyck-k parenthesis dataset is included in the data/mbounded-dyck-k
directory.
To acquire Penn Treebank and Wiki2 datasets:
sh getdata.sh
run.sh
includes all training and testing commands.
- When
--log <LOG_NAME>
is specified, the model parameters, checkpoints, and tensorboard log will be saved under/log/LOG_NAME
. Without the flag, no logs will be recorded. - If
--is-stream 0
is specified, we treat each line in the dataset as a separate sample sentence.<start>
and<end>
tokens are added to each sentence and all sentences are padded to the same length then batched. The number of samples in the dataset must be divisible by the batch size. This option is used for the parenthesis dataset and syntax evaluation dataset. - If
--is-stream 1
is specified, we treat the entire dataset as one continuous stream. The last hidden state of a batch is used to initialize the LSTM for the next batch. The dataset to padded to a multiple of batch size and bptt. The backprop through time flag--bptt
determines the sequence length of each batch. This option is used for natural language datasets like Penn Treebank and Wiki2.
- To show the principal angles between sub-spaces (PABS) for pairs of hidden-to-hidden weight matrices, run
compare_weights.py
with a model checkpoint. - To evaluate the syntax of generated texts, see the documentation in
LM_syneval
. - (Experimental) To generate heatmaps, go to the
heatmap
branch and run:
python3 run.py --mode train --model attention --train-path=./data/wikitext-2/train.txt --valid-path=./data/wikitext-2/valid.txt --checkpoint ./LOG_PATH/best_val_ppl.pth --hidden-size 600 --embedding-size 300 --batch-size 10 --bptt 70 --dropout 0.5 --lr 1e-4 --is-stream 1 --epochs 1000 --second-order-size 5 > ./data/heatmaps/sample_att_wiki_5cell.txt
- Mode should be set to train because we need temperature set to high to be able to mark how confident each word is assigned to each cell. The batch info and attention info is printed and piped to an output file. We use the output file to generate heatmaps.
python3 utils/heatmap.py --input data/heatmaps/sample_att_wiki_5cell.txt --output data/heatmaps/ --prefix att_wiki_5cell
We have modified the baseline and attention model to work with RNN instead of LSTM in the rnn
branch. However we did not have time to verify the correctness of the code and aggregate results for those models.
test_dataset.py
tests the validity of the dataset objecttest_eval.py
tests the validity of the evaluation metricstest_model.py
tests the validity of the models