I'm trying to train tagging probes using precomputed representations from the encoder of my Transformer-based machine translation model, but am getting the following error when training attempts to begin:
2020-07-29 05:23:27,936 - INFO - allennlp.training.trainer - Training
...
File "/raj-learn/contextual-repr-analysis/contexteval/models/tagger.py", line 340, in forward
average=self.loss_average)
File "/raj-learn/envs/contextual_repr_analysis/lib/python3.6/site-packages/allennlp/nn/util.py", line 633, in sequence_cross_entropy_with_logits
negative_log_likelihood_flat = - torch.gather(log_probs_flat, dim=1, index=targets_flat)
RuntimeError: invalid argument 2: Input tensor must have same size as output tensor apart from the specified dimension at /pytorch/aten/src/THC/generic/THCTensorScatterGather.cu:29
[INFO/MainProcess] process shutting down
I've computed these representations and saved them in a correctly-formatted hdf5 file using the make_hdf5_file
function from the README. I then confirmed that this hdf5 file looks as expected: the keys are string id's that map to (N_tokens, 1024) numpy arrays. N_tokens is the length of that sentence and 1024 is my encoder representation dim (which I correctly pass into my config file). sentence_to_index also looks fine, mapping newline-stripped sentences to string id's.
I get this error when training begins with both the CCG supertagging task and the PTB POS tagging task (which both use the WSJ sentences dataset).
Here's my config file for CCG supertagging (modeled off of yours for ELMo original):
{
"dataset_reader": {
"type": "ccg_supertagging",
"contextualizer": {
"type": "precomputed_contextualizer",
"representations_path": "../data/precomputed_reps/ccg_wsj_sentences_all.hdf5"
}
},
"validation_dataset_reader": {
"type": "ccg_supertagging",
"contextualizer": {
"type": "precomputed_contextualizer",
"representations_path": "../data/precomputed_reps/ccg_wsj_sentences_all.hdf5"
}
},
"train_data_path": "../data/probing_task_data/ccg/train.txt",
"validation_data_path": "../data/probing_task_data/ccg/dev.txt",
"test_data_path": "../data/probing_task_data/ccg/test.txt",
"evaluate_on_test" : true,
"model": {
"type": "tagger",
"token_representation_dim": 1024
},
"iterator": {
"type": "basic",
"batch_size" : 80
},
"trainer": {
"num_epochs": 50,
"patience": 3,
"cuda_device": 0,
"validation_metric": "+accuracy",
"optimizer": {
"type": "adam",
"lr": 0.001
}
}
}
I'm going to try looking into this error now and will update if I figure out what's going on. I cloned the contextual-repr-analysis repo and installed dependencies from requirements.txt, so I imagine I should have the version of allennlp + conllu that were used at the time this package was written.