ictnlp-wshugen/annotated-transformer_codes

A partial runnable code repo for annotated-transformer

annotated-transformer_codes's Issues

performance on WMT14 en-de

how's the bleu score on the dataset WMT14,please?

RuntimeError: Expected object of backend CUDA but got backend CPU for argument #3 'index'

i have four GPUs，so i changed devices to 【0，1，2，3】and change batchsize to 2000，but when it runs for 10 iterations it cames out the runtime error

Reminder: Call List Typo in "realworld.py"

sym = TGT.vocab.itos(batch.trg.data[i, 0])

should be:

sym = TGT.vocab.itos[batch.trg.data[i, 0]]

TGT.vocab.itos is a list, should use "[]" to index

Train with custom dataset

Hi, I am pretty new here. I would like to know how do we provide the dataset instead of using the one that comes with the library. I have a parallel corpus that is stored in two text files as source.txt and target.txt. I see all the annotated transformer model use the dataset that comes with the library and not with a custom dataset. Please provide a link or tutorial so that I can modify my dataset in order to feed into this model. Many Thanks.

AssertionError when running on 4 GPU's: len(modules) == len(inputs)

When running on 2 or 3 GPU's everything works fine. When running on 4 GPU's however, during the first epoch:

Epoch step: 1 Loss 9.134843 Tokens per Sec: 161.252618
Traceback (most recent call last):
  File "realworld.py", line 83, in <module>
    MultiGPULossCompute(model.generator, criterion, devices=devices, opt=model_opt))
  File "/export/home1/NoCsBack/hci/rubenc/transformer-v2/transformer/flow.py", line 53, in run_epoch
    for i, batch in enumerate(data_iter):
  File "realworld.py", line 82, in <genexpr>
    run_epoch((rebatch(pad_idx, b) for b in train_iter), model_par,
  File "/export/home1/NoCsBack/hci/rubenc/miniconda3/envs/transfenv/lib/python3.7/site-packages/torchtext/data/iterator.py", line 141, in __iter__
    self.init_epoch()
  File "/export/home1/NoCsBack/hci/rubenc/miniconda3/envs/transfenv/lib/python3.7/site-packages/torchtext/data/iterator.py", line 117, in init_epoch
    self.create_batches()
  File "/export/home1/NoCsBack/hci/rubenc/transformer-v2/transformer/my_iterator.py", line 18, in create_batches
    from realworld import dividable_size
  File "/export/home1/NoCsBack/hci/rubenc/transformer-v2/realworld.py", line 83, in <module>
    MultiGPULossCompute(model.generator, criterion, devices=devices, opt=model_opt))
  File "/export/home1/NoCsBack/hci/rubenc/transformer-v2/transformer/flow.py", line 55, in run_epoch
    loss = loss_compute(out, batch.trg_y, batch.ntokens)
  File "/export/home1/NoCsBack/hci/rubenc/transformer-v2/transformer/multi_gpu_loss_compute.py", line 35, in __call__
    gen = nn.parallel.parallel_apply(generator, out_column)
  File "/export/home1/NoCsBack/hci/rubenc/miniconda3/envs/transfenv/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 37, in parallel_apply
    assert len(modules) == len(inputs)
AssertionError

I suspect this has something to do with pytorch/pytorch#5587 or pytorch/pytorch#11793. Any tips?