lixinsu/lion

semantic matching toolkit

lion's Issues

Bug in load_embedding function

The word_dict is not passed to the function and limit max text length

Add contextual vectors

• ELMo

Automatically set `classes`

Detect the number of classes in the training set, rather than manually setting classes

Add snli dataset

add script in data/datset_utils

Fix evaluate predict mode bugs

Add QNLI dataset

This dataset is smaller than both QQP and SNLI, it has only 10k training data. It's more convenient to first test model performance in this dataset.

Logger config

Configure logging and add some necessary log

Add bert

add/XLNet

Improve Params class

Existing implementation cannot support vars param[k]=v and param.k=v

support RoBERTA

Fix make_char bugs and update the qnli performance for esim/bimpm model

Add ignore, requirments and other file

ignore is already on

[dataset error]

Traceback (most recent call last):
File "lion/training/trainer.py", line 33, in
train_model('lion/configs/test_bimpm_1.yaml')
File "lion/training/trainer.py", line 25, in train_model
model.train_epoch(train_loader)
File "/home/fanyixing/users/mxy/lion/lion/training/model.py", line 85, in train_epoch
for ex in tqdm(data_loader):
File "/home/fanyixing/users/wangsu/anaconda3/envs/pytorch/lib/python3.6/site-packages/tqdm/_tqdm.py", line 979, in iter
for obj in iterable:
File "/home/fanyixing/users/wangsu/anaconda3/envs/pytorch/lib/python3.6/site-packa
ges/torch/utils/data/dataloader.py", line 336, in next
return self._process_next_batch(batch)
File "/home/fanyixing/users/wangsu/anaconda3/envs/pytorch/lib/python3.6/site-packa
ges/torch/utils/data/dataloader.py", line 357, in _process_next_batch
raise batch.exc_type(batch.exc_msg)

TypeError: Traceback (most recent call last):
File "/home/fanyixing/users/wangsu/anaconda3/envs/pytorch/lib/python3.6/site-packa
ges/torch/utils/data/dataloader.py", line 106, in worker_loop
samples = collate_fn([dataset[i] for i in batch_indices])
File "/home/fanyixing/users/wangsu/anaconda3/envs/pytorch/lib/python3.6/site-packa
ges/torch/utils/data/dataloader.py", line 106, in
samples = collate_fn([dataset[i] for i in batch_indices])
File "/home/fanyixing/users/mxy/lion/lion/data/dataset.py", line 26, in getitem
return self.vectorize(self.examples[index])
File "/home/fanyixing/users/mxy/lion/lion/data/dataset.py", line 56, in vectorize
Bchar = torch.LongTensor([make_char(char_dict, w) for w in ex['Btokens']])
File "/home/fanyixing/users/mxy/lion/lion/data/dataset.py", line 56, in
Bchar = torch.LongTensor([make_char(char_dict, w) for w in ex['Btokens']])
File "/home/fanyixing/users/mxy/lion/lion/data/dataset.py", line 48, in make_char
return [char_dict(t) for t in token[:8]] + [char_dict(t_) for t_ in token[-8:]]
File "/home/fanyixing/users/mxy/lion/lion/data/dataset.py", line 48, in │
return [char_dict(t_) for t_ in token[:8]] + [char_dict(t_) for t_ in token[-8:]]
TypeError: 'Dictionary' object is not callable

Add bimpm model

add model

support ernie, xlnet, bert, roberta 中文

add model

https://arxiv.org/pdf/1511.04108.pdf
baseline

Add esim model

add esim model

Fix bimpm elmo bug

add model

https://www.aclweb.org/anthology/P19-1465.pdf

Support chinese

Add test and class doc for the whole project

Add model

• Fusionnet

Add model

• Match lstm

Complete the model save and load function in trainer.py

Check model essential param

Vocabulary class bug

For ner or pos label mapping, there is no need for unk and null tok.

Feature/unify_mask

Now, esim and bmipm model use 1 mask value, but bert use 0 mask value.
Unify the mask mechanism which use 0 mask value, this is also consistent with tf.

Add Readme: including model performance on QQP