not good when I use BERT for seq2seq model in keyphrase generation

[huggingface]

not good when I use BERT for seq2seq model in keyphrase generation

[whqwill]

I know, but ... the same problem ... my memory is limited .. so ...

PS. I am Chinese

[waynedane]

是不是语料的问题，bert是在wiki上训练的。我用kp20k训练了一个mini bert，在测试集上的accuracy目前是80%，你要不要试试用我这个作为encoder？

[thomwolf]

Hi guys,
I would like to keep the issues of this repository focused on the package it-self.
I also think it's better to keep the conversation in english so everybody can participate.
Please move this conversation to your repository: https://github.com/memray/seq2seq-keyphrase-pytorch or emails.
Thanks, I am closing this discussion.
Best,

[waynedane]

have u tried transformer decoder ?instead of rnn decoder.

[whqwill]

not yet, I will try. But I think rnn decoder should not be such bad.

[waynedane]

not yet, I will try. But I think rnn decoder should not be such bad.

emmm，maybe u should used mean of last layer to initialize decoder, not the last token representation of last layer.
I am also very concerned about the results of using transformer decoder. If you are done, can you tell me? Thank you.

[waynedane]

I think the batch size of RNN with BERT is too small. pleas see

https://github.com/memray/seq2seq-keyphrase-pytorch/blob/master/pykp/dataloader.py
line 377-378

[whqwill]

I don't know what you mean by giving me this link. I set to 10 really because of the memory problem. Actually, when sentence length is 512, the max batch size is only 5, if it is 6 or bigger there will be memory error for my GPU.

[whqwill]

not yet, I will try. But I think rnn decoder should not be such bad.

emmm，maybe u should used mean of last layer to initialize decoder, not the last token representation of last layer.
I am also very concerned about the results of using transformer decoder. If you are done, can you tell me? Thank you.

You are right. Maybe the mean is better, I will try as well. Thanks.

[waynedane]

May i ask a question? R u chinese?23333

[waynedane]

Cause for one example, it has N targets. We wanna put all targets in the same batch. 10 is too small that the targets of one example would be in different batches probably.

[waynedane]

I know, but ... the same problem ... my memory is limited .. so ...

PS. I am Chinese

i am as well hahaha

[whqwill]

这个80%具体是什么数值这么高？f1 score吗？ 你的encoder能不能发来看一下呢 谢 waynedane <[email protected]>于2018年11月28日 周三下午11:14写道：

…

是不是语料的问题，bert是在wiki上训练的。我用kp20k训练了一个mini bert，在测试集上的accuracy目前是80%，你要不要试试用我这个作为encoder？ — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#59 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHCjdIT1G6Icse3LK2SZXO194JJTiM1Qks5uzqhMgaJpZM4Y3HWV> .

[waynedane]

accuracy 是masklm和nextsentence两个任务的，不是key phrase generation，我没说清楚，抱歉。我的算力有限，两块p100, 快一个月了，目前还没训练完。80%是当前的表现。

[whqwill]

你提到的mini bert 是什么意思？

[whqwill]

我大概理解你的意思了，你相当于是用kp20重新预训练一个bert，不过这样做... 感觉确实蛮麻烦。

[waynedane]

我大概理解你的意思了，你相当于是用kp20重新预训练一个bert，不过这样做... 感觉确实蛮麻烦。

是的，用的是 Junseong Kim的代码：https://github.com/codertimo/BERT-pytorch ，模型规模比谷歌的BERT-Base Uncased都小很多。这个是L-8 H-256 A-8.我把目前训练的checkpoint和vocab文件发给你

[whqwill]

但是你这个checkpoint，我的这个版本能直接用吗，还是说我必须装你的那个版本的代码？

[whqwill]

你可以发到我邮箱 [email protected] ， 谢

[waynedane]

但是你这个checkpoint，我的这个版本能直接用吗，还是说我必须装你的那个版本的代码？

可以根据Junseong Kim 的代码创建一个bert model然后加载参数，不一定得安装

[whqwill]

好的把。那你把checkpoint 发给我试试。

[InsaneLife]

accuracy 是masklm和nextsentence两个任务的，不是key phrase generation，我没说清楚，抱歉。我的算力有限，两块p100, 快一个月了，目前还没训练完。80%是当前的表现。
你好，能把mini版模型发我一下吗，[email protected]，谢谢啦。

[Accagain2014]

hi, @whqwill I have some doubts about the usage manner of bert with RNN.
In bert with RNN method, I see you only consider the last term's representation (I mean the TN's) as the input to RNN decoder, why not use the other term's representation, like T1 to TN-1 ? I think the last term's information is too less to represent all the context information.