paper-reading's People
paper-reading's Issues
ð[2016] Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
0. è«æ
ã¿ã€ãã«
Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
ãªã³ã¯
https://arxiv.org/abs/1506.02142
èè
Yarin Gal, Zoubin Ghahramani (University of Cambridge)
æçš¿æ¥ä»
2015/06/06 ICML 2016
1. ã©ããªãã®ïŒ
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
5. è°è«ã¯ããïŒ
6. 次ã«èªãã¹ãè«æã¯ïŒ
[2019] UniformFace: Learning Deep Equidistributed Representation for Face Recognition
0. è«æ
ã¿ã€ãã«
UniformFace: Learning Deep Equidistributed Representation for Face
Recognition
ãªã³ã¯
èè
Yueqi Duana, Jiwen Lu, Jie Zhou
æçš¿æ¥ä»
CVPR 2019
1. ã©ããªãã®ïŒ
é¡èªèã¿ã¹ã¯ã«ãããŠãåã¯ã©ã¹ã®ä»£è¡šãã¯ãã«ãå€æ§äœäžã«åäžã«åºãããããªå¶çŽãå ããæ倱é¢æ°ã§ãã Uniform Loss ãææ¡ã
æ¢åææ³ã§ã¯ã¯ã©ã¹éè·é¢ã倧ãããã¯ã©ã¹å
åæ£ãå°ãããããããªãã¹ãå€ã䜿ãããŠããããããã¯ç¹åŸŽç©ºéã®å
šäœãèæ
®ããŠããããè¯ããªãå±ææ§ãã¢ã³ãã©ã³ã¹ãªååžãçºçããã
å³ã¯ Fig. 1 ããã2D, 3D ãšãã«SphereFace ãšæ¯ã¹ãŠåäžãªååžãšãªã£ãŠããã
MegaFace Challenge ã YTF, LFW ãšãã£ãæ§ã ãªã¿ã¹ã¯ã§å®éšã«ããå¹æã瀺ããã
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
æ¢åç 究ã§ã¯çç®ãããŠããªãã£ããå€æ§äœäžã§ã®ã¯ã©ã¹å šäœã§ã®ååžã«çŠç¹ãåœãŠãã
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
åã¯ã©ã¹ã®ä»£è¡šãã¯ãã«ãåäœéã®é»è·ãšããŠèãããšãUniformLoss ã¯å šä»£è¡šãã¯ãã«ã®äœçœ®ãšãã«ã®ãŒãšèããããšãã§ããåé¡ãåã¯ã©ã¹åã®é»è·ã«ããäœçœ®ãšãã«ã®ãŒã®æå°ååé¡ãšå€ããã
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
MegaFace Challenge1, IJB-A, YTF, LFW ã®ããŒã¿ã»ããã§å®éšãå®éšæ°ã¯å€ãã¯ãªãããCosFace, ArcFace ãªã©ã®æ¢åææ³ãšçµåããŠäœ¿çšããŠãæ¢åã®æ§èœãæ¹åããŠããã
衚㯠Table 4 㧠MegaFace Challenge 1 ã§ã®æ¢åææ³ãšã®çµåãã®äŸã
ãã ããArcFace ã®æ¹ã CosFace ããæªãã£ãããCosFace ã®ã¹ã³ã¢ãå¥ã®è¡šãšåããªããªã©å°ãå€ãªç¹ãããã
5. è°è«ã¯ããïŒ
ä»åã¯ç䟡ãªé»è·ãèããããdata-dependent ãªé»è·ãäžããäžã§ã®ãã现ããè¡šçŸã®ååžãèããã®ã¯é¢çœãã
6. 次ã«èªãã¹ãè«æã¯ïŒ
ä»ãšãã¶ãã®ã§å²æ
[2019] Timeception for Complex Action Recognition
0. è«æ
ã¿ã€ãã«
Timeception for Complex Action Recognition
ãªã³ã¯
http://openaccess.thecvf.com/content_CVPR_2019/papers/Hussein_Timeception_for_Complex_Action_Recognition_CVPR_2019_paper.pdf
https://arxiv.org/abs/1812.01289
èè
Noureldien Hussein, Efstratios Gavves, Arnold W.M. Smeulders
æçš¿æ¥ä»
CVPR 2019
1. ã©ããªãã®ïŒ
ãããªå
ã®ããã€ãã®åçŽãª one-action ãè€æ°å«ã complex-action ãèªèããããã«ãæéæ¹åã® multi scale convolution ã§ãã Timeception ãææ¡ããã
æ¢åææ³ã§ã¯ 3D convolution ã䜿çšããŠãããé·ãæéæ¹åã®ã¢ããªã³ã°ã«ã¯èšç®ã³ã¹ããé«ããªã£ãŠããŸã£ãŠããããTimeception ã¯æ¢åææ³ã®8åã®é·ã(2014 timesteps) ã®ã¢ããªã³ã°ãå¯èœã
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
1024 timestep ãã®é·ãç³»åããPooling ãªã©ã§èµ·ããçãã¹ãã³ã®æ
å ±æ倱ãªãã«ã¢ããªã³ã°ãè¡ãææ³ã¯åã
ãŸããcomplex-action ã®æéçãªç¹åŸŽ (åã
ã® action ã®é·ããé çªãå€ãããã) ãèæ
®ããã¢ããªã³ã°ãè¡ã£ãŠããã
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
Charades, Breakfast Actions, MultiTHUMOS ãšãã£ããè€é㪠action ãå«ãé·ãã¬ã³ãžã®èªèã¿ã¹ã¯ã§ SOTA ãæŽæ°ã
5. è°è«ã¯ããïŒ
6. 次ã«èªãã¹ãè«æã¯ïŒ
[2019] 2.5D Visual Sound
0. è«æ
ã¿ã€ãã«
2.5D Visual Sound
ãªã³ã¯
http://openaccess.thecvf.com/content_CVPR_2019/papers/Gao_2.5D_Visual_Sound_CVPR_2019_paper.pdf
https://arxiv.org/abs/1812.04204
èè
Ruohan Gao, Kristen Grauman
æçš¿æ¥ä»
CVPR 2019
1. ã©ããªãã®ïŒ
ã¢ãã©ã«ãªé³å£°ããæ åãã空éçãªæ å ±ãåã蟌ãã§ããã€ããŒã©ã«ãªé³å£°ã«å€æãã 2.5D visual sound ãææ¡ããããŸããææ¡ææ³ã«ãã£ãŠåŸããããã€ããŒã©ã«ãªé³å£°ããèªå·±æåž«ããåŠç¿ã«ãã audio-visual source separation ã¿ã¹ã¯ã«å¹æçãªè¡šçŸãåŸãããŠããããšã瀺ãããåŠç¿ã«äœ¿çšãã 5.2æéåã®ãã€ããŒã©ã«ãªé³å£°ããŒã¿ãå ¬éãããŠããã
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
äžè¬çãªèŠçã®ãããªãçšããŠãã¢ãã©ã«ãªé³å£°ããã€ããŒã©ã«ãªé³å£°ã«å€æããè©Šã¿ã¯åã
audio-visual source separation task ã«å¯ŸããŠãã€ããŒã©ã«ãªé³å£°ç¹åŸŽãçšããã¢ãããŒãã¯åã
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
FAIR ã§åéãã FAIR-Play ãå§ããšãã4ã€ã®ããŒã¿ã»ããã§ãambisonics (360床æ åãšé³å£°) ã®ããã®ã¢ãã«ãšãææ¡ææ³ããå°ãæ
å ±ãåã£ãã¢ãã« (Audio-only, Flipped-Visual, Mono-Mono) ã§å®éšããautometric ã«ããè©äŸ¡ã human test ã«ããè©äŸ¡ãæãè¯ãçµæãšãªã£ãã
ãŸãã source separation ã¿ã¹ã¯ã«ã€ããŠããã€ããŒã©ã«ãªè¡šçŸãã¢ãã©ã«é³å£°ããåé¢ã«å¹æçã§ããããšã瀺ããã
5. è°è«ã¯ããïŒ
6. 次ã«èªãã¹ãè«æã¯ïŒ
[2019] Do Neural Dialog Systems Use the Conversation History Effectively? An Empirical Study
0. è«æ
ã¿ã€ãã«
Do Neural Dialog Systems Use the Conversation History Effectively? An Empirical Study
ãªã³ã¯
https://arxiv.org/abs/1906.01603
ã³ãŒã: https://github.com/chinnadhurai/ParlAI/
èè
Chinnadhurai Sankar, Sandeep Subramanian, Christopher Pal, Sarath Chandar, Yoshua Bengio
æçš¿æ¥ä»
ACL 2019
1. ã©ããªãã®ïŒ
察話ã¢ãã«ã«å¯Ÿããææã®1ã€ãšããŠãéå»ã®å¯Ÿè©±å±¥æŽãæå¹æŽ»çšã§ããŠããªããšãããã®ããããæ¬ç 究ã§ã¯ã察話履æŽã®äžŠã³æ¿ããåé€ãªã©ã®æäœãè¡ã察話ã¢ãã«ã®å±¥æŽã«å¯Ÿãã sensitivity ã調æ»ããããšã§ãã©ã®ããã«éå»ã®ã³ã³ããã¹ããç解ã»æŽ»çšããŠãããã調ã¹ãå®éšãè¡ã£ãã
察話履æŽããã¡ããã¡ãã«ããŠãã¢ãã«ã®è¿çãåãã«ãªã£ãããããã¯å¯Ÿè©±å±¥æŽãæå¹æŽ»çšã§ããŠãªãã? ãšãã話ã
çµæã LSTM ã Transformer ããŒã¹ã®äžè¬çãªã¢ãã«ã¯å¯Ÿè©±å±¥æŽã®å€åã«ããŸã sensitive ã§ãªã(察話履æŽã®æ
å ±ããããŸã§æŽ»çšããŠããªã) ããšãåãã£ãã
ç»åã¯è¡ã£ã Word-Level ã® Perturbation ã®äŸãæç« ã¯ãã¡ããã¡ãã ããè¿çã¯åããšãªã£ãŠããã
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
- 察話å¿ççæã®ã¢ãã«ã¯éå»çºè©±ã®ã³ã³ããã¹ããã¡ãããšæŽ»çšã§ããŠããªããšããææãå®éšçã«æ€èšŒããã
- èšèªã¢ãã«ãäœãåŠç¿ããŠããã®ããæ¡ä»¶ã« pertubation ãªã©ããã£ãå Žåã©ããªããããªã©ã®ç 究ã¯ãããã察話ã¢ãã«ã§ã®å¯Ÿè©±å±¥æŽã«çç®ãæ€èšŒããã®ã¯åã
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
- utterance ã word-level ã§ã® perturbation ãå ¥ããããŒã¹ã©ã€ã³ã¢ãã«ã§ãã¹ãæã®æåãèŠã
- ãã perturbation ãå ¥ãããšãã®åèªããšã® PPL ãèšæž¬ãã倧ãã PPL ãäžãã£ã (粟床ãäžãã£ã) å Žåã¯ãã¢ãã«ãã³ã³ããã¹ãã掻çšã§ããŠãã=è¯ãããšããããšã«ãªãã
- éã« PPL ãäžãããªã (粟床ãé«ããŸãŸ) ã®å Žåã¯ãã¢ãã«ãã³ã³ããã¹ããããŸãèŠãŠããªããšå€æã§ããã
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
4ã€ã® multi turn dialogue dataset 㧠LSTMããŒã¹ã Transformer ããŒã¹ã®ã¢ãã«ãããããåŠç¿ããäžèšã® perturbation ãå ããŠãã¹ãæšè«ãè¡ããPPL ã®å€åãèŠãããšã§ã³ã³ããã¹ãã®æŽ»çšåºŠåããèšæž¬ããã
çµæãLSTM ããŒã¹ã Transformer ããŒã¹ãã»ãšãã©å¯Ÿè©±å±¥æŽã掻çšã§ããŠãããããŸã LSTM ã®æ¹ã Transformer ãããã·ããšããçµè«ã«ãªã£ãŠããã
5. è°è«ã¯ããïŒ
ä»åã®ã³ãŒãã¯å ¬éããŠããã®ã§ãä»åŸã®äŒè©±ã¢ãã«ã®ç解ãé²ããšãããªãŒçãªããšãæžãããŠããã
6. 次ã«èªãã¹ãè«æã¯ïŒ
Related Work ã«ã¯åæ§ã« RNN ã Transformer ãå ·äœçã«ã©ããã£ãç¹ãåŠç¿ã§ããã»ã§ããŠããªããªã©ã«ã€ããŠãŸãšããè«æã玹ä»ãããŠããã
[2019] Incremental Transformer with Deliberation Decoder for Document Grounded Conversations
[2020] Low-Resource Knowledge-Grounded Dialogue Generation
0. è«æ
ã¿ã€ãã«
Low-Resource Knowledge-Grounded Dialogue Generation
ãªã³ã¯
https://arxiv.org/abs/2002.10348
èè
Xueliang Zhao, Wei Wu, Chongyang Tao, Can Xu, Dongyan Zhao, Rui Yan
æçš¿æ¥ä»
ICLR 2020
1. ã©ããªãã®ïŒ
ç¥èãçšãã察話ã·ã¹ãã ãå°ãªãããŒã¿ãªãœãŒã¹ã§å¹çããåŠç¿ããããã®ã¢ãã«ã»æ¹æ³ã®ææ¡ãã¢ãã«ã®ãã©ã¡ãŒã¿ãéåžžã®äŒè©±éšåãšç¥èã«ããäŒè©±éšåã«åããããããã®éšåã§åããŠäºååŠç¿ãè¡ããdisentangled ãããè¡šçŸã decoder ã§ããŸãæ±ãããšã§å¹çåã
Wizard of Wikipedia, CMU Document Grounded Conversation ã®2ã€ã§å®éšããæ¢åã¢ãã«ã«å¯Ÿã㊠1/8 çšåºŠã®ããŒã¿é㧠SoTA ã¹ã³ã¢ãæŽæ°ããåŠç¿ããŒã¿ãšã¯ç°ãªããã¡ã€ã³ã«å¯ŸããŠãå°ãªãããŒã¿ã§é©å¿ã§ããããšã瀺ããã
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
- å°ãªã knowledge-grounded ãªããŒã¿ãªãœãŒã¹ã§ç¥èãçšãã察話ã¢ãã«ã®åŠç¿ãè¡ãæ¹æ³ãææ¡ããã
- 察話ã®è¡šçŸã»ç¥èã®è¡šçŸãããŸã disentangled ãã decoder ãææ¡ãããããããå¥ã« pre-train ããããšã§ç¥èãçšãã察話ã®ç²ŸåºŠãäžããæ¹æ³ãææ¡ããã
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
- 次ã®ããã«ã³ã³ããŒãã³ããåããŠã¢ãã«ãäœæãã³ã³ããŒãã³ã GRU + attn ããŒã¹, decoder ã¯æŽã« copy mechanism ãå
¥ãããã®ã
- encoder
- encoding context: äŒè©±å±¥æŽã GRU encode
- encoding knowledge: å€éšç¥èã BiGRU encode
- decoder
- language model: èšèªã¢ãã«
- context processor: GRU + attention + copy mechanism
- knowledge processor: GRU + æã¬ãã«ã® attention + åèªã¬ãã« attention 㧠copy
- decoder manager: äžèš3ã€ã®åºåãã次ã®åèªäºæž¬, gumbel softmax (å¿ èŠã?)
- encoder
- ãã©ã¡ãŒã¿ãåãã€ã€ pre-training ã training ãè¡ã
- ã³ã³ããŒãã³ãããšã®ãã©ã¡ãŒã¿ãåã㊠pre-training
- context encoder, decoder, context processor ã察話ããŒã¿ (Reddit) ã§åŠç¿
- language model ã察話ããŒã¿ (Reddit) ã®ããããã®æã䜿ã£ãŠåŠç¿, ãã®æ decoder ã®ãã©ã¡ãŒã¿ã¯ fixed
- knowledge encoder ãç¥èããŒã¿ (wikipedia) ã䜿ã£ãŠåŠç¿
- æ®ãã®ãã©ã¡ãŒã¿ (knowledge processor, decoding manager) ã¯ç¥èä»ãã®å¯Ÿè©±ããŒã¿ (Wizard of Wikipedia, CMU_DoG)ã§åŠç¿, ãã®æä»ã®å šãŠã®ãã©ã¡ãŒã¿ã¯ fixed
- ã³ã³ããŒãã³ãããšã®ãã©ã¡ãŒã¿ãåã㊠pre-training
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
Wizard of Wikipedia, CMU_DoG ããŒã¿ã»ããããæ¯èŒã« Tranformer Memory Network (Wizard of Wikipedia ã®ããŒã¹ã©ã€ã³) ãš Incremental Transformer with Deliberation Decoder (ITDD) ã䜿çšããŠå®éšãææ¡ææ³ã«ã€ããŠã¯ low-resource ã§ã®å¹æãèŠãããã«ãããŒã¿ã»ããã full, 1/2, 1/4, 1/8, 1/16 ã«ããŠåŠç¿ãè¡ã£ãã
çµæã¯ãããããã圢ã«ãªã£ãŠãããTraining ãšåããã¡ã€ã³ã®ããŒã¿ (Test Seen) ã«ã€ããŠã¯ ITDD ãæãè¯ãã¹ã³ã¢ã ããTest Unseen (Wizard of Wikipedia 㯠test ããŒã¿ã Test Seen ãš Test Unseen ã«åãããŠãã) ã«ã€ããŠã¯ ITDD ã®æ§èœã¯å€§ããèœã¡ãŠããã®ã«å¯ŸããŠãææ¡ææ³ã¯ã»ãšãã© Test Seen ãšå€ãããªãããã€ãããªã«æªããªãã¹ã³ã¢ã«ãªã£ãŠããã
5. è°è«ã¯ããïŒ
次ã®çåã«ã€ããŠè«æå ã§æ瀺ã»å®éšãè¡ã£ãŠããã
- Q1: åã³ã³ããŒãã³ãã pre-train ããã¢ãã«ã« knowledge-grounded ãªããŒã¿ã§ finetune ãããšããpre-train éšåã fix ãã/ããªãã©ã£ã¡ããã?
- å®éšãããšãããããŒã¿ãå€ããšãã¯ããªãã»ããè¯ãããããŒã¿ãå°ãªããš finetune æã®ããŒã¿ã« overfitting ããã®ã§ããªãã»ããè¯ãã
- Q2: Transformer Memory Network ã«ãåæ§ã« knowledge ãå«ãŸãªã察話ããŒã¿ (Reddit) 㧠pre-train ãããã©ããªã? ( knowledge encode ãš decoder ã®ãã©ã¡ãŒã¿ã finetune)
- å®éšãããšãããäŸç¶ãšããŠææ¡ææ³ã®æ¹ãé«ãã¹ã³ã¢ã ã£ããæ瀺çã«ãã©ã¡ãŒã¿ã disentangled ããã®ãå¹ããŠãããšèããããã
- Q3: component ããããã« pre-train ãããŠããããã©ããå¹ããŠãã?
- ablation study ãå®æœããŠãå šãŠå¹ããŠããããšã確èªã
6. 次ã«èªãã¹ãè«æã¯ïŒ
- Wizard of Wikipedia: Knowledge-Powered Conversational Agents
- https://qiita.com/cfiken/items/97c87e6162fbd4a806bc
- ããŒã¹ã©ã€ã³ã¢ãã« (TMN) & ããŒã¿ã»ãã (Wizard of Wikipedia)
- Incremental Transformer with Deliberation Decoder for Document Grounded Conversations
- nlp-survey-kb-response/nlp-survey-kb-response#41
- ããŒã¹ã©ã€ã³ã¢ãã« ITDD
- A Dataset for Document Grounded Conversations
- https://arxiv.org/abs/1809.07358
- ããŒã¿ã»ãã (CMU_DoG)
[2019] Towards Knowledge-Based Recommender Dialog System
0. è«æ
ã¿ã€ãã«
Towards Knowledge-Based Recommender Dialog System
ãªã³ã¯
https://arxiv.org/abs/1908.05391
èè
Qibin Chen, Junyang Lin, Yichang Zhang, Ming Ding, Yukuo Cen, Hongxia Yang, Jie Tang
æçš¿æ¥ä»
EMNLP 2019
1. ã©ããªãã®ïŒ
察話ããŒã¹ã®ã¬ã³ã¡ã³ãã·ã¹ãã ã«ã€ããŠãã¬ã³ã¡ã³ãã·ã¹ãã ãšå¯Ÿè©±ã·ã¹ãã ã end-to-end ã«çµã¿åãããããšã§ããããã®æ§èœãåºäžããã KBRD (Knowledge-Based Recommender Dialog) ãææ¡ããã
äžèšå³ã®ããã«ãåçŽã«å¯Ÿè©±ã·ã¹ãã ãšã¬ã³ã¡ã³ãã·ã¹ãã ã®åºåãåãæ¿ããã ãã§ã¯ãªãããäºãã®æ
å ±ãããŸã掻çšããããšã§ããããã®ç²ŸåºŠãæ¹åããã
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
end-to-end ã«ã¬ã³ã¡ã³ãã·ã¹ãã ãšå¯Ÿè©±ã·ã¹ãã ãäºãã®æ å ±ã掻ããã€ã€çµã¿åãããææ³ãææ¡ããã
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
次ã®æé ã§ãŠãŒã¶ã®è¡šçŸãåŸãã
- 察話ã®äžãã entity linking ã«ãã item ãš non-item ãåããªã entity ã®ãªã¹ããäœæãã
- knowledge graph ãš GCN ãçšããŠå entity ããšã³ã³ãŒããã
- self-attention ãé©çšããŠåºå®æ¬¡å
ã®è¡šçŸã«å€æããããããŠãŒã¶ã®è¡šçŸãšãã
- ããã (item ã® entity ã«ã®ã¿) softmax ãåãããšã§ã¬ã³ã¡ã³ãã®ç¢ºçååžãšãªã
次ãã§ãã¬ã³ã¡ã³ãã·ã¹ãã ã§åŸãããæ å ±ã察話ã·ã¹ãã ã«çµã¿èŸŒãã
- ã¬ã³ã¡ã³ãã·ã¹ãã ã§åŸããããŠãŒã¶è¡šçŸãèªåœã®æ¬¡å ã«å€æãã
- Transformer ã® decoder ã®æåŸã®åé¡ã¬ã€ã€ã« bias ãšããŠå€æããè¡šçŸã足ã
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
DIALog (REDIAL) ãšãã察話ã«ããã¬ã³ã¡ã³ããŒã·ã§ã³ã®ããŒã¿ã»ããã䜿ã£ãŠæ€èšŒã
ã¬ã³ã¡ã³ã粟床ã察話è©äŸ¡ãšãã«ææ¡ææ³ãããŒã¹ã©ã€ã³ã¢ãã«ãäžåã£ããablation study ãè¡ããã¢ãã«ã®å€å(HRED -> Transformer)ãå€éšç¥èã®å°å
¥ã察話æ
å ±ã®çµã¿åããã®ã©ããæ¹åã«è²¢ç®ããŠããããšãããã£ãã
ç¹ã«å¯Ÿè©±ã«ãããŠã¯ãTransformer ã PPL ã«å€§ããè²¢ç®ããä»ã®ææ³ã«ããã¢ãã«åºåã®å€æ§æ§ã«å€§ããæ¹åãèŠãããã
5. è°è«ã¯ããïŒ
åæã«ããã察話å ã§èšåãããæ ç»(item)ã®æ°ãå°ãªãå Žåã¯å¯Ÿè©±ãåãå ¥ããææ³ã§å€§ããæ¹åã§ãã察話å ã§èšåãããæ ç»ãå€ãå Žåã¯å€éšç¥èãåãå ¥ããææ³ã§å€§ããæ¹åã§ããããšãããã£ãã
6. 次ã«èªãã¹ãè«æã¯ïŒ
- Extreme Adaptation for Personalized Neural Machine Translation
- Paul Michel, Graham Neubig, ACL 2018
- bias ã«ä»ã®æ å ±èŒããæèã§åºãŠãã
[2019] Boosting Dialog Response Generation
0. è«æ
ã¿ã€ãã«
Boosting Dialog Response Generation
ãªã³ã¯
https://www.aclweb.org/anthology/P19-1005/
èè
Wenchao Du, Alan W Black
æçš¿æ¥ä»
ACL 2019
1. ã©ããªãã®ïŒ
察話ã¢ãã«ã®å€æ§æ§ã®äœãã Boosting ãçšããããšã§æ¹åããã¢ãããŒãã®ææ¡ã
ããã€ãã®ãã¬ãŒãã³ã°æ¹æ³ã decoding æ¹æ³ (MMI: Maximum Mutual Information, RAML: Reward-augmented maximum likelihood learning) ã§ããŒã¹ã¢ãã«ãäœæããBoosting ã«çµã¿åãããŠåŠç¿ããã¢ãã«ã®æ¯èŒã»å®éšãè¡ã£ãã
çµæãå®éçã»å®æ§çã« Boosting ã¢ãã«ã«ãã粟床ãèœãšããã«å€æ§æ§åäžãèŠãããã
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
- Boosting ãçšããŠå¯Ÿè©±ã®å€æ§æ§ãšåŠ¥åœæ§ããããè©Šã¿ã¯åã
- 察話ã®ããã« RAML ã boosting ã§åŠç¿ãè¡ã£ãã®ã¯åã
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
MMI ã RAML ãšãã£ã察話ã¢ãã«ã®å€æ§æ§ãäžããããã®ã¢ãããŒããšãBoosting ãçµåããŠåŠç¿ãè¡ã£ãç¹ã
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
ããã€ãã®æ¹æ³ã§åŠç¿ããããŒã¹ã¢ãã«ãšããã® Boosting çãæ¯èŒå®éšãã察話ã®ç²ŸåºŠãããŸãèœãšããªããŸãŸå€æ§æ§ãäžãã£ãããšã auto metrics ã«ããè©äŸ¡ãšäººæè©äŸ¡ã«ãã瀺ããã
5. è°è«ã¯ããïŒ
ç¹ã«ãªã
6. 次ã«èªãã¹ãè«æã¯ïŒ
- Boosted Generative Models
- https://arxiv.org/abs/1702.08484
- Aditya Grover, Stefano Ermon
- AAAI 2018
- å ã®ã¢ã€ãã£ã¢ãšããŠé »ç¹ã«åŒçšãããŠãã
- A Diversity-Promoting Objective Function for Neural Conversation Models
- https://arxiv.org/abs/1510.03055
- Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, Bill Dolan
- NAACL 2016
- MMI ã®ææ¡è«æ
- Reward Augmented Maximum Likelihood for Neural Structured Prediction
- https://arxiv.org/abs/1609.00150
- Mohammad Norouzi, Samy Bengio, Zhifeng Chen, Navdeep Jaitly, Mike Schuster, Yonghui Wu, Dale Schuurmans
- NIPS 2016
- RAML ã®ææ¡è«æ
[2019] AdaptiveFace: Adaptive Margin and Sampling for Face Recognition
0. è«æ
ã¿ã€ãã«
AdaptiveFace: Adaptive Margin and Sampling for Face Recognition
ãªã³ã¯
http://openaccess.thecvf.com/content_CVPR_2019/html/Liu_AdaptiveFace_Adaptive_Margin_and_Sampling_for_Face_Recognition_CVPR_2019_paper.html
https://github.com/haoliu1994/AdaptiveFace ã³ãŒãå
¬éäºå®ãšã®ããšã ããŸã ãªã (2019/09/07)
èè
Hao Liu, Xiangyu Zhu, Zhen Lei, Stan Z. Li
æçš¿æ¥ä»
CVPR 2019
1. ã©ããªãã®ïŒ
ããŒãžã³ä» Softmax Loss ã®ææ¡ã«ããé¡èªèã¿ã¹ã¯ã®æ§èœã¯å€§ããæ¹åãããããåã¯ã©ã¹ã®ååžãè¡šçŸãããµã³ãã«æ°ãååã«ãããšããåæããã£ãã
ããŒãžã³ã¯ã¯ã©ã¹ã«ãã£ãŠé©å¿çã«æ±ºããããã¹ããšããèãæ¹ã§ãAdaptive Margin Softmax ãšããæ°ããæ倱é¢æ°ãææ¡ã
ãŸãåŠç¿æã®ãµã³ããªã³ã°ã«ã Hard Prototype Mining ãš Adaptive Data Sampling ãå°å
¥ãããããããŸãšã㊠AdaptiveFace ãšããŠææ¡ãã(詳现åŸè¿°)ã
LFW, LFWBLUFR, MegaFace ã§å®éšããSOTA ã®ã¹ã³ã¢ãæŽæ°ããã
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
åã¯ã©ã¹ã«å
¥ããããŒãžã³ã learnable parameters ãšãã softmax loss ãææ¡ããã
æ倱é¢æ°ã ãã§ãªããé©å¿çãªãµã³ããªã³ã°ã hard prototype mining ãçµã¿åãããææ³ãææ¡ããã
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
- Adaptive Margin Softmax
- Hard Prototype Mining
- ã¡ãã£ãšäºåç¥èããªããŠåãããªãã£ãã®ã§åŸ©è®ããã
- Adaptive Data Sampling
- åé¡ã¬ã€ã€ããããŒã¿ã¬ã€ã€ã« feedback layer ãã€ãªãã§ãããŒã¿ã®ãµã³ããªã³ã°ã®ç¢ºçã«æãå ¥ããã
- åŠç¿äžã«æ£ããåé¡ãããå Žåããµã³ããªã³ã°ç¢ºçãäžããã
- éã®å Žåã¯ãµã³ããªã³ã°ç¢ºçãäžããã
- ãã€ãºããŒã¿å¯Ÿçã®ãããäžå®ã®ãããå€ãè¶ ããå·®ããã£ãå Žåã¯ãµã³ããªã³ã°ç¢ºçã倧ããäžããã
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
MegaFace Challenge 1, LFW, LFW BLUFR ã§å®éšãããŒã¹ã©ã€ã³ã¯ SoftMax, SphereFace, CosFace, ArcFaceã
ãã¹ãŠã§ AdaptiveFace ãæãé«ãã¹ã³ã¢ãç²åŸã
衚㯠MegaFace Challenge 1 ã®ã±ãŒã¹ã
Ablation study ãè¡ãã CosFace ã®ããŒã¹ã©ã€ã³ã«å¯ŸããŠããããã®ææ³ã ãã§ãå¹æãããããšã瀺ããŠããã
5. è°è«ã¯ããïŒ
6. 次ã«èªãã¹ãè«æã¯ïŒ
- SphereFace: Deep Hypersphere Embedding for Face Recognition
- https://arxiv.org/abs/1704.08063
- Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, Le Song
- CVPR 2017
- SphereFace ææ¡è«æ
- CosFace: Large Margin Cosine Loss for Deep Face Recognition
- https://arxiv.org/abs/1801.09414
- Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, Wei Liu
- CVPR 2018
- CosFace ææ¡è«æãã®1
- Additive Margin Softmax for Face Verification
- https://arxiv.org/abs/1801.05599
- Feng Wang, Weiyang Liu, Haijun Liu, Jian Cheng
- CosFace ææ¡è«æãã®2
- ArcFace: Additive Angular Margin Loss for Deep Face Recognition
- #20
- ArcFace ææ¡è«æ
[2019] Structured Binary Neural Networks for Accurate Image Classification and Semantic Segmentation
0. è«æ
ã¿ã€ãã«
Structured Binary Neural Networks for Accurate Image Classification and
Semantic Segmentation
https://arxiv.org/abs/1811.10413
ãªã³ã¯
èè
Bohan Zhuang, Chunhua Shen, Mingkui Tan, Lingqiao Liu, Ian Reid
æçš¿æ¥ä»
CVPR 2019
1. ã©ããªãã®ïŒ
ç»ååé¡ãªã©ã® CNN ã®ã¢ãã«ã«ãããéã¿ãš activation ã®äž¡æ¹ã binary ã§è¡šçŸãã Binary NNs ã«ãããŠãæ¢åææ³ã§ããå
ã®(ãã«ç²ŸåºŠã®)ã¢ãã«ã®éã¿ã activation ã®å€ãè¿äŒŒãã value approximation ã§ã¯ãªããå
ã®ã¢ãã«ã®ãã£ãã·ãã£ãåçŸãããã㪠binary ã®ã¢ãã«ãèšèšãã structure approximation ã«ããã¢ãããŒããææ¡ããã
ç»ååé¡ã ãã§ãªããsemantic segmentation ãªã©ã«ãæ¡åŒµã§ããã
æ¢åææ³ãšæ¯ã¹ãŠãaccuracy ãšèšç®éã®ãã¬ãŒããªã㧠SOTA.
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
ä»ãŸã§ã® binary NNs ã®å®çŸææ³ã«ãããŠãvalue approximation ã§ã¯ãªããå
ã®ã¢ãã«ãåçŸã§ããããã«æ§é ããå€ãã structure approximation ã®ã¢ãããŒãã¯åã
ãŸããsemantic segmentation ãžã®å¿çšãšããŠãå解ãããããã¯ãŒã¯ã§ç°ãªã dilated rate ã䜿ã Binary Parallel Atrous Convolution ãææ¡ã
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
å SOTA ã ã£ã ABC-Net ãå«ãæ¢åææ³ãš ImageNet ã§æ¯èŒããbinary, fixed-point ã®äž¡æ¹ã§ SOTA ãæŽæ°ã
ãŸãã semantic segmentation ã¿ã¹ã¯ã§ãã PASCAL VOC ã§ãæ¢åææ³ãšæ¯èŒãè¯ãã¹ã³ã¢ãšãªã£ãã
5. è°è«ã¯ããïŒ
6. 次ã«èªãã¹ãè«æã¯ïŒ
[2019] Ranking and Sampling in Open-Domain Question Answering
0. è«æ
ã¿ã€ãã«
Ranking and Sampling in Open-Domain Question Answering
ãªã³ã¯
https://www.aclweb.org/anthology/D19-1245/
èè
Yanfu Xu, Zheng Lin, Yuanxin Liu, Rui Liu, Weiping Wang, Dan Meng
æçš¿æ¥ä»
EMNLP IJCNLP 2019
1. ã©ããªãã®ïŒ
Open-domain Question Answering ã¯è³ªåã«ããã€ãã®ãã©ã°ã©ãããçããäœæããããæ¢åææ³ã®å€ãã¯è³ªåãšãã©ã°ã©ãã®é¢ä¿ã«ããæåž«ããåŠç¿ãçšããŠããã
æ¬ç 究ã§ã¯ãå€ãã®åè£ãšãªããã©ã°ã©ãã®äžã«ã¯ noisy (åçã®åèªã¯å«ãŸãããåçã¯å°ããªã)ã distracting (åçã®åèªãå«ãŸããªã)ãå«ãŸãããšãã ãã©ã°ã©ããšãã©ã°ã©ãã®é¢ä¿ã掻çšããŠæçšãªãã©ã°ã©ããèŠã€ããããã®ã©ã³ãã³ã°ã¢ãã«ãããããçšããåŠç¿æã® sampling strategy ãææ¡ããã
Quasar-T, SearchQA, TriviaQA ã®ããŒã¿ã»ããã§SoTAãæŽã«æ¹åã
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
è€æ°ãã©ã°ã©ãã®åè£ã®äžããåçãèŠã€ãããã㪠QA ã¿ã¹ã¯ã«ã€ããŠã質å-ãã©ã°ã©ãã®é¢ä¿ã ãã§ã¯ãªããã©ã°ã©ãéã®é¢ä¿ã«çç®ãããã©ã°ã©ãã®ã¹ã³ã¢ã䜿ã£ã sampling strategy ãææ¡ããã
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
bidirectional LSTM ãš attention ãçšããŠåãã©ã°ã©ãã®ã¹ã³ã¢ãèšç®ããã
èšç®ããã¹ã³ã¢ãçšããŠåŠç¿æã® sampling strategy ãæ¹è¯ãããå ·äœçã«ã¯ãã¹ã³ã¢ã®é«ã k åã®ãã©ã°ã©ãã®ã¿åŠç¿ã«äœ¿ã RK (Ranking) ããã¹ã³ã¢ãçšã㊠sanpling ãã WS (Weighted Sampling) ãªã©ãçµã¿åãããŠããã
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
Quasar-T, SearchQA, TriviaQA ã®ããŒã¿ã»ããã䜿ã£ãŠãå€ãã®ããŒã¹ã©ã€ã³ã¢ãã«ãšæ¯èŒããæãè¯ãçµæãšãªã£ãã
è€æ°ã®å®éšãšåæãè¡ããranker, sampling strategy ãšãã«æ§èœã«è²¢ç®ããŠããããšã瀺ããã
5. è°è«ã¯ããïŒ
6. 次ã«èªãã¹ãè«æã¯ïŒ
[2019] Effective Adversarial Regularization for Neural Machine Translation
0. è«æ
ã¿ã€ãã«
Effective Adversarial Regularization for Neural Machine Translation
ãªã³ã¯
https://www.aclweb.org/anthology/P19-1020
http://sato-motoki.com/pdf/acl2019.pdf
https://github.com/pfnet-research/vat_nmt
èè
Motoki Sato, Jun Suzuki, Shun Kiyono
æçš¿æ¥ä»
ACL 2019
1. ã©ããªãã®ïŒ
Neural Machine Translation (NMT) ã«ãããŠãå token ã® embedding ã«å¯Ÿã㊠loss ã倧ãããªãæ¹åã«æåãä»å ãã Adversarial Regularization ã®å¹æãã¡ãžã£ãŒãªã¢ãã«ã§å®éšããæ§èœãæ¹åãããããšã瀺ããã
LSTM ãš Transformer ã§å®éšãè¡ããéåžžã® Adversarial Regularization ã ãã§ãªã VAT ã䜿ã£ããã®ããBack-Translation ã«ããããŒã¿æ¡åŒµãçµã¿åãããææ³ã§ãå®éšããå
šãŠã§æ¹åã確èªããŠããã
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
ããã¹ãåé¡ã«ãã㊠Adversarial Regularization ã®å¹æãèŠãç 究ã¯ãã£ãããNMT ã®ãã㪠enc-dec ã¢ãã«ã«ãããŠæ§ã ãªæ¯èŒãè¡ã£ãã®ã¯åã
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
adversarial 㪠perturbation ã (1) encoder only, (2) decoder only, (3) both ãšããæ¯èŒããLSTM ããŒã¹ãš Transformer ããŒã¹ã®ã¢ãã«ã®æ¯èŒãVAT ãšã®æ¯èŒãåã³ Back-Translationãå«ãã æ¯èŒãªã©ãå€å²ã«ããã£ãŠæ€èšŒãè¡ã£ãã
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
åäžã
5. è°è«ã¯ããïŒ
ç¹ã«ãªã
IMO: NMT 以å€ã® enc-dec ã«ãå¹æããããã¯æ°ã«ãªããšããã
6. 次ã«èªãã¹ãè«æã¯ïŒ
- Adversarial Training Methods for Semi-Supervised Text Classification
- Takeru Miyato, Andrew M. Dai, Ian Goodfellow
- https://arxiv.org/abs/1605.07725
- ICLR 2017
- text classification ã¿ã¹ã¯ã§ã® adversarial regularization ã®å¹ææ€èšŒãè¡ã£ãç 究
- Distributional Smoothing with Virtual Adversarial Training
- Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, Ken Nakae, Shin Ishii
- https://arxiv.org/abs/1507.00677
- ICLR 2016
- VAT
[2015] From Word Embeddings To Document Distances
0. è«æ
ã¿ã€ãã«
From Word Embeddings To Document Distances
ãªã³ã¯
http://mkusner.github.io/publications/WMD.pdf
èè
Kusner, M. J. and Sun, Y. and Kolkin, N. I. and Weinberger, K. Q.
æçš¿æ¥ä»
ICML 2015
1. ã©ããªãã®ïŒ
Earth Mover's Distance ãããã¹ãã«å¿çšãã Word Mover's Distance ã®ææ¡ãdocument ã®é¡äŒŒåºŠã枬ãææ³ããŸã è¯ããã®ããªããšããŠãword2vec 空éã§ã®2ã€ã®ææžéã®è·é¢ãããããããäžèŽãããããã«å¿
èŠãªæå°ã®ç§»åéãšããŠå®çŸ©(Earth Mover's Distance)ããWord Mover's Distance ãšããã
8ã€ã®ã¿ã¹ã¯ã§ã§æ§ã
ãªä»ã®ææšãšæ¯èŒããkNN ã«ããåé¡ã§æãäœã誀差çãšãªã£ãã
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
2ã€ã®ææž d_0, d_1 ã®é¡äŒŒåºŠãèãããšã次ã®ãããªæç« ã¯æå³ã¯ã»ãŒåãã ãåãåèªã¯å«ãŸããªããã BOW ç³»ã®ç¹åŸŽéã ãšé¡äŒŒåºŠã¯ãŒãã«ãªã£ãŠããŸãã
d_0: Obama speaks to the media in Illinois
d_1: The President greets the press in Chicago
WMD ã§ã¯ãããããã®åèªå士㮠word2vec 空éã§ã®è·é¢ããèãããè·é¢ã¯ãŠãŒã¯ãªããè·é¢ã䜿çšããåèª x_i ãšåèª x_j ã®è·é¢ã¯ ã§èšç®ã§ããã
æçµçã«ç¥ãããã®ã¯ææžéã®è·é¢(é¡äŒŒåºŠ)ãªã®ã§ããã® c_{i, j} ãåèª x_i ããåèª x_j ãžã®ç§»éã³ã¹ããšåŒã¶ã
ææžéã®æå°ç§»éè·é¢ã¯ã移éã³ã¹ããšãã©ã®åèªãã©ã®åèªã«ç§»éãããããå®çŸ©ãã flow matrix T ã䜿ã£ãŠã次ã®ããã«å®åŒåã§ããã
d_0 ã®ååèªããäžçªç§»åãå°ãªããªãããã« d_1 ã«å¯Ÿå¿ãããã€ã¡ãŒãžãTã«ã€ããŠã®å¶çŽä»ãæé©ååé¡ã解ãããšã«ãªãã
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
5. è°è«ã¯ããïŒ
6. 次ã«èªãã¹ãè«æã¯ïŒ
[2019] Self-Supervised Dialogue Learning
0. è«æ
ã¿ã€ãã«
Self-Supervised Dialogue Learning
ãªã³ã¯
https://arxiv.org/abs/1907.00448
èè
Jiawei Wu, Xin Wang, William Yang Wang
æçš¿æ¥ä»
30 Jun 2019
ACL 2019
1. ã©ããªãã®ïŒ
察話ã«ãããŠçºè©±ã®é åºãéèŠãšããèå¯ãã inconsistent order detection ãšããã¿ã¹ã¯ã解ã self-supervised ãªåŠç¿ææ³ãšãã®ããã®ã¢ãã«ã§ãã Self-Supervised Network (SSN) ãææ¡ã
éåžžã®å¯Ÿè©±ã¢ãã«åŠç¿ãš SSN ã«ããåŠç¿ã GAN ã®ä»çµã¿ãçšããŠçµã¿åãããããšã§ããäžè²«ããåŠç¿ãã§ããããã«ããOpen-Subtitles ãš Movie-Ticket Booking ã®ã¿ã¹ã¯ã§ SOTA ãæŽæ°ããã
éåžžã®å¯Ÿè©±ã¢ãã«ã Generator ãšããŠãSSN ããããçææãè¯ãè¿çãªãé åºã¿ã¹ã¯ã解ããããããšãã Discriminator ãšããŠäœ¿ãããšã§çµã¿åãããŠããã
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
察話ã¢ãã«ã®åŠç¿ã«ãããŠã察話ã®é åºã«çç®ããããã¿ã¹ã¯ãšããŠåãå
¥ããç 究ã¯åã
ãŸããã®é åºèå¥ã¿ã¹ã¯ãäžæã Adversarial Learning ã«åãå
¥ããã
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
- Inconsistent Order Detection
- 察話ã»ããããã©ã³ãã ã§ãµã³ããªã³ã°ãã3ã€ã®çºè©±ã»å¿çã»ããã®é åºãé転ããŠãããã©ãããäºæž¬ããã¿ã¹ã¯
- Self-Supervised Network SSN
- taget pair ãããéå»ã®å¯Ÿè©±ãã 2ã€ã® triple ã§ãã T' ãš T'' ããµã³ããªã³ã°
- T', T'' 㯠Ordered / Misordered ã®çµã¿åãããå€ããŠå®éš
- target T ã ordered/misordered ãã©ããã T', T'' ã®æ¡ä»¶ã§ã®ç¢ºçãšãªãããã« SSN ãèšèš
P(T|T', T'') = S(T, T', T'')
- ã¢ãã«ã¯ LSTM ããŒã¹
- taget pair ãããéå»ã®å¯Ÿè©±ãã 2ã€ã® triple ã§ãã T' ãš T'' ããµã³ããªã³ã°
- SSN + Dialogue
- 察話ã«ã¯é åºã®åœ±é¿ã倧ããã®ã§ãçææ A_t ãè¯ããã° inconsistent order ã®å€æã¯ç°¡åã«ãªã
- ããã§ãªãå Žåã¯ãçæãã A_t ã埮åŠ
- ãããéã«äœ¿ã£ãŠãmisordered 㪠triple ãäžããããæ A_t ã§ã©ã®ãããæ£ããèå¥ã§ããããèŠã
- ããã«ããã A_t ãã©ããããè¯ãããè©äŸ¡ãã (ç°¡åã«ãããã° A_t ã¯è¯ãå¿ç)
- ããã Discriminator ãšããŠäœ¿ã£ãŠãGenerator ã Adversarial ã«åŠç¿ãã
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
OpenSubtitles ãš Movie-Ticket Booking ã®ããŒã¿ã»ããã§å®éšãããŒã¹ã©ã€ã³ã¢ãã«ãšã㊠adversarial ãªåŠç¿ãçšããã¢ãã«ã§ãã REGS (Li et al., 2017) ãšAEL (Xu et al., 2017) ãšæ¯èŒã
æ§ã
ãªå®éšãè¡ããã©ã®ææšã§ãææ¡ææ³ãäžåã£ãã
ç¹ã«äººéè©äŸ¡ã§ã¯å€§ããªå·®ããã£ãã
5. è°è«ã¯ããïŒ
- åãæ çµã¿ãä»ã® NLP ã¿ã¹ã¯ã«ãé©çšã§ãããã©ãã
6. 次ã«èªãã¹ãè«æã¯ïŒ
- Adversarial Learning for Neural Dialogue Generation
- https://arxiv.org/abs/1701.06547
- Jiwei Li, Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter, Dan Jurafsky
- REGS ã®ã¢ãã«
- Neural Response Generation via GAN with an Approximate Embedding Layer
- https://www.aclweb.org/anthology/D17-1065
- Zhen Xu, Bingquan Liu, Baoxun Wang, Chengjie Sun, Xiaolong Wang, Zhuoran Wang, Chao Qi
- AEL ã®ã¢ãã«
- Neural Approaches to Conversational AI
- https://arxiv.org/abs/1809.08267
- Jianfeng Gao, Michel Galley, Lihong Li
[2019] Compressive Transformers for Long-Range Sequence Modelling
0. è«æ
ã¿ã€ãã«
Compressive Transformers for Long-Range Sequence Modelling
ãªã³ã¯
https://arxiv.org/abs/1911.05507
èè
Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Timothy P. Lillicrap
æçš¿æ¥ä»
ICLR2020 (Submitted on 13 Nov 2019)
1. ã©ããªãã®ïŒ
TransformerXL ã®ã¢ã€ãã£ã¢ãå
ã«ããŠãããé·æéã®ç³»åãã¡ã¢ãªã«ä¿åã»æŽ»çšã§ãã Compressive Transformers ãææ¡ã
TransformerXL ã§ã¯éå»ã®ç³»åã® activation ãã¡ã¢ãªãšããŠä¿åããŠããããã¡ã¢ãªãµã€ãº n 以äžã®éå»ã®ç³»åã¯æšãŠãããšã«ãªããCompressive Transformer ã§ã¯ãFig.1 ã®ããã«æšãŠãããã¡ã¢ãªãå§çž®ã㊠Compressive Memory ãšããŠä¿åããŠããããšã§ãããé·æéã®æ
å ±ã掻çšã§ããã
WikiText-103 ãš Enwik8 ã§
é·ãç³»åã®èšèªã¢ãã«ã®ããã® PG-19 ãšãã book based ãªãã³ãããŒã¯ãäœæããã
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
å
è¡ç 究ã§ã¯ attention ã®ç¯å²ãåºãã工倫ããattention ã®æŒç®ã³ã¹ããäžããç 究ãå€ãã
æ¬ç 究ã§ã¯å€ãã¡ã¢ãªãå§çž®ããŠä¿åããããšã§ activation ã®éã¯å°ãªããã€ã€é·æã®ã¡ã¢ãªãä¿åããããšãã§ããã
ãŸããã¹ããŒãã®èªèã匷ååŠç¿ã«ãããã¡ã¢ãªæ§é ã«ãé©çšããå¹çããé·æã®ã¡ã¢ãªãä¿æã§ããŠããããšã瀺ããã
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
- å€ãã¡ã¢ãªãå§çž®ããŠæã€ãšããã·ã³ãã«ãªæ¹æ³ã§ attention èªäœã®ã¢ã«ãŽãªãºã ã®ä¿®æ£å¿ èŠãªãã«ãé·æéã®ã¡ã¢ãªä¿åãå¯èœã«ããã
- å€ãã¡ã¢ãªã®ãŠããã n ååã n/c åã®ãŠãããã«å§çž®ããŠä¿åãã(äžèšã§å®çŸ©ãããé¢æ°ãé©çš)ã
- ãã©ã¡ãŒã¿ãæã€å§çž®é¢æ°ã«ã€ããŠã¯ãauxiliary loss ãå®çŸ©ããŠåŠç¿ããã
- (1) auto-encoding: å§çž®ååŸã®ã¡ã¢ãªã®åçŸ loss
- (2) attention-reconstruction loss: å§çž®ååŸã®å
¥å(ã®é ãå±€)ãšã® attention ã®åçŸ loss
- ãããäžçªè¯ãã£ã
- TransformerXL ã«æ¯ã¹ãŠ c = 3 ã®ãšãåãèšç®éã§äºåã®ç³»åãèŠãããšãã§ãã
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
å®éšã¯ææ¡ãããã³ãããŒã¯ã§ãã PG-19 ã«å ããŠãEnwik8, WikiText-103 ã§å®éšãããããã§ããŒã¹ã©ã€ã³ã§ãã TransformerXL ã®ã¹ã³ã¢ãæŽã«æ¹åããã
Speech ã¢ããªã³ã° (vs. WaveNet), 匷ååŠç¿ (vs. IMPALA ã® LSTM(?)) ã®ã¿ã¹ã¯ã§ãããããå®éšãè¡ããCompressed Memory ã«ããå¯èœæ§ã確èªããã
5. è°è«ã¯ããïŒ
äžèšã¯ç 究課é¡ãšããŠãããšã®ããš
- ä»ã®ãã¡ã€ã³ãžã®å¿çšã¯æŽã«å¯èœã
- compression rate ã¯ã¬ã€ã€ããšã«èª¿æŽãã¹ãã
- compressor ã« RNN ã®å©çš
6. 次ã«èªãã¹ãè«æã¯ïŒ
- Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
- Sparse Transformer
[2019] Meta-Transfer Learning for Few-Shot Learning
0. è«æ
ã¿ã€ãã«
Meta-Transfer Learning for Few-Shot Learning
ãªã³ã¯
http://openaccess.thecvf.com/content_CVPR_2019/papers/Sun_Meta-Transfer_Learning_for_Few-Shot_Learning_CVPR_2019_paper.pdf
https://arxiv.org/abs/1812.02391
èè
Qianru Sun, Yaoyao Liu, Tat-Seng Chua, Bernt Schiele
æçš¿æ¥ä»
CVPR 2019
1. ã©ããªãã®ïŒ
Few-Shot Learning ã«ãããŠãMeta Learning ãš Transfer Learning ãçµã¿åãããããšã§ããæ·±ãã¢ãã«ã§ãé«éã« Few-Shot ãªã¿ã¹ã¯ã«é©çšå¯èœãª Meta-Transfer Learning ãææ¡ã
Meta Learning ã®æ¢åææ³ã§ã¯ãå€ãã®ã¿ã¹ã¯ãå¿
èŠã§ããã€éåŠç¿ãé¿ããããå°ããªãããã¯ãŒã¯ã䜿ãå¿
èŠããã£ãããã©ã¡ãã®åé¡ã軜æžãããŠããã
䜵ããŠåŠç¿å¹çåã® hard task meta-batch ãææ¡ããminiImageNet ãªã©ã®ããŒã¿ã»ãã㧠SOTA ãäžåã£ãã
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
Meta-Learning ã«ãã Deep ãªã¢ãã«ã䜿çšãéåŠç¿ãæããŠè¯ãçµæãšãªã£ãã±ãŒã¹ã¯åã
倧ããªã¿ã¹ã¯ã§åŠç¿ãããã®ãã©ã¡ãŒã¿ãåºå®ãã€ã€ Scaling ãš Shift ã ãåŠç¿å¯èœãªãã©ã¡ãŒã¿ã§å€æããŠå°ããªã¿ã¹ã¯ã解ãããã¢ã€ãã£ã¢ã¯åã
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
miniImageNet, Fewshot-CIFAR100 ã§å®éšããå€ãã® Few-Shot Learning ã®ææ³ (data augmentation ç³», metric learning ç³», memory network ç³», gradient descent ç³») ãšæ¯èŒããŠã»ãšãã©ã§ SOTA ãªæ§èœãšãªã£ãã
5. è°è«ã¯ããïŒ
6. 次ã«èªãã¹ãè«æã¯ïŒ
[2020] Synthesizer: Rethinking Self-Attention in Transformer Models
0. è«æ
ã¿ã€ãã« | Synthesizer: Rethinking Self-Attention in Transformer Models |
---|---|
ãªã³ã¯ | https://arxiv.org/abs/2005.00743 |
èè | Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, Che Zheng |
æçš¿æ¥ä» | 2020/05/02 on arxiv |
1. ã©ããªãã®ïŒ
Transformer ã® self-attention æ©æ§ã® attention weight ã«ã€ããŠèª¿æ»ã»å®éšãè¡ã£ãç 究ãéåžžã® self-attention ã§ã¯ãdot-product ã«ããå
¥åæã® token ããšã®çžäºäœçšã§èšç®ãããçµæãšããŠå
¥åãµã³ãã«ããšã«ç°ãªããã®ãšãªãããããæ¬åœã«æå³ãããã®ãæ€èšŒãè¡ããããattention weight ãå¥ã®ãã®ã§çœ®ãæããã¢ãã« SYNTHESIZER ãå®çŸ©ãå®éšã
çµæãšããŠãtoken å士ã®äœçšãšã¯ç¡é¢ä¿ã« attention weight ãåŠç¿ããããã® (Dense Synthesizer) ã§ã Transformer ãšåçã®æ§èœãšãªãããã€ãŸãã©ã³ãã è¡åã attention weight ã«çšãããã®(Random Synthesizer)ã§ãäžéšã¿ã¹ã¯ã§ã¯ååè¯ãæ§èœãšãªãããšç¢ºèªããã
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
Transformer ã® self-attention ã®å¹æã«é¢ããå 容ã¯çããããã€ãã®å¹æãã»ãŒãªããšããçµæã¯ç®æ°ãããã®ã§é©ãã倧ããã
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
2ã€ã® Synthetic Attention ãææ¡ã»å®éšã
- Dense Synthesizer
- Key, Query ã® dot-product ã®ä»£ããã«ãtoken ããšã® dense layer ãéã㊠shape ãæãããã®ã
- 次ã®ãããª
[length, dim]
->[length, length]
ãšãªãé¢æ°ãçšæãã - attention layer ã®åºåã¯, ããã« value ã§ãã
G(X)
ãããã次ã«ãªãã - å®éã¯ã次ã®ããã«
F(X)
ã¯2局㮠FFN ã䜿ã (èšå·ãåãã ãå®éã¯éãéã¿ããšæããã? shape åããªãã®ã§)
- Random Synthesizer
- Dense Synthesizer ã§ã¯é¢æ°ãéã㊠input tokens ããèšç®ãã B ããã©ã³ãã åæåã㊠R ãšãã
- input tokens ã«äŸåãããtask ã«å¯ŸããŠæå¹ã§ global 㪠alignment ãšãªãããšãç®çã
- attention layer ã®åºå㯠value ããããŠæ¬¡ã«ãªãã
- ãã® R ã¯ãtrainable ãš fixed ã®äž¡æ¹ãè©Šãã çµæçã« relative positional representation ã®ãããªãã®ã«ãªã£ãŠãããšæšæž¬ã§ããã
- ãã©ã¡ãŒã¿æ°ã¯ length x length
- ãã®2ã€ã«å ããŠãDense, Random ããããã®ãã©ã¡ãŒã¿åæžã®ããã® Factorized ããŒãžã§ã³ãšãæŽã« Vanilla Transformer ãå«ããŠçµã¿åããã mixing ããŒãžã§ã³ãªã©å®éšããŠããã
Transformers ãå«ããéãã®äžèŠ§ã¯è¡š1ã«èŒã£ãŠããã
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
ææ¡ããè€æ°ã®ã¢ãã«(çµã¿åããå«ã)ã WMT (翻蚳), LM1B (LM), CNN/Dailymail (èŠçŽ), PersonaChat (察話) ã§å®éšãè¡ããVanilla Transformer æ¯èŒããã
çµæã¯äžèšè¡šãå
šäœãšããŠãvanilla transformer ã§ãªããŠãæ§èœã倧差ãªããrandom matrix (fixed) ã§ãå€å°å£ããããããã®æ§èœãåºãããŸããattention éšåã Vanilla + Dense ãªã©çµã¿åãããããšã§ Vanilla Transformer ãè¶
ããæ§èœã«ãªã£ãŠããã
ãã®ä»ã®å®éšã¯ã³ã¡ã³ãã§ã
5. è°è«ã¯ããïŒ
- ã¿ã¹ã¯ã«ãã£ãŠãdot-product ãè¯ããã®ãããã°ãdense ã random ã® synthesizer ãè¯ããã®ãããã£ãœã
- èŠçŽã¯ dot-product ããå€ãããšç²ŸåºŠãèœã¡ããtoken-token ã® interaction ãéèŠãªã®ãããããªãã
- 察話㯠dot-product ã ãšç²ŸåºŠãèœã¡ããmix ã§ãèœã¡ããtoken-token interaction ã粟床ãäžããå¯èœæ§ãããã
- GLUE ãªã©ã§ããã®2ã€ã®æãå ¥åããã¿ã¹ã¯ã¯çµæçã« self-attention ãäºæã® attention (cross-attention) ã«ãªãã®ã§ãé€ããšç²ŸåºŠãèœã¡ãã
6. 次ã«èªãã¹ãè«æã¯ïŒ
- Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation
- https://arxiv.org/abs/2002.10260 Alessandro Raganato, Yves Scherrer, Jörg Tiedemann
- 䌌ãŠããå è¡ç 究ãšããŠçŽ¹ä»
- fixed 㪠attention weights (人æã§äœæ) ãçšããŠæ©æ¢°ç¿»èš³ã§ããŸãæ§èœãå€ãããªãããšã確èª
[2020] BERTScore: Evaluating Text Generation with BERT
0. è«æ
ã¿ã€ãã«
BERTScore: Evaluating Text Generation with BERT
ãªã³ã¯
https://arxiv.org/abs/1904.09675
https://github.com/Tiiiger/bert_score
èè
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, Yoav Artzi
æçš¿æ¥ä»
ICLR2020, arxiv on 21 Apr 2019
1. ã©ããªãã®ïŒ
ããã¹ãçæãšããè©äŸ¡ã®é£ãããã¡ã€ã³ã«ãããŠãèªåè©äŸ¡ææšã§ãã BERTScore ãææ¡ãreference æãš candidate (äºæž¬) æã® BERT ã® contextual embeddings ãçšããŠé¡äŒŒåºŠãèšç®ããã
æ©æ¢°ç¿»èš³ãš image captioning ã®ã¿ã¹ã¯ã§å®éšããæ¢åã®èªåè©äŸ¡ææšã learnable ãªè©äŸ¡ææšãšæ¯èŒãå®å®ããŠè¯ãã¹ã³ã¢ãç²åŸãããã¹ãæ§æ€èšŒã®ãã PAWS ã§ãå®éšãè¡ããæ¢åææ³ãšæ¯ã¹ãŠããªãé«ãããã¹ãæ§ã瀺ããã
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
BERT ã® contextual embedding ãçšããããšã§ãåçŽãªåèªã®ãããã³ã°ãåèªã®æå³ã®é¡äŒŒåºŠã§ã¯ãªããæå
šäœã§ã®æå³çãªé¡äŒŒåºŠãæäŸã§ããã
äžæ¹ã§è¿œå åŠç¿ã®å¿
èŠããªããBERT ãªã© pre-trained model ãæ¢ã«å€æ°ã®èšèªã§æäŸãããŠããããå€ãã®èšèªã»ãã¡ã€ã³ã§ç°¡åã«äœ¿ãããšãã§ããã
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
BERTScore ã¯ã·ã³ãã«ã«ç®åºã§ãã次ã®å³ãããããããã
- reference æ, candidate æããããã BERT ã«å ¥åã contextual embedding ãåŸã
- candidates/reference ã®å token ãã¢ããšã«é¡äŒŒåºŠãèšç®ãã (å³ã® Pairwise Cosine Similarity)
- æãé«ããã®ããã® token ã®é¡äŒŒåºŠãšãã (å³ã® maximum similarity)
- idf ã§æåŸ å€ãåã
ããã«ããåŸãããã¹ã³ã¢ãèšç®ã®ä»æ¹ã§ Precision, Recall, F1, åã³ idf ã«ããéã¿ä»ãã®æç¡ã§ããããå®éšããã¿ã¹ã¯å·®ã¯ãããã®ã®ã©ããåããããã®çµæãåŸãŠããã
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
æ©æ¢°ç¿»èš³ãimage captioning, ããã¹ãæ§ã®æ€èšŒãã®3ã€ã«ã€ããŠåºç¯å²ã«å®éšã
- æ©æ¢°ç¿»èš³
- Image captioning
- ããã¹ãæ§ã®æ€èšŒ
- PAWS (Paraphrase Adversaries from Word Scrambling dataset) ã® QQP (Quora Question Pair data) ã䜿çš
- äžéšã®åèªããæå³ãå€ãã (= paraphrase ã§ã¯ãªã) ããã« swapping ããŠãã
- QQP ã§åŠç¿ããã¢ãã«ããPAWS ã§ã©ãæ§èœãèœã¡ãããèŠãŠãã
- æ¢åææ³ã倧ããæ§èœãèœã¡ãŠããã®ã«å¯ŸããŠãBERTScore ã¯å°ãèœã¡ãçšåºŠããã€ä»ã®å ç«¥è©äŸ¡ææšãããè¯ãã¹ã³ã¢ã«
- PAWS (Paraphrase Adversaries from Word Scrambling dataset) ã® QQP (Quora Question Pair data) ã䜿çš
5. è°è«ã¯ããïŒ
- BERTScore 㯠task agnostic ãªææšãªã®ã§ããããåºã«ããã¿ã¹ã¯ç¹åã®ææšãäœããã
- BERTScore 㯠differentiable ãªã®ã§ããããåŠç¿ã«çµã¿èŸŒãããšãã§ãããã
6. 次ã«èªãã¹ãè«æã¯ïŒ
- PAWS: Paraphrase Adversaries from Word Scrambling
- ããã¹ãæ§ã®å®éšã«çšããããŒã¿ã»ãã
- https://arxiv.org/abs/1904.01130
- MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance
- åææã«åºããEarth Mover distance ãçšããè©äŸ¡ææš
- https://arxiv.org/abs/1909.02622
[2019] Deep RNN Framework for Visual Sequential Applications
0. è«æ
ã¿ã€ãã«
Deep RNN Framework for Visual Sequential Applications
ãªã³ã¯
http://openaccess.thecvf.com/content_CVPR_2019/papers/Pang_Deep_RNN_Framework_for_Visual_Sequential_Applications_CVPR_2019_paper.pdf
https://github.com/BoPang1996/Deep-RNN-Framework
èè
Bo Pang, Kaiwen Zha, Hanwen Cao, Chen Shi, Cewu Lu
æçš¿æ¥ä»
CVPR 2019
1. ã©ããªãã®ïŒ
Visual Sequential ç³»ã¿ã¹ã¯ã®ããã®ã¢ãã«ã«2ã€ã®æ°ããèšèšãåãå
¥ããããšã§ããã Deep ãªæ§é ã®ã¢ãã«ã§ãç°¡åã«æé©åãã§ããããã«ãªãããšãå®éšçã«ç€ºããã
å®éšã§ã¯ãæ¢åã® Shallow ãªã¢ãã«ã«å¯ŸããŠ15å±€ã®ãããã¯ãŒã¯ãææ¡ææ³ã§åŠç¿ã«æåããæ§èœã倧ããæ¹åããã
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
Visual Sequence ã®ã¿ã¹ã¯ã«ãã㊠Deep 㪠RNN ãçšããããšã¯åŠç¿ã®é£ãããèšç®éã®èŠ³ç¹ã§çŸå®çã§ã¯ãªãã£ãããææ¡ææ³ã§ãããæ¹åããDeep 㪠RNN ã§åŠç¿ãã§ããããã«ãªã£ãã
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
2ã€ã®æ°ããèšèš: Context Bridge Module (CBM) ãš Overlap Coherence Training Scheme ãææ¡ã
- CBM:
- 2ã€ã®æ å ±ãåããŠåŠçãããæ°ã㪠RNN ã¢ãžã¥ãŒã«
- temporal flow ãš representation flow ã«åããŠåŠçãè¡ã
- 2ã€ãåããããšã§ã Temporal Dropout (TD) ãå ããããšãå¯èœã«
- Overlap Coherence Training Scheme
- èšç®éå¢å€§ã«ã€ãªãã£ãŠããé·ã sequence ã®åŠçãå¹çåããåŠç¿ã¹ããŒã
- é·ã sequence ãçã sequence ã« overlap ãæã€ããã«åå²ãã
- ãã sequence ã®åŠçãå§ãããšãã® initial state ã¯ãoverlap ããŠããå¥ã® clip ã®å¯Ÿå¿ãã timestamp ã® hidden state ãåã£ãŠãã
- overlap coherent loss ãè¿œå ããç°ãªã clips éã§ççŸããªãããã«åŠç¿
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
15局㮠RNN (æ®éã¯2å±€ãšããªã®ã§ãçŽ7å)ã§ãã¬ãŒãã³ã°ã«æåããKinetics ãªã©ã®ã¿ã¹ã¯ã§ Shallow ãªãããã¯ãŒã¯ã«æ¯ã¹ãŠ11%ã®æ¹åã«æåããã
ãŸããæ¢åã¢ãã« (Polygon-RNN) ã®ãããã¯ãŒã¯éšåã15å±€ã®ææ¡ã¢ãã«ã«çœ®ãæããããšã§ãã¹ã³ã¢ã 14.7% æ¹åã§ããã
video future prediction ã¿ã¹ã¯ã§ã¯ãæ¢åã®SOTAã¢ãã«ã 2.4%æŽæ°ããã
5. è°è«ã¯ããïŒ
- ãããªã validation ãããã¹ãæ§ã®æ€èšŒã¯å¿ èŠ
6. 次ã«èªãã¹ãè«æã¯ïŒ
[2019] Frustratingly Easy Natural Question Answering
0. è«æ
ã¿ã€ãã«
Frustratingly Easy Natural Question Answering
ãªã³ã¯
https://arxiv.org/abs/1909.05286
èè
Lin Pan, Rishav Chakravarti, Anthony Ferritto, Michael Glass, Alfio Gliozzo, Salim Roukos, Radu Florian, Avirup Sil
(IBM)
æçš¿æ¥ä»
2019/09/11
1. ã©ããªãã®ïŒ
Natural Questions ãšãã£ãé£ãã QA ã¿ã¹ã¯ã«ãããŠãLeaderboard äžäœã«ãããã®ã®è«æã§èª¬æãããŠããªããããªææ³ãçšããŠãããã¯ããã¯ãªã©ã玹ä»ããè«æã
Attention-over-Attention, Data Augmentation, Ensemble strategy, pretrain æ¹æ³ãªã©ãã¹ã³ã¢ãäžããããã®ææ³ã説æããã
Natural Questions 㧠SoTA ãæŽæ°ã
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
Attention-over-Attention ãå§ããšãããè«æãªã®ã§ã¯è§ŠããããŠããªã QA ã®ã¹ã³ã¢ãæ¹åããããã®ãã¯ããã¯ã玹ä»ãã€ã€ãå®éšã§æ€èšŒããã
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
- Attention-over-Attention
- question ã® tokens Q, context ã® tokens C ã«å¯ŸããŠãM = CQ^T ãèšç®
- M ã row-wise, column-wise ã«ãããã softmax ãåããβ, α ãšãã
- βã row ã§å¹³åããšãã s = α^T β ^T ãèšç®ãã
- äžèšã®æç¶ãã BERT ã® output ã«å¯Ÿã㊠2heads ã§è¡ããåŸããã s_1, s_2 ã start, end ã® logits èšç®ã«éã¿ä»ãåã®åœ¢ã§å ãã
- Data Augmentation
- å è¡ç 究ã§ãã SOS (Sentence Order Shuffling)
- æ¢åããŒã¿ã»ãã (SQuAD 2.0, NewsQA, TriviaQA) ã®ã©ã³ãã æœåºãšãNQã«äŒŒãããŒã¿ãæœåºãããã®
- Ensemble æŠç¥
- Seed ã¢ã³ãµã³ãã«ãšè€æ°ã¢ãã«ã®ã¢ã³ãµã³ãã« ããããå®éš
- hyper parameters tuning 㧠stride ã sampling rate ã埮調æŽ
- stride 㯠192 ã«
- sampling rate ã¯åçå¯èœãã©ããã§å€ãã
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
Natural Questions ã§å®éšããå è¡ç 究㮠SoTA ãäžåã£ã
5. è°è«ã¯ããïŒ
- Learning rate ã Batch size ã¯å°ãå€ããã ãã§å€§ããçµæãç°ãªããããä»åã¯ã¡ãããšèª¿ã¹ãŠãªããšã®ããš
- BERT 以å€ã® pre-train ã¢ãã«ãè©Šããã
6. 次ã«èªãã¹ãè«æã¯ïŒ
- Synthetic QA Corpora Generation with Roundtrip Consistency
- https://arxiv.org/abs/1906.05416
- Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, Michael Collins (Google Research)
- ACL2019
- BERT joint ã Data Augmentation ã§ã¹ã³ã¢æ¹åããç 究
[2019] Class-Balanced Loss Based on Effective Number of Samples
0. è«æ
ã¿ã€ãã«
Class-Balanced Loss Based on Effective Number of Samples
ãªã³ã¯
https://arxiv.org/abs/1901.05555
ã³ãŒã: https://github.com/richardaecn/class-balanced-loss
èè
Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, Serge Belongie
æçš¿æ¥ä»
CVPR 2019
1. ã©ããªãã®ïŒ
åé¡åé¡ã«ãããŠãäžéšã¯ã©ã¹ãããŒã¿ã®ã»ãšãã©ãå ããå€ãã®ã¯ã©ã¹ã®ããŒã¿æ°ã¯å°ãªããšãã£ããããªãããŒã¿ã®ååžã long-tail ã«ãªã£ãŠããåé¡ã«å¯ŸããŠãre-weight ããŒã¹ã®æ°ããã¢ãããŒã class-balanced loss ãææ¡ããã
ã¯ã©ã¹ããšã®èŠ³æž¬ããŒã¿æ°ã®éæ°ãéã¿ãšãããªã©ããŠè£æ£ããã®ãäžè¬çãªã¢ãããŒãã ããæ¬ç 究ã§ã¯ãããŒã¿æ°ãå¢ãããšæ°ããããŒã¿ã®éèŠåºŠã¯äžãããšããŠãããŒã¿æ°ã§ã¯ãªãããŒã¿ã®æå¹æ° (effective number of samples) ãå®çŸ©ãããããçšã㊠loss ã調æŽããŠããã
CIFAR ã® long-tail çããImageNet, iNaturalist ãªã©ã§å®éšãã倧ããªæ¹åãèŠãããã
å³ã¯ã€ã¡ãŒãžã®èª¬æãéåžžã®åŠç¿ã ãšæ±ºå®å¢çã¯ãµã³ãã«æ°ã®å°ãªãæ¹ã«å¯ã£ãŠæ±ºå®å¢çãé»ãç·ã«ãªã£ãŠããŸããåã«ã¯ã©ã¹æ¯ã®éæ°ã§éã¿ä»ãããããšåŒ·ãããŠæ±ºå®å¢çãèµ€ãç·ã«ãªã£ãŠããŸããæå¹ããŒã¿æ°ã®éæ°ã䜿ãããšã§éãç·ãç®æãããšããè©Šã¿ã
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
ããŒã¿ã®æå¹æ°ãšããæŠå¿µãææ¡ããåŠç¿æã®ã¯ã©ã¹ã®éã¿ã«äœ¿ã£ãäŸã¯åã
æ¢åã®æ倱é¢æ° (focal loss ãªã©) ãšçµåããŠäœ¿ãããšãåºæ¥ãã
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
- ããŒã¿ã®æå¹æ°ã random covering ãçšããŠå®çŸ©ããããšã§ããŒã¿å¢å ã«ããç¹åŸŽç©ºéäžã§ã®ã¯ã©ã¹ã®é åãžã®å¯äžãèšç®ããåŠç¿æã®ã¯ã©ã¹ã®éã¿ãšããŠããã
- æçµçã«ã¯å¥éèšç®ããä¿æ°ã loss ã«ãããã ããªã®ã§ã·ã³ãã«
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
long-tailed CIFAR (10,100), iNaturalist (2017, 2018), ILSVRC 2012 ã§å®éšã
softmax loss, sigmoid loss, focal loss ãšçµåããŠå®éšãããã¹ãŠã§ class-balanced 㪠loss ã䜿ã£ããšãã®ã»ããè¯ãçµæãšãªã£ãŠããã
æé©ãª β ã«ã€ããŠãè°è«ããŠããã CIFAR 10 ã®å Žå㯠β=0.9999, CIFAR 100 ã®å Žå㯠β=0.9 ãšãªã£ãŠãããããããç¹åŸŽç©ºéäžã®ã¯ã©ã¹ã®äœç©ã®ç解ãšãã£ãŠããã
CIFAR 10 ã®å Žåã¯1ã€ã®ã¯ã©ã¹ãã§ãã(äŸ: é³¥) ãªã®ã§æå¹ããŒã¿æ° N ã®å€ã倧ãããCIFAR 100 ã®å Žåã¯1ã€ã®ã¯ã©ã¹ããã£ãšçŽ°ãã (äŸ: ã¹ãºã¡) ã®ã§æå¹ããŒã¿æ° N ãå°ãããªãã
5. è°è«ã¯ããïŒ
ããŒã¿ã®ååžã«äœãä»®å®ã眮ããŠããªããããäœã«ã§ãé©çšã§ããäžæ¹ã§ãããŒã¿ã®ååžã«ä»®å®ããããå Žåã®æ¡åŒµããé©å¿çã«åŠç¿ãããããªæ¹æ³ã«ã€ããŠã¯ future work ãšããŠããã
6. 次ã«èªãã¹ãè«æã¯ïŒ
- Focal Loss for Dense Object Detection
- https://arxiv.org/abs/1708.02002
- Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, Piotr Dollár
- ICCV 2017?
- Focal Loss ã®ææ¡è«æ
[2020] Dynamic Knowledge Routing Network For Target-Guided Open-Domain Conversation
0. è«æ
ã¿ã€ãã«
Dynamic Knowledge Routing Network For Target-Guided Open-Domain Conversation
ãªã³ã¯
https://arxiv.org/abs/2002.01196
èè
Jinghui Qin, Zheng Ye, Jianheng Tang, Xiaodan Liang
æçš¿æ¥ä»
AAAI2020 (Submitted on 4 Feb 2020 (v1), last revised 6 Mar 2020 (this version, v2))
1. ã©ããªãã®ïŒ
Target-guided open-domain conversation ãšãããããã¿ãŒã²ãããšãªããããã¯ãããŒã¯ãŒãããŽãŒã«ãšããŠèšå®ããéè«ã¿ã¹ã¯ã«ã€ããŠã®æ°ããã¢ãã« dynamic knowledge routing network (DRKN) ãææ¡ããã
ãã®ã¿ã¹ã¯ã¯ Tang et al. (2019) ã«ãã£ãŠæè¿ææ¡ããããã®ã§ãopen-domain ãªéè«ã«ãããŠãæ確ã§ã¯ãªããç®çããããšããæéã¿ãŒã³ã§ãã人éãš agent ã®éè«ã®äžã§ãããªãŒãããã¯ããå§ãŸãã決ããããããŒã¯ãŒãã«èªç¶ã«å°éããããããªãã®ã
çºè©±å ã®ããŒã¯ãŒãã察話履æŽãããã£ã«ã¿ãªã³ã°ãè¡ã DKRNãçºè©±å ã®ããŒã¯ãŒãã«é¢é£ããããŒã¯ãŒãããæœåºããè¿çåè£ãããtarget ã«è¿ããã®ãéžã¶äºæ®µéã®æŠç¥ãªã©ãåãå ¥ãããããã¯ãŒã¯ãææ¡ãã keyword prediction, response retrieval ãšãã«æ¢åææ³ãäžåã粟床ãšãªã£ãã
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
æ¢åã¢ãã« (Tang et al. 2019) ã§ã¯ keyword ã䜿çšããéã«ãã® keyword ã®æå³çãªé¢ä¿ã䜿çšããŠããªãç¹ããDKRN ã§ã¯æŽ»çšããããšã§ç²ŸåºŠãæ¹åããã
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
ã¢ãã«ãè¥å¹²è€éã ãããã¢ãšãªãã®ã¯æ¬¡ã®2ç¹ã
- DKRN ãçšããŠé¢é£ããŒã¯ãŒãããã£ã«ã¿ãªã³ã°
- ãŸã RNN ãçšããŠçŸåšã®çºè©±ãã次ã®äŒè©±ã®ããŒã¯ãŒãã®ååžãäºæž¬
- 次ã«å¯Ÿè©±ã³ãŒãã¹ããåŠç¿ããé¢é£ã°ã©ã(åã«ååŸã®äŒè©±ã§ããŒã¯ãŒãã®çµã¿åãããååšãããã©ãã)ãçšããŠé¢é£ã®ãªãããŒã¯ãŒãããã¹ã¯
- dual discourse-level target guided strategy (äºéã®æŠç¥)
- 察話ã®ã¹ã ãŒãºæ§ (æ¥ã« target ã«åããŠè©±é¡ãå€ãããšå¥åŠ) ãš target éæãžã®é²åºŠã®ãã¬ãŒããªããããæãã«ããããã®ãã®
- DKRN ãªã©ãéããŠéžæãããããŒã¯ãŒãããé¢é£ããããŒã¯ãŒããçšããŠåè£æãäœæ (話é¡ãèªç¶)
- ãã®äžã§ target ãšã®èŠªåæ§ãæãé«ããã®ãå床éžæãã (target éæãžé²ã)
ãŸããåéããããŒã¿ã»ããã§ãã CWC (æ¢åã®ãã®ããããªã倧ãã) ãå ¬éããã
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
Tang et al. (2019) ã§ææ¡ãããŠããããŒã¿ã»ãã (Target-guided PersonaChat Dataset: TGPC, PersonaChat ã target-guided ã«æ¡åŒµãããã®) ãšãæ°ãã«éããããŒã¿ã»ãã(Chinese Weibo Conversation Dataset: CWC) ãçšããŠã䞻㫠Tang et al. ã®ã¢ãã«ãšæ¯èŒå®éšã
å
šãŠã«ãããŠæãè¯ãã¹ã³ã¢ãšãªã£ãŠããã
human evaluation
ã¢ãã«åäœããšã®è©äŸ¡ | ã¢ãã«å士ã®æ¯èŒ |
---|---|
![]() |
![]() |
5. è°è«ã¯ããïŒ
6. 次ã«èªãã¹ãè«æã¯ïŒ
- Target-Guided Open-Domain Conversation (ACL 2019)
- https://arxiv.org/abs/1905.11553
- Target-Guided ãªã¿ã¹ã¯ãå®çŸ©ããç 究, TGPC ã®ããŒã¿ã»ããã®å ¬éãè¡ã£ãŠãã
[2019] Large Scale Incremental Learning
0. è«æ
ã¿ã€ãã«
Large Scale Incremental Learning
ãªã³ã¯
http://openaccess.thecvf.com/content_CVPR_2019/papers/Wu_Large_Scale_Incremental_Learning_CVPR_2019_paper.pdf
https://arxiv.org/abs/1905.13260
èè
Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, Yun Fu
æçš¿æ¥ä»
CVPR 2019
1. ã©ããªãã®ïŒ
åŸã
ã«æ°ããã¯ã©ã¹ã®ããŒã¿ãå¢ãããããªåé¡åé¡ã«ãããŠãå€ãã¯ã©ã¹ã®èå¥ãã§ããªããªãç Žæ»
çå¿åŽãé²ããã㪠Incremental Learning ã®ææ³ã§ãã BiC (bias correction) ãææ¡ã
ç Žæ»
çå¿åŽã¯ãclassifier layer (æåŸã® dense layer) ã§ãæ°ããã¯ã©ã¹ã«å¯ŸããŠåŒ·ã bias ãããããšãèŠå ã®1ã€ã§ããããšãçºèŠããç·åœ¢ã¢ãã«ã§ bias ãè£æ£ããã¬ã€ã€ãŒãè¿œå ããŠããã
Incremental Learning ã®æ¢åææ³ã§ãã iCaRL ã EEIL ãšæ¯èŒããImageNet ã10åã® increment batch ã«åããå®éšã§ï¿œSOTA ãæŽæ°ããã
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
ç Žæ» çå¿åŽã«ã€ããŠã®åæã§æçµ dense ã¬ã€ã€ã§ã® bias ã«çç®ãããããè£æ£ããã¬ã€ã€ãè¿œå ããããšã§ Incremental Learning ã®æ§èœãäžããåãçµã¿ã
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
åŠç¿ã stage1 ãš stage2 ã«åããããŒã¿ãåãããŠæ°ããã¯ã©ã¹ãå€ãã¯ã©ã¹ã®äžéšããããã train ãš valid ã§åããŠãstage 1 ã§ã¯éåžžéãã¢ãã«ãåŠç¿ããstage 2 㧠stage 1 ã®ã¢ãã«ããã©ã¡ãŒã¿åºå®ããŠäœ¿ããbias correction layer ã®ãã©ã¡ãŒã¿ãåŠç¿ããã
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
Incremental Learning ã®æ¢åææ³ã§ãã iCaRL ã EEIL ãšæ¯èŒããImageNet ã10åã® increment batch ã«åããå®éšã§ï¿œSOTA ãæŽæ°ãããç¹ã«æ°ããã¯ã©ã¹ã倧éã«ãããšãã«å€§ããªå¹æããã£ãã
5. è°è«ã¯ããïŒ
6. 次ã«èªãã¹ãè«æã¯ïŒ
[2019] Pretraining Methods for Dialog Context Representation Learning
0. è«æ
ã¿ã€ãã«
Pretraining Methods for Dialog Context Representation Learning
ãªã³ã¯
https://arxiv.org/abs/1906.00414
èè
Shikib Mehri, Evgeniia Razumovskaia, Tiancheng Zhao, Maxine Eskenazi
æçš¿æ¥ä»
ACL 2019
1. ã©ããªãã®ïŒ
察話æèã®è¡šçŸåŠç¿ããã®äºååŠç¿(æåž«ãªãåŠç¿ã®ç®çé¢æ°)ã«ã€ããŠèª¿æ»ããç 究ã
æ¢åã®äºååŠç¿ã¢ãã«ã¯èšèªã¢ãã«ãããŒã¹ã«ããŠãããtoken ã¬ãã«ã§ã®è¡šçŸãåŸããããã察話ã§ã¯è€æ°ã¿ãŒã³ã®æèãå¿
èŠã«ãªããtoken ã¬ãã«ã§ã¯ãªã discourse ã¬ãã«ã®è¡šçŸãåŠç¿ã§ããäºååŠç¿ã®ãããã³ã«4ã€(ãã¡2ã€ã¯æ°ãã«ææ¡)ã®æ¹æ³ã«ã€ããŠå®éšãåæãè¡ã£ãã
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
æã¬ãã«ã®ç解ãå¿
èŠã«ãªã察話ãã¡ã€ã³ãã¿ãŒã²ãããšããŠãdiscourse ã¬ãã«ãåŠç¿ã§ãããããªæ¢åã®äºååŠç¿ãããã³ã«2ã€ãšãæ°ãã«ææ¡ãã2ã€ã®èš4ã€ã«ã€ããŠèª¿æ»ã»å®éšã»åæãè¡ã£ãŠããã
word ã¬ãã«ã®ç 究ã¯å€ããã察話ã«ç¹åããäºååŠç¿ææ³ã«ã€ããŠãã®æ§èœã調æ»ããäŸã¯åã
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
äºååŠç¿ã®ç®çé¢æ°ãšããŠã次ã®4ã€ãè©Šãã
- Next-Utterance Retrieval (NUR)
- 次ã®çºè©±ãåè£ã®äžããéžæãã
- loss ã¯åè£ã®äžã§ã®åé¡ã§ã® cross entropy
- Next-Utterance Generation (NUG)
- 次ã®çºè©±ãçæãã
- loss ã¯å token ã®ç¢ºçã®è² ã®å¯Ÿæ°å°€åºŠ (= cross entropy)
- Masked-Utterance Retrieval (MUR)
- MLM ã®æ ver. ã§ã察話履æŽã®äžã€ã®æã replace ããŠãæ£è§£ãåè£ããåœãŠã
- loss ã¯åè£ã®äžã§ã®åé¡ã§ã® cross entropy
- Inconsistency Identification (InI)
- MUR ã§ã¯äžã€ã®æã replace ããæ£è§£ãåè£ã®äžããæ¢ãããInI 㯠replace ãããæãã©ãããã€ã³ããã¯ã¹ã§åœãŠã
- loss ã¯ã€ã³ããã¯ã¹ã®åé¡ã§ã® cross entropy
ãŸãã次ã®äžäœã¿ã¹ã¯ã§æ§èœãèŠãã
- Belief State prediction (BSP)
- 1784次å ã®äŒè©±ã¹ããŒãã®äºæž¬ã27åã® Entity ããããããããã® one-hot vector ãã€ãªãããã®ã
- å
容ã¯ãã¡ã€ã³æ¯ã«è²ã
ãã£ãŠãããTaxi ãªã
leaveAt
,destination
ãªã©ã® entity ãæã£ãŠããã
- Dialog Act Prediction (DAP)
- BSP ãšã«ãŠãããã察話ã®çµæ system ããšã action ã®äºæž¬ã32次å ã® multi-hot vector ãäºæž¬ããã
- é»è»ã®æå»ãçããããããã«ã« query ãæããããªã©ã
- Next-Utterance Generation
- 次ã®çºè©±ã®çæ
- BLEU-4 ã§èšæž¬
- Next-Utterance Retrieval
- 次ã®çºè©±ã k åã®åè£ããéžæ
- æ§èœè©äŸ¡ãšããŠè¯ãããã (Lowe et al., 2016)
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
(1) äºååŠç¿ + finetuning ã«ããæ§èœã(2) åææ§ã (3) finetuning æã®ããŒã¿ãå°ãªãå Žåã®æ§èœã(4) ãã¡ã€ã³æ±åæ§èœãã®4ç¹ã§æ§èœãæ€èšŒã
- external data ã䜿ããã«ã finetuning ã§äœ¿çšããããŒã¿ (Multiwoz dataset) 㧠pretrain ãè¡ã£ãŠãäžèšã®äžäœã¿ã¹ã¯ã§æ€èšŒ
- additional data ã«ããæ©æµã§ã¯ãªããpretrain ã«ããæŽã«æ å ±ãåŸãããŠãããã©ãããèŠãããã
- åçŽã«ããããã®ã¿ã¹ã¯ã«ã€ããŠè©Šããã®ã Table 1 (äžèš)ã
- limited data ã§å®éšãfinetuning æã®ãã¬ãŒãã³ã°ããŒã¿ãæžãããŠæ§èœãèŠã
- ãã¡ã€ã³æ±åæ§èœããã
- InI, MUR ã¯å¯Ÿè©±å±¥æŽãé·ã䜿ããå Žåã«å¹æç (Table 5)
5. è°è«ã¯ããïŒ
- NUR ãš NUG ã¯ãäºãã«è£å©çãªã¿ã¹ã¯ã«ãªã£ãŠãã
- NUG ã¯ãã®ã¿ã¹ã¯ã®é£ããããã general ãªè¡šçŸãåŠç¿ã§ãã
- InI ãš MUR ã¯åçºè©±ã® local representation ãããªãåŠç¿ã§ããŠãã
- ãã® pretrain ã«ãã NUG 粟床ã¯äžãã
- NUG ã¯å šäœã®æè (global representation) ã ãã§ã¯ãªããåçºè©±ã®ç解 (local representation)ãå¿ èŠ
- pretrain ãš finetune ã§åãããŒã¿ã䜿ãããšã«ã€ããŠ
- åãããŒã¿ã§äºåã«ç°ãªãç®çé¢æ°ã§åŠç¿ãããããšã§ãããããè¡šçŸãç²åŸã§ããŠããããšãåãã£ã
- future work ãšããŠ
- ãã倧ã㪠external data ã§ã®äºååŠç¿
- é¢ä¿æ§ã®èãããŒã¿ã»ããã§ã®äºååŠç¿ã«ãããã¹ã
- word-level ã®ç®çé¢æ°ãè¿œå ãããã¹ã
6. 次ã«èªãã¹ãè«æã¯ïŒ
- MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for
Task-Oriented Dialogue Modelling- https://arxiv.org/abs/1810.00278
- PaweÅ Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, Milica GaÅ¡iÄ
- EMNLP 2018
- ããŒã¿ã»ãããšããŠäœ¿çšããã
[2019] Improving Multi-turn Dialogue Modelling with Utterance ReWriter
0. è«æ
ã¿ã€ãã«
Improving Multi-turn Dialogue Modelling with Utterance ReWriter
ãªã³ã¯
https://arxiv.org/abs/1906.07004
https://github.com/chin-gyou/dialogue-utterance-rewriter (2019/08/26 æç¹ã§ã¯ã³ãŒãæªå
Ž)
èè
Hui Su, Xiaoyu Shen, Rongzhi Zhang, Fei Sun, Pengwei Hu, Cheng Niu, Jie Zhou
æçš¿æ¥ä»
ACL 2019
1. ã©ããªãã®ïŒ
察話ã¢ãã«ã®èª²é¡ãšããŠãéå»ã®è€æ°ã¿ãŒã³ã®ããåãã®ã³ã³ããã¹ããä¿ã£ããŸãŸå¯Ÿè©±ãç¶ããã®ãé£ãããšãããã®ããããæ¬ç 究ã§ã¯ããŠãŒã¶ã®çºè©±ããéå»ã®çºè©±ã®ã³ã³ããã¹ããå«ãããã«ãã©ã€ããããããšã§ãã³ã³ããã¹ããä¿æãããŸãŸè€æ°ã¿ãŒã³ã®å¯Ÿè©±ãå¯èœã«ããã¢ãããŒããææ¡ããã
äŸãã°ããã©ããªæ ç»ã奜ã?ããã¿ã€ã¿ããã¯ããããªã?ããšããäŒè©±ããã£ããšãã«ã次ã®å¿çãäºæž¬ããããšãããšãããªã?ã->ããªãã¿ã€ã¿ããã¯ã奜ããªã®?ããšãªã©ã€ãããŠããããšã§ãéå»ã®çºè©±ãåç
§ããªããšãã³ã³ããã¹ããä¿ã£ããŸãŸå¯Ÿè©±ãç¶ããããšãåºæ¥ãã
ããå
·äœçã«ã¯ãçºè©±ãªã©ã€ãã¢ãã«ãšã㊠Transformer ãš Pointer Network ããŒã¹ã®ã¢ãã«ãææ¡ããåŠç¿åŸã«æ¢åã® chatbots ã«çµã¿èŸŒãããšã§å
ã®ã·ã¹ãã ãããæ¹åãèŠãããã
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
- ãªã©ã€ããè¡ãããšã§ multi turn ã®ã³ã³ããã¹ãã single turn ã«èœãšã蟌ãã§å¯Ÿè©±ã¢ãã«ã®ç²ŸåºŠããããããšããã¢ã€ãã£ã¢ã¯åã
- çºè©±ãã³ã³ããã¹ããå«ãããã«äžæžããè¡ã£ãããŒã¿ã»ãããåéããã(å ¬éäºå®?)
- åéããããŒã¿ã§ãTransformer x Pointer Network ããŒã¹ã® rewrite ã¢ãã«ãææ¡ãã
- ææ¡ã¢ãã«ãæ¢åã® chatbot ã·ã¹ãã ã«çµã¿èŸŒãããšã§ãæ¹åã確èªãã
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
- multi turn ã®ã³ã³ããã¹ãã single turn ã« rewrite ãå ããããšã§å®çŸãããšããã¢ã€ãã£ã¢ããã¢
- Pointer network ã«ãããŠã察話履æŽéšåãšãªã©ã€ã察象ã®çºè©±éšå㧠attention ãåããŠé©çšããå¥ã§åŠç¿ãã lambda ã§éã¿ä»ãåããšãããšã§ç²ŸåºŠUP
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
äžèšã®ãã¿ãŒã³ã§å®éšãè¡ã£ãã
- ã¢ãŒããã¯ãã£ã Transformer / LSTM ã®2çš®é¡
- pointer network ã®å ¥ãããã pointer only / generator only / both / pointer lambda (proposed) ã®4çš®é¡
Transformer 㧠pointer (lambda) ãæãããããã¹ãããŒã¿ã§ exact match ããªã©ã€ãããããŒã¿ã§ 55%, ãªã©ã€ããªãããŒã¿ã§ 98% ã®ã¹ã³ã¢ã«ãã»ãã¯è¡šåç §ã
ãŸããåŠç¿ãã rewriter ã¢ãã«ã察話ã·ã¹ãã ã«çµ±åããæ¢ã«åããŠãã Task-Oritend ãªã¢ãã«ãšéè«ã¢ãã«ã§çµ±åãããªãã®æ¯èŒãè¡ã£ããšãããäž¡æ¹ã§æ¹åãèŠãããã
5. è°è«ã¯ããïŒ
- è«æå ã§ã¯ç¹ã«èšåãªãã
6. 次ã«èªãã¹ãè«æã¯ïŒ
- Scaling Multi-Domain Dialogue State Tracking via Query Reformulation
- https://arxiv.org/abs/1903.05164
- Pushpendre Rastogi, Arpit Gupta, Tongfei Chen, Lambert Mathias
- NAACL 2019
- è±èªã§ã®å¯Ÿè©±ã® multi turn ã®ã³ã³ããã¹ãç解ã®ããã«åããããªã¢ã€ãã£ã¢ãé©çšããŠãã
[2019] Unequal-training for Deep Face Recognition with Long-tailed Noisy Data ð
0. è«æ
ã¿ã€ãã«
Unequal-training for Deep Face Recognition with Long-tailed Noisy Data
ãªã³ã¯
èè
Yaoyao Zhong, Weihong Deng, Mei Wang, Jiani Hu, Jianteng Peng, Xunqiang Tao, Yaohai Huang
æçš¿æ¥ä»
CVPR 2019
1. ã©ããªãã®ïŒ
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
5. è°è«ã¯ããïŒ
6. 次ã«èªãã¹ãè«æã¯ïŒ
[2019] Striking the Right Balance with Uncertainty
0. è«æ
ã¿ã€ãã«
Striking the Right Balance with Uncertainty
ãªã³ã¯
http://openaccess.thecvf.com/content_CVPR_2019/papers/Khan_Striking_the_Right_Balance_With_Uncertainty_CVPR_2019_paper.pdf
https://arxiv.org/abs/1901.07590
èè
Salman Khan, Munawar Hayat, Waqas Zamir, Jianbing Shen, Ling Shao
by Inception Institute of AI
æçš¿æ¥ä»
CVPR 2019
1. ã©ããªãã®ïŒ
class imbalanced ãªã¿ã¹ã¯ã«å¯ŸããŠããã€ãºæšå®ã®æ çµã¿ã§ã¯ã©ã¹ããšãåã³ãµã³ãã«ããšã®äžç¢ºå®æ§ã«çç®ããunbiased ãªã¢ãã«ãåŠç¿ããããã®æ°ã㪠loss ãææ¡ããã
Loss ã¯åã¯ã©ã¹ããšã®äžç¢ºå®æ§ãèæ
®ããŠã¯ã©ã¹éã®ããŒãžã³ãæ倧åããããã«ãªã£ãŠããã®ã«å ããŠãåãµã³ãã«ãå€å€éã¬ãŠã¹ååžã䜿çšããŠã¢ããªã³ã°ãããã®äºæ¬¡ã¢ãŒã¡ã³ããŸã§èæ
®ããŠã¯ã©ã¹å¢çã調æŽããŠããã
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
ãã€ãºã«ããäžç¢ºå®æ§ã®æšå®ã®æ çµã¿ã class imbalanced ãªåé¡ã«å¿çšããã
ã¯ã©ã¹éã ãã§ãªããµã³ãã«ã®äžç¢ºå®æ§ãèæ
®ãã決å®å¢çã«ãªã£ãŠããã
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
- Bayesian Uncertainty Estimates ãè¡ãããã«ãdropout ãå«ããããã¯ãŒã¯ã Gaussian Process ã«è¿äŒŒã§ããããšãå©çšããŠãã
- dropout ã§åŸãããã¢ã³ãµã³ãã«ã¢ãã«ã®ç©ºéãã Nåã®ã¢ãã«ããµã³ããªã³ã°ãããã® Nåã®åºåããååžã®äžæ¬¡ãäºæ¬¡ã¢ãŒã¡ã³ããæ±ããŠããã
- äºæ¬¡ã¢ãŒã¡ã³ãã倧ãã(=äžç¢ºå®æ§ãé«ã)ãš logits ãå°ãããªãããã«ããŒãžã³ãå®çŸ©ããSoftmax Loss ã«éã¿ãå ããã
- ãµã³ãã«ããšã«ã€ããŠãäžç¢ºå®æ§ãèæ
®ããŠãã
- ãµã³ãã«ã誀åé¡ãããŠããŸã確çãèšç®ãã誀åé¡ããã確çãé«ããš logits ãå°ãããªãããã« Softmax Loss ã«éã¿ãšããŠãããã
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
é¡æ€åºãã¯ãããšãã6ã€ã®ããŒã¿ã»ããã§å®éšãè¡ã£ããããŒã¿ã»ãããç°ãªãç¹åŸŽã®ãã®ã®ãããããŒã¹ã©ã€ã³ã¢ãã«ãšããŠãã¢ãã«ãéããã®ãAugmentation ã䜿ã£ãŠãããã®ãä»ã® class imbalanced ãªã¿ã¹ã¯ã®ããã® loss ã䜿ã£ãŠãããã®ãªã©å€ããšããããç®çãæã£ãŠæ¯èŒãããã®ã»ãšãã©ã§æãè¯ãæ§èœãšãªã£ãŠããã
5. è°è«ã¯ããïŒ
è«æå
ã§èšåãããŠããè°è«ã¯ãªãã
Affinity Loss ã®è«æã¯çŸåšã® SOTA ãšããŠåŒçšããŠãããæ¯èŒã¯ããŠããªãã
6. 次ã«èªãã¹ãè«æã¯ïŒ
Related Works ãšããŠæããããŠãã代衚äŸ
- [CVPR2019] Class-Balanced Loss Based on Effective Number of Samples
- Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, Serge Belongie
- https://arxiv.org/abs/1901.05555
- ãµã³ãã«æ°ã®éæ°ã weight ãšããŠãšããã€ã®çºå±ç
- é¢é£ https://qiita.com/tancoro/items/c58cbb33ee1b5971ee3b
- Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, Serge Belongie
- [2019] Max-margin Class Imbalanced Learning with Gaussian Affinity
- Munawar Hayat, Salman Khan, Waqas Zamir, Jianbing Shen, Ling Shao
- by Inception Institute of AI
- https://arxiv.org/abs/1901.07711
- Affinity Loss é¢é£ https://qiita.com/koshian2/items/20af1548125c5c32dda9
- Munawar Hayat, Salman Khan, Waqas Zamir, Jianbing Shen, Ling Shao
- [2015] Cost Sensitive Learning of Deep Feature Representations from Imbalanced Data
- Salman H. Khan, Munawar Hayat, Mohammed Bennamoun, Ferdous Sohel, Roberto Togneri
- https://arxiv.org/abs/1508.03422
- [CVPR2016] Learning Deep Representation for Imbalanced Classification
- Chen Huang, Yining Li. Chen Change Loy. Xiaoou Tang,
- éŠæž¯å€§åŠ?
- http://home.ie.cuhk.edu.hk/~ccloy/files/cvpr_2016_imbalanced.pdf
- Chen Huang, Yining Li. Chen Change Loy. Xiaoou Tang,
- [CVPR2017] SphereFace: Deep Hypersphere Embedding for Face Recognition
- Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, Le Song
- Georgia Institute of Technology
- https://arxiv.org/abs/1704.08063
- Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, Le Song
- [ICML2018] Learning to Reweight Examples for Robust Deep Learning
- Mengye Ren, Wenyuan Zeng, Bin Yang, Raquel Urtasun
- Uber, Tronto Univ.
- https://arxiv.org/abs/1803.09050
- Mengye Ren, Wenyuan Zeng, Bin Yang, Raquel Urtasun
[2019] Investigating Evaluation of Open-Domain Dialogue Systems With Human Generated Multiple References
0. è«æ
ã¿ã€ãã«
Investigating Evaluation of Open-Domain Dialogue Systems With Human Generated Multiple References
ãªã³ã¯
https://arxiv.org/abs/1907.10568
èè
Prakhar Gupta, Shikib Mehri, Tiancheng Zhao, Amy Pavel, Maxine Eskenazi, Jeffrey P. Bigham
æçš¿æ¥ä»
ACL 2019
1. ã©ããªãã®ïŒ
察話ã¢ãã«ã®ç 究ã«ãããŠã察話ãšãããã¡ã€ã³ã®æ§è³ªäžãèªåè©äŸ¡ææš (BLEU ãªã©ãèªåèšç®ã§ããææš)ãšäººéã«ããæåè©äŸ¡ãšã®çžé¢ã匱ãããšãç¥ãããŠããã
æ¬ç 究ã§ã¯ã1ã€ã®ãã¹ãããŒã¿ã«å¯ŸããŠè€æ°ã®æ£è§£ã®è¿çã人éã«ãã£ãŠçšæ (multiple reference) ãããããçšããããšã§ã¢ãã«ã®çæçµæã®è³ªãšå€æ§æ§ã«ã€ããŠè©äŸ¡ã®ãããç·©åã§ããããšãå®éšã«ãŠç€ºããã
ãŸããDailyDialog ã®ãã¹ãããŒã¿ã« multiple reference ãè¿œå ããããŒã¿ãå
¬éããŠããã
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
察話ã®è³ªãšå€æ§æ§ã®è©äŸ¡ã«ã€ããŠãè©äŸ¡æã«äººéã®å€æãå¿ èŠã ã£ãããmultiple refrerence åããã¹ãããŒã¿ãå°éãããªãããªã©ã®æ¢åç 究ã§ã®åé¡ã解決ããŠããã
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
察話ã®è³ªãšå€æ§æ§ã«ã€ããŠãmultiple reference ã®å Žåã§ã®èšç®æ¹æ³ãææ¡ããŠããã
- 質ã«ã€ããŠã¯åçŽã«äžçªã¹ã³ã¢ã®é«ããã®ãæ¡çš
- å€æ§æ§ã«ã€ããŠã¯ãæ£è§£ R = {r1, ..., rn} ãšã¢ãã«åºå Y = {y1, ..., ym} ãçšæããŠã次åŒã§èšç®ããã
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
è€æ°ã®è©äŸ¡ææšãç¹ã«åèªã®éè€ããŒã¹ã®ãã®ãš embedding ããŒã¹ã®ãã®ããããããã4ã€ã®ã¢ãã«+人éã®çºè©±ã䜿ã£ãŠæšè«ãèšæž¬ããã
single-reference ãš multi-reference ã§èšæž¬ãã人éè©äŸ¡ã«ããçµæãšã®çžé¢ãèŠãã
Quarity
Diversity
5. è°è«ã¯ããïŒ
6. 次ã«èªãã¹ãè«æã¯ïŒ
[2020] Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
0. è«æ
ã¿ã€ãã«
Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
ãªã³ã¯
https://arxiv.org/abs/2002.06305
èè
Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, Noah Smith
æçš¿æ¥ä»
(Submitted on 15 Feb 2020)
1. ã©ããªãã®ïŒ
BERT ãªã©ã® pretrain ã¢ãã«ã finetune ããéã®åŠç¿ã®ãã€ããã¯ã¹ã«é¢ããç 究ãGLUE ãæ§æããã¿ã¹ã¯ã®äžéš (MRPC, RTE, CoLA, SST) ã«ã€ããŠãã¢ãã«ã pretrained weight, ãã€ããŒãã©ã¡ãŒã¿, åŠç¿æ¹æ³ã¯åããŸãŸ seed ã ããå€ã㊠2,100 åã® finetune ãè¡ãããã®æ§è³ªãåãèšç®ãªãœãŒã¹å
ã§ç²ŸåºŠãäžããããã® early stopping ã«ã€ããŠè°è«ããŠããã
äžéšã¿ã¹ã¯ã§ã¯ seed ã ã㧠7% ãã¹ã³ã¢ãæ¹åããŠããã
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
次ã®2ã€ã® random æ§ã®ããéšåãåºå¥ã㊠seed ãå€ããŠå®éšããŠããã
- weight initialization (WI)
- training data order (DO)
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
BERT ã«ã€ããŠæ§ã 㪠seed 㧠finetune ã®å®éšããããšããã次ã®ãããªçºèŠãåŸãããã
- äžéšã¿ã¹ã¯ã§ BERT ææ¡ã¢ãã«ã®ç²ŸåºŠã©ããã XLNet, RoBERTa, ALBERT ãäžåã£ã
- RTE ãš CoLA ã¿ã¹ã¯ã«ãããŠã¯æ¢åã¹ã³ã¢ã(絶察éã§)7%ãäžåã粟床ãåŸããã
- WI ãš DO ã§ã¯ã©ã¡ããåããããæ§èœã«å¯äžããŠããããšã確èªãã
- ç¹å®ã®ããŒã¿ãœãŒã¹ã«å¯ŸããŠå ±éã§ãããªã WI, DO ã® seed ãçºèŠãã
- ä»åå®éšãããã¹ãŠã®ã¿ã¹ã¯ã«å¯ŸããŠæå¹ãšãªã WI ã® seed ãçºèŠãã
ä»ã«ã
- ããèšç®ãªãœãŒã¹å ã§ã®æåŸ æ§èœããããããã® early stopping ã®æŠç¥ã«ã€ããŠè°è«
- ä»åŸã®åŠç¿ãã€ããã¯ã¹ã«é¢ããç 究ã®ããã2,100ååã®åŠç¿ã«é¢ããããŒã¿ãå šãŠå ¬é(ãŸã ?èŠã€ãããªã)
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
5. è°è«ã¯ããïŒ
- ä»ã® pretrain ã¢ãã«ã§ãåæ§ã (XLNet, RoBERTa, ALBERT, etc.)
- Binary Classification 以å€ã®ã¿ã¹ã¯ã§ãåæ§ã
- ããŒã¿æ°ã®å€ã finetune ã¿ã¹ã¯ã§ãåæ§ã (SST ã§ã¯ variance ãæããã«å°ãããªã£ãŠãã)
- weight initialization ã§è¯ãã£ã seed æã®äžèº«ãåæããŠæ°ãã initialization scheme ã«ã€ããŠèãã
- data order shuffling ã«ã€ããŠã¯ active learning ãšã®é¢é£ãèŠãäºå®ãšã®ããšã§ãããã«ãªãã»ã©ãšããæã
6. 次ã«èªãã¹ãè«æã¯ïŒ
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
[2019] Long-Term Feature Banks for Detailed Video Understanding
0. è«æ
ã¿ã€ãã«
Long-Term Feature Banks for Detailed Video Understanding
ãªã³ã¯
http://openaccess.thecvf.com/content_CVPR_2019/papers/Wu_Long-Term_Feature_Banks_for_Detailed_Video_Understanding_CVPR_2019_paper.pdf
https://github.com/facebookresearch/video-long-term-feature-banks
èè
Chao-Yuan Wu, Christoph Feichtenhofer, Haoqi Fan, Kaiming He, Philipp KrÀhenbÌhl, Ross Girshick (FAIR)
æçš¿æ¥ä»
CVPR 2019
1. ã©ããªãã®ïŒ
Video Understanding ã«ãããŠãå€ãã®æ¢åã¢ãã«ã§ã¯2-5ç§ã®çãæéããèŠãŠããªãã®ã«å¯ŸããŠãã³ã³ããã¹ããæ£ããç解ããã«ã¯æ åå
šäœããæ
å ±ãæœåºããå¿
èŠããããšããŠãæ¢åã¢ãã«ãæ¡åŒµããããã® long-term feature bank ãææ¡ã
3D CNN ã®æ¢åã¢ãã«ã«ææ¡ææ³ãåãå
¥ããããšã§ãAVA, EPIC-Kitchens, Charades ãªã©ã® Video Dataset 㧠SoTA ãæŽæ°ããã
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
é·ãã¹ãã³ã®æ
å ±ãåãå
¥ããææ³ã¯ååšããããéå»ã®ããŒã¿ãå
ã«äœããã®ã¢ãã«ã§ç¹åŸŽéã«å€æããŠãããã¢ãã«ã«å
¥åãšããŠå
¥ãããšããææ³ãåã£ãŠãããend-to-end ã§ãªããå
ã«èšç®ããŠããç¹åŸŽéãã¿ã¹ã¯ã«ã€ããŠæé©åãããŠããªãããªã©ã®èª²é¡ãããã
ææ¡ææ³ã§ã¯2ã€ãåé¢ãã long-term feature bank ã¯ãããŸã§äºåã®ã³ã³ããŒãã³ããšããŠååšããæ¢åã¢ãã«ã«æ¡åŒµãšããŠè¿œå ã§ããããã«ãªã£ãŠããã
End-to-end ãªåŠç¿ã§ãçãã¹ãã³ã®ç¹åŸŽã匷ãèæ
®ãã€ã€ãlong-term ãªæ
å ±ãåãå
¥ããããšãåºæ¥ãç 究ã¯åã
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
Long-term Feature Bank ãå€éšã³ã³ããŒãã³ããšããããšã§ãæ¢åã¢ãã«ãæ¡åŒµããŠäœ¿ãããšãåºæ¥ãã
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
AVA, EPIC-Kitchens, Charades ã®ããŒã¿ã»ããã«å¯ŸããŠæ¢åã¢ãã«ãæ¡åŒµãã圢ã§å®éšããSOTAãæŽæ°ã
5. è°è«ã¯ããïŒ
ããé·ãã³ã³ããã¹ããå¿ èŠãªã¿ã¹ã¯ã§ããå¹æãèŠãããã®ã§ããã®ãããªããŒã¿ã»ãããæ°ãã«åºããå¹æãçºæ®ã§ãããã
6. 次ã«èªãã¹ãè«æã¯ïŒ
[2019] A BERT Baseline for the Natural Questions
0. è«æ
ã¿ã€ãã«
A BERT Baseline for the Natural Questions
ãªã³ã¯
https://arxiv.org/abs/1901.08634
ã³ãŒã: https://github.com/google-research/language/tree/master/language/question_answering/bert_joint
èè
Chris Alberti, Kenton Lee, Michael Collins (Google)
æçš¿æ¥ä»
20190124
1. ã©ããªãã®ïŒ
Natural Questions ã®ã¿ã¹ã¯ã«ã€ããŠãBERT ããŒã¹ã®æ°ããããŒã¹ã©ã€ã³ãææ¡ãããã¯ãã«ã«ããŒãã
Natural Questions 㯠long answer ãš short answer ã®2ã€ãç¹å®ããã¿ã¹ã¯ã ãã2ã€ãåæã«è§£ã BERT ããŒã¹ã®ã¢ãã«ã§å
è«æã®ããŒã¹ã©ã€ã³ã倧å¹
ã«äžåãæ§èœãšãªã£ãŠããã
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
- ããŒã¿ååŠç
- åçãå«ã document ãéåžžã«é·ã (wikipedia èšäº) ãªã®ã§ãçªå¹ 512, stride 128 (128 ãã€ååŸéãªãããã«ããã)ã§åããŒã¿ãåå²
- [CLS] question_text [SEP] åå²ãããdocument_text [SEP] (èš512)ãšããããŒã¿ãäœæ
- 1åã®ããŒã¿ããå¹³å30åã®ãµã³ãã«ãã§ãã
- short answer ãè€æ°ããå Žåãäžçªçããã®ãéžã¶
- short answer ããªã long answer ããµã³ãã«ã«å šãŠå ¥ã圢ã§ããå Žåããããæ£è§£ index ãšãã
- short ã long ããªããã° CLS ãéžæãã
- ããããããšã»ãšãã©ã negative data ã«ãªãã®ã§ã50åã® downsample ãè¡ã£ã
- çµæ 500,000 ã® 512 tokens ãµã³ãã«ãäœæ
- special tokens ãè¿œå ãã
- [Paragraph=N], [Table=N], [List=N] ããããããã©ã°ã©ããããŒãã«ããªã¹ãã®æåã«è¿œå ãã
- æåã®æ°ãã©ã°ã©ãã«åçãããããšãå€ãããšãã
- answer type ã5çš®é¡ã«åé¡
- short: short answer ããã
- yes/no: yes/no aswer ããã
- long: long answer ããããshort, yes/no ããªã
- no-answer: ã©ãããªã
- ã¢ãã«
- æšè«
- 1ã€ã®ããã¥ã¡ã³ããåãµã³ãã«ã«åå²ãããŠããã®ã§ãããã¥ã¡ã³ãã§åãµã³ãã«ã®æšè«ãéããŠã(s, e) ã«ã€ã㊠rank ä»ããè¡ã
- rank ã®ããã® score ã¯æ¬¡ã®åŒã§èšç®ãã
- short answer ãæšè«ããå Žåã® long answer ã¯ãshort answer ãå«ã DOM ã® top level ã® start, end ãæå®ãã
- long answer ã®ã¿ or no-answer ã®åé¡ã® threshold ã¯ãofficial ã®ããŒã¹ã©ã€ã³ã®ã¹ã¯ãªããã䜿çšãã
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
Natural Questions ã®ããŒã¹ã©ã€ã³ã¢ãã«ãšæ¯èŒãã倧ãã粟床åäžããã
5. è°è«ã¯ããïŒ
ãããŸã§ããŒã¹ã©ã€ã³ã¢ãã«ãªã®ã§ããŸã ãŸã ã¢ã€ãã£ã¢ã¯ããã£ãœãã
- åžžã« short answer ã® span ãåœãŠãã®ã§ã¯ãªãã yes/no ãäºæž¬ãã
- è€æ°ã® short answer ã® span ãäºæž¬ã§ããããã«ãã
6. 次ã«èªãã¹ãè«æã¯ïŒ
- Natural Questions: a Benchmark for Question Answering Research
- 2019 by Google Research
- ããŒã¿ã»ããææ¡ & ããŒã¹ã©ã€ã³ææ¡ã®è«æ
[2019] Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answering
0. è«æ
ã¿ã€ãã«
Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answering
ãªã³ã¯
https://arxiv.org/abs/1908.08167
èè
Zhiguo Wang, Patrick Ng, Xiaofei Ma, Ramesh Nallapati, Bing Xiang
æçš¿æ¥ä»
EMNLP 2019
1. ã©ããªãã®ïŒ
OpenDomain 㪠QA ã®ã¿ã¹ã¯ã¯ BERT ã®ç»å Žã§å€§ããæ¹åãããããæ¢åææ³ã¯äžã€ã®è³ªåã«å¯ŸããŠè€æ°ã®åè£æãããããç¬ç«ã«è³ªå-åè£æãã¢ãšããŠåŠç¿ããããããã§ã¹ã³ã¢ã®é«ãåçãéžæããŠããã
æ¬ç 究ã§ã¯ãæ¢åææ³ã§ã¯åè£ãšãªãæç« å
šäœã§ã®ã¹ã³ã¢ãé©åã«èšç®ã§ããªããªã©ã®çç±ãããå
šãŠã§ Normalize ãã¹ã³ã¢ãèšç®ãããªã©ã®å·¥å€«ãå ãã Multi-passage BERT ãææ¡ã
OpenSQuAD, TriviaQA, Quasar-T, SearchQA ã§å
šãŠã® SoTA ãæŽæ°ããã
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
å šãŠã®åè£æç« ããåçãäºæž¬ãã global normalization ãããã¢ãã«ãå®å®åãããããšãããã®ä» ranker ã®è²¢ç®ãåè£æã® stride ãªã©ã®å¹æãããã€ãã®å®éšã«ãã確èªããã
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
éåžžã® BERT ã«ãã QA ã¿ã¹ã¯ã§ã¯ã質å + åè£æãã¢ãã¢ãã«ã«å
¥åããåçã®éå§ãšçµäºäœçœ®ãäºæž¬ãããåè£æãã¢ãè€æ°ããå Žåãããããç¬ç«ã«ãã®åŠçãè¡ããæãã¹ã³ã¢ã®é«ããã®ãéžã¶ã
Multi-passage BERT ã§ã¯ãå
šãŠã®åè£æã«ã€ã㊠logits ãèšç®ããããšã«ãå
šãŠã®åè£æ㧠Softmax ãåãããšã§ãå
šäœã§ã®ã¹ã³ã¢ãèšç®ããã
ä»ã«ããååè£æã® ranker ãåæ§ã«å
šäœã§ã¹ã³ã¢ãäœæãããstride ãå°å
¥ãããªã©ããŠã¢ãã«ãæ¹åã§ããã
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
OpenSQuAD, TriviaQA, Quasar-T, SearchQA ã®4ã€ã®ããŒã¿ã»ããã§æ€èšŒã
å
šãŠã®ããŒã¿ã»ãã㧠SoTA ãæŽæ°ããã
5. è°è«ã¯ããïŒ
ä»åŸ passage éã®çžé¢ãåãå ¥ããããšã®ããš.
6. 次ã«èªãã¹ãè«æã¯ïŒ
- Simple and Effective Multi-Paragraph Reading Comprehension
- https://arxiv.org/abs/1710.10723
- Christopher Clark, Matt Gardner, ACL 2018
- multi paragraph 㪠QA ã«é¢ããç 究ã global normalizing ã®ãšããã§åŒçšãããŠãã
ð[2019] Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
0. è«æ
ã¿ã€ãã«
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
ãªã³ã¯
https://arxiv.org/abs/1901.05415
èè
Braden Hancock, Antoine Bordes, Pierre-Emmanuel Mazaré, Jason Weston
Stanford & FAIR
æçš¿æ¥ä»
ACL 2019
arxiv 2019/01/16
1. ã©ããªãã®ïŒ
ãããã€åŸã®å¯Ÿè©±ã¢ãã«ãèªèº«ã®å¯Ÿè©±ããåŠç¿(self-feeding)ããæ¹æ³ãææ¡ããç 究ããããã€åŸã®ãŠãŒã¶ãšã®å¯Ÿè©±ãããŠãŒã¶ã®æºè¶³åºŠãäºæž¬ããããšã§ãè¯ããã®ã«ã€ããŠã¯åŠç¿ããŒã¿ãšããæªã察話ã«ã€ããŠã¯ãŠãŒã¶ã«ãã£ãŒãããã¯ããããããã«ããã
PERSONACHAT ã® 131k ã®ããŒã¿ã§åŠç¿ããäžã§å®éšãè¡ã£ããšãããself-feeding ãè¡ãããšã§å€§ããæ§èœåäžããã
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
5. è°è«ã¯ããïŒ
6. 次ã«èªãã¹ãè«æã¯ïŒ
[2019] Max-margin Class Imbalanced Learning with Gaussian Affinity
0. è«æ
ã¿ã€ãã«
Max-margin Class Imbalanced Learning with Gaussian Affinity
ãªã³ã¯
https://arxiv.org/abs/1901.07711
èè
Munawar Hayat, Salman Khan, Waqas Zamir, Jianbing Shen, Ling Shao
æçš¿æ¥ä»
20190123
1. ã©ããªãã®ïŒ
Class-Imbalanced ãªã¿ã¹ã¯ã«å¯Ÿããæ°ããæ倱é¢æ° Affinity Loss ãææ¡ããã
è¿å¹Žæ³šç®ãããŠãã max-margin loss ç³»ã®ææ³ã¯ãç¹åŸŽç©ºéãè¶
çé¢äžã®å€æ§äœãšããä»®å®ã眮ã㊠angular distance (cosine similarity) ã䜿ã£ãŠãããé¡èªèã¿ã¹ã¯ä»¥å€ã®æ±åæ§èœã®äœäžã«ã€ãªãããšããŠããã
ææ¡ææ³ã§ã¯ãé¡äŒŒåºŠãã¬ãŠã¹ã«ãŒãã«ã§å®çŸ©ããããåºã«ãã¹ãèšèšããããšã§ãããŒãžã³ãæ瀺çã«å
¥ããããšãã§ãããã€æé©åã®éçšã§ã¯ã©ã¹ã¿ãªã³ã°ãåæã«è¡ããããã«ãªããçµæçã«ã¯ã©ã¹å
åæ£ã¯å°ãããã¯ã©ã¹éè·é¢ãé¢ããããã«è¡šçŸãåŠç¿ã§ããã
å³ã¯ MNIST ã«ãããŠã0~4 ã10%ã®ãµã³ãã«ã§åŠç¿ããããšãã®ç¹åŸŽã®ååžã
å·Šã® Softmax ã®å Žåã¯ãã¯ã©ã¹å
åæ£ã¯ããŒã¿æ°ã®å€ãã¯ã©ã¹ã倧ãããªã£ãŠãããã¯ã©ã¹ã¿ãªã³ã°ãåæã«è¡ã£ãã®ã**ã ããäžéšã®ã¯ã©ã¹ã**äžã«åºãŸã£ãŠããŸã£ãŠãããå³ã®ææ¡ææ³ã§ã¯ã¯ã©ã¹å
åæ£ã¯äžå®ã§ããã€åããããã®è·é¢ã§ã¯ã©ã¹ããšã«ååžããŠããã®ãåããã
å€ãã®å®éšã§æ¢åææ³ã«æ¯ã¹ãŠæ§èœãæ¹åããã
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
è¿å¹Žè¡ãããŠãã hypersphere ã®è¡šé¢äžã«ååžãå¶éããããŒãžã³ç³»æ倱é¢æ°ãšã¯ç°ãªãã¢ãããŒãã«ããææ¡ã
åé¡ãšã¯ã©ã¹ã¿ãªã³ã°ãåæã«ããŸãè¡ãããæ倱é¢æ° Affinity Loss ãå®çŸ©ããã
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
Affinity Loss
- Softmax loss ã䜿ãããå®çŸ©ããé¡äŒŒåºŠã«åºã¥ããŠãã¹ãèšç®ããŠããã
- ææ¡ãããŠããé¡äŒŒåºŠã¯ãç¹åŸŽ f ãšåã¯ã©ã¹ã®ä»£è¡šãã¯ãã« w ã®ã¬ãŠã¹ã«ãŒãã«ã«ãããã®ã
- æ倱é¢æ°ã¯ãã¿ãŒã²ãããšãã®ä»ã®ä»£è¡šãã¯ãã«ãšã®é¡äŒŒåºŠã®å·®ã«ããŒãžã³ λ ãå ãããã®
Diversity Regularizer
- äžèš Affinity Loss ã«å ããŠã diversity regularizer ãšåŒãã§ããã¯ã©ã¹éè·é¢ãäžå®ã«ããããã®æ£ååé ãè¿œå ããŠããã
Multi centered learning
- 代衚ãã¯ãã«ã1ã€ã ãšãè€éãªåœ¢ç¶ã®ååžãåŠçã§ããªãã
- è€æ°ä»£è¡šãã¯ãã«ãããããããšã§ããã«å¯Ÿå¿ããããããã©ãã«ãã€ãºãžã®å¯Ÿå¿ã«ããªãã
- å·®åã¯é¡äŒŒåºŠã®èšç®åŒãå°ãå€ããã ããm åã®ä»£è¡šãã¯ãã«ãåã¯ã©ã¹ã§æã€ãšãããšã
- diversity regularizer ã¯ããã¹ãŠ (m à C) ã®ä»£è¡šãã¯ãã«ã«ã€ããŠèšç®ããã
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
DIL ã«ããç®èç
æ€ç¥ã¿ã¹ã¯ãé¡èªèã® LFW, YTF, CFP, AgeDB ãªã©ã§ SoTA ãæŽæ°ã
MNIST ã人工çã« Imbalance ã«ããããŒã¿ã»ãã (0-4 ãæžãã) ã§æ§ã
ãªå®éšãè¡ããææ¡ææ³ã®æå¹æ§ã確èªããã
å
·äœçã«ã¯ Softmax loss ãšæ¯èŒã㊠Imbalance ã§ããã»ã©é«ãåé¡ç²ŸåºŠããã€ãºã人工çã«å«ããããŒã¿ã§å®éšãããã¹ãæ§ãæ¹åãªã©ã
ablation study ãšã㊠multi centered ã¢ãã«ã§ã® m ãååžã®åºãããè¡šã Ï ãå€åãããå®éšãè¡ã£ãŠããã
5. è°è«ã¯ããïŒ
è«æäžã§ã¯ç¹ã«ãªãã
ã¡ã€ã³ã®å®éšã¯ MNIST ã§ãä»ã®ããŒã¿ã§ã確èªããããšããã
ç®èãšé¡èªèã§ã¯ãã€ããŒãã©ã¡ãŒã¿ã§ãã m ãš Ï ãç¹ã«èšèŒãããŠããããåçŸæ§ãæªããã
6. 次ã«èªãã¹ãè«æã¯ïŒ
[2019] SG-Net: Syntax-Guided Machine Reading Comprehension
0. è«æ
ã¿ã€ãã«
SG-Net: Syntax-Guided Machine Reading Comprehension
ãªã³ã¯
https://arxiv.org/abs/1908.05147
èè
Zhuosheng Zhang, Yuwei Wu, Junru Zhou, Sufeng Duan, Hai Zhao, Rui Wang
æçš¿æ¥ä»
AAAI-2020
1. ã©ããªãã®ïŒ
èšèªèªè§£ã¿ã¹ã¯ã«ãããæ¢åã® attention ããŒã¹ã®ã¢ãã«ã¯ãäžèŠãªåèªã«ã€ããŠã attend ããŠããŸããself attention network ã« syntax ã䜿ã£ãå¶çŽãè¿œå ãã SDOI-SAN (syntactic dependency of interest - self attention network) ãææ¡ããBERT ããŒã¹ã®ã¢ãã«ã«é©çšããã
SQuAD 2.0, RACE ã®ã¿ã¹ã¯ã§æ§èœãæ¹åã
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
syntax æ å ±ã self-attention ããŒã¹ã®ã¢ãã«ã«çµã¿èŸŒãæ¹æ³ãææ¡ããMRC ã¿ã¹ã¯ã«å¿çšããã
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
Self Attention Network (SAN) ã«å¯Ÿã㊠syntactic dependency of interest (SDOI) ã®æ
å ±ãä»å ãã SAN-SDOI ã¬ã€ã€ãŒãææ¡ããã
BERT ãªã©ã® Transformer ããŒã¹ã® Encoder ã®åºåã«å¯Ÿã㊠SAN-SDOI ã¬ã€ã€ãŒãè¿œå ããå
ã
ã® Encoder ã®åºå H ãšãSAN-SDOI ã¬ã€ã€ãŒã®åºå H' ã®éã¿ä»ãåãæçµåºåãšããã
SAN-SDOI ã¬ã€ã€ãŒã§ã¯ããããã¯ãŒã¯æ§é ã¯éåžžã® SAN ãšåæ§ã§ self-attention + FFN ã®æ§æãšãªã£ãŠããããself-attention å
ã®ãã¹ã¯ãç°ãªã£ãŠãããå¥ã§çšæãã syntax parser ããåŸãããæšæ§é ãããšã«ãã¹ã¯ãäœæãé©çšããŠããã
éåžžã® self-attention ãš SDOI ã§ã® attention ã®å€ã¯æ¬¡ã®å³ã®ããã«å€§ããå€ããã
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
BERT ããŒã¹ã®ã¢ãã«ã§ãspan based ãªã¿ã¹ã¯ãšã㊠SQuAD 2.0, multi choices ãªã¿ã¹ã¯ãšã㊠RACE ã§å®éšãããããããŒã¹ã©ã€ã³ã¹ã³ã¢ãäžåã SoTA ãããã¯ããã«è¿ãã¹ã³ã¢ãç²åŸã
åæã«ãããããŒã¹ã©ã€ã³ãšãªã BERT ãããé·ã Question ã«å¯Ÿãã粟床ãäžãã£ãŠãããSyntax ã«ããæ
å ±ãå¹ããŠããããšã確èªããã
5. è°è«ã¯ããïŒ
é·ãæç« ã§ã¯ãattention ããŒã¹ã®ã¢ãã«ã§ã¯äžèŠãªåèªãããèŠãŠããŸããšãã仮説ããææ¡ããææ³ã§ãã£ãããå®éšçã«ææ¡ææ³ã®æ¹ãé·ã Query ã«å¯ŸããŠã粟床ãå£åããªãããšã瀺ããŠããã
syntax æ
å ±ãçŽæ¥çã«çµã¿èŸŒãããšã§ç²ŸåºŠåäžãã§ããäŸãšãªã£ãã
6. 次ã«èªãã¹ãè«æã¯ïŒ
[2019] Learning Loss for Active Learning
0. è«æ
ã¿ã€ãã«
Learning Loss for Active Learning
ãªã³ã¯
https://arxiv.org/abs/1905.03677
http://openaccess.thecvf.com/content_CVPR_2019/papers/Yoo_Learning_Loss_for_Active_Learning_CVPR_2019_paper.pdf
èè
Donggeun Yoo, In So Kweon
æçš¿æ¥ä»
CVPR 2019
1. ã©ããªãã®ïŒ
loss ãæšæž¬ããã¢ãã«ãåŠç¿ããunlabeled ãªããŒã¿ã®äžã§ãæšå®ãã loss ã®å€§ããããŒã¿ã«åªå
çã«ã¢ãããŒã·ã§ã³ãè¡ãããšã§ãactive learning ãå¹ççã«è¡ãææ³ã®ææ¡ã
ã¿ã¹ã¯äŸåæ§ããªãããã€èšç®éã®å¿
èŠã«ãªã Deep ãªãããã¯ãŒã¯ã«ã€ããŠã察å¿ã§ããæ±çšçãªææ³ã§ãImage Classification, Human Pose Estimation, Object Detection ã®3ã€ã®ç°ãªãã¿ã¹ã¯ã«ã€ããŠææ°ã¢ãã«ã§å®éšããæ¢åææ³ãäžåãæ§èœãšãªã£ãã
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
ã¿ã¹ã¯äŸåãªãã§ããã€èšç®éã倧ãã Deep ãªãã¢ãã«ã«ãé©çšã§ãã Active Learning ææ³ã¯åã
Loss ã®å€§å°ã Active Learning ã«å¿çšããç 究ã¯åã
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
3ã€ã®ã¿ã¹ã¯ã§ææ°ã®ã¢ãã«ã䜿ã£ãŠæ¢åææ³ãšæ¯èŒå®éšã
3ã€ã®ã¿ã¹ã¯ã¯åé¡ã¿ã¹ã¯ã« Image Classification, ååž°ã¿ã¹ã¯ã« Human Pose Estimation, ãã€ããªããã¿ã¹ã¯ãšã㊠Object Detection ãéžæã
å
šãŠã§æ¢åææ³ãäžåãæ§èœãšãªã£ãã
5. è°è«ã¯ããïŒ
ãŸã Loss ãæšå®ããã¢ãã«ã® accuracy ã«èª²é¡ããããè€éãªã¢ãã«ã ãšãããããããé£ããããã粟床ããããå¿
èŠãããã
ãŸããdiversity ãªã©ä»ã®èŠçŽ ãçµã¿åããã Active Learning ææ³ã«ã€ããŠãä»åã¯èæ
®ããŠããªãã
ããŒã¿ã®ååžãèæ
®ããããšã§ loss accuracy ãåäžããããšãåºæ¥ãããã
6. 次ã«èªãã¹ãè«æã¯ïŒ
[2019] Generating Logical Forms from Graph Representations of Text and Entities
0. è«æ
ã¿ã€ãã«
Generating Logical Forms from Graph Representations of Text and Entities
ãªã³ã¯
https://arxiv.org/abs/1905.08407
èè
Peter Shaw, Philip Massey, Angelica Chen, Francesco Piccinno, Yasemin Altun
æçš¿æ¥ä»
ACL 2019
1. ã©ããªãã®ïŒ
æ§æ解æã®ã¿ã¹ã¯ã«ãããŠã Transformer ã® self-attention layer ã Graph Neural Network ãçšããããšã§ãäžããããèªç¶èšèªå
ã® entity ããã®é¢é£æ
å ±ã®ç解ãæ¹åããææ³ãææ¡ããã
decode æã« copy mechanism ã䜿çšã㊠entity ãå«ãã è«ç圢åŒãçæããã¢ãã«ãäœæã3ã€ã®æ§æ解æã¿ã¹ã¯ã§ pre-training ãªã㧠SoTA ã«å¹æµãæŽã« BERT ãšçµã¿åãããããšã§å€ãã®æ¢åææ³ãäžåãã¹ã³ã¢ãšãªã£ãã
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
- Transformer ã« GNN ãçµã¿åãããäŸã¯å?
- æ§æ解æã¿ã¹ã¯ã§ã Entity ãšãã® relation ã GNN ãçšããŠã·ã³ãã«ã«çµã¿åãããææ³ãææ¡ããã
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
Transformer ã® sub-layer ã§ãã self-attention layer ããGNN ãçšããŠæ¹åããŠããã
- node ãš edge label ã®é¢æ° f ãèãã
- å node ã u, å edge ã® label ã r ãšããŠãå€æåŸã®è¡šçŸã¯
- ãã㧠α 㯠attention weights ã§ã次㮠s ã® softmax ããšã£ããã®
- éåžžã® Transformer ã® self-attention ã¯ãW^r ããã©ã¡ãŒã¿ã®è¡åãšããŠäžèšãšãããšåãã
- ããããrelation ãçšãã2ã€ã®ææ³ãææ¡ã
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
3ã€ã®æ§æ解æã¿ã¹ã¯ãGEO, ATIS, SPIDER ã§å®éšã
GNN ãçšããææ¡ææ³ãæ¢åææ³ã® SoTA ãšã»ãŒåçã®ã¹ã³ã¢ãšãªããBERT ã«ãã pre-train ãçµã¿åãããã¢ãã«ã¯æŽã«ãããäžåãã¹ã³ã¢ãšãªã£ãã
5. è°è«ã¯ããïŒ
- decoder ã« constraint ãå ããããšã§æŽã«è¯ããªããã
- ä»åã®ææ³ã¯ã°ã©ãæ§é ãæã€ãããªå€éšç¥èãçæã«çµã¿åãããæ¹æ³ãšããŠè¯ããããããªã
- å°ãªããšã semantic parsing ã«ã¯è¯ãã£ãäºãããã£ã
6. 次ã«èªãã¹ãè«æã¯ïŒ
- Graph Attention Networks
- https://arxiv.org/abs/1710.10903
- Petar VeliÄkoviÄ, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, Yoshua Bengio
- ICLR 2018
- Graph ãš Attention çµã¿åããã®ææ¡è«æ (Convolution ã®ä»£æ¿)
- Self-Attention with Relative Position Representations
- https://arxiv.org/abs/1803.02155
- Peter Shaw, Jakob Uszkoreit, Ashish Vaswani
- NAACL 2018
- relative positional representation ã«é¢ããææ¡è«æãæ¬ç 究ã§ã¯ edge label ãèããéã«äœ¿çšãããŠããããã«èŠããã
[2019] Explain Yourself! Leveraging Language Models for Commonsense Reasoning
[2020] BLEURT: Learning Robust Metrics for Text Generation
0. è«æ
ã¿ã€ãã«
BLEURT: Learning Robust Metrics for Text Generation
ãªã³ã¯
https://arxiv.org/abs/2004.04696
èè
Thibault Sellam, Dipanjan Das, Ankur P. Parikh
æçš¿æ¥ä»
ACL 2020
1. ã©ããªãã®ïŒ
Neural Language Generation (NLG) ã®ããã®ãBERT ã䜿ã£ãæ°ããªè©äŸ¡ææšã§ãã BLEURT ãææ¡ãè©äŸ¡ãé£ãããšèšãããŠããããã¹ãçæã«ã€ããŠãã人éã«ããè©äŸ¡ããã¢ãã«åããããã«å€§éã®ããŒã¿ã§ BERT ã2段éã« pre-train ãããã®ã䜿çšããã
backtranslation ãªã©ã«ãã倧éã®ç䌌ããŒã¿ã䜿ã£ã9çš®é¡ã® pre-train ææ³ãçµã¿åãããããšã§ã¢ãã«ãäœæããWMT Metrics Shared Task 㧠SoTA ã¹ã³ã¢ãç²åŸã
衚㯠NLG metrics åãã® finetune ã«äœ¿çšãã signal (=target). ããããçšã㊠BERT ãæŽã« pre-train ããæåŸã«å°éã®äººéè©äŸ¡ã®ããŒã¿ã§ finetune ããã
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
- BERT ãæŽã« NLG metrics ã®ããã« pre-train ãè¡ããå€æ¿ã®ãããªç¶æ³ã«ãèãããããã¹ããªã¢ãã«ãææ¡ããã
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
- åŠç¿ã®ããã® reference ãš prediciton ã®2ã€ã®ç䌌çãªæãã¢ã®çæ
- BERT ã® MASK: MASK -> LM ã§çœ®ãæããããã®ã prediction ãšããŠäœ¿ã
- Backtranslation: English -> SomeLanguage -> English ãšãããã®ã prediction ãšããŠäœ¿ã
- Dropping words: äžéšåèªãèœãšãããã®ã prediction ãšããŠäœ¿ã
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
人éè©äŸ¡ãšã®å·®å
2017, 2018, 2019 ã® WMT Metric Shared Task ã§ã人éã«ãã rating ãšåææ¡ææšã®å·®ãèŠããå€ãã®æ¢åææ³ãšãšãã«å®éšãmetric ã¯ãã®å¹Žã®å
¬åŒã®ãã®ãšãå幎床ã§äžè²«æ§ãèŠããã Kendall's Tau ã®2ã€ã® metric ã䜿çšã
ææ¡ææ³ã¯ãBERT-base, BERT-large ããäœæãããã® (BLUERT-base, BLUERT)ãšããããããã pre-traing ãæãããã® (-pre) ã®4ã€ã
2017, 2018 ã§ã¯ææ¡ææ³ãæãè¯ãã¹ã³ã¢ãç²åŸã
2019 ã§ãã»ãšãã©ã§ TOP ã®ã¹ã³ã¢ãç²åŸã幎床ãæ°ããã»ã©åŠç¿ããŒã¿ãå€ãã®ã§ãå€ã幎床ã®ãã®ã¯ pre-training ã®å¹æãããå¹ããŠãããšèŠãããã
Robustness
pre-training ã«ãã robustness ãæ¹åããŠãããã©ããã確èªãããããtrain/valid 㯠low-rating, test 㯠high-rating ãªããŒã¿ãšãªãããã«æå³çã«ååžãåããŠãµã³ããªã³ã°ããåŠç¿ã»å®éšãpre-training ã«ããååžã®åã(å€æ¿ã«è¿ãã·ãã¥ãšãŒã·ã§ã³)ã«å¯ŸããŠããã¹ãã«ãªã£ãŠããããšã確èªããã
ãŸããBERT, synthetic, WMT ã®ããŒã¿ã§ 3 step ã§åŠç¿ãããã¢ãã«ã§ã¯ãå¥ã®ã¿ã¹ã¯ã§ãã WebNLG ã«ãããŠåŠç¿ããŒã¿ãªãã§ãæ¢åææ³ãäžåãé«ãã¹ã³ã¢ãåºããã
5. è°è«ã¯ããïŒ
- pre-train æã®ã©ãã«ã« BLUE ã ROUGE ãå
¥ãããã©ãã
- Ablation Study ã§ãå
ã
人éè©äŸ¡ãšã®çžé¢ãäœãã£ã BLEU ã ROUGE ã pre-train æã®ã©ãã«ãšããŠå ããªãã»ããã¹ã³ã¢ãäžããããšãåãã£ã
- (å®éã« WMT ã§å®éšãããšãæãããšãã®ã»ãã粟床æ¹å)ã
- å°æ¥çã« BLUE ã ROUGE ãšçžé¢ããã¿ã¹ã¯ãåºãéã«æ§èœãäžãããããããªã
- Ablation Study ã§ãå
ã
人éè©äŸ¡ãšã®çžé¢ãäœãã£ã BLEU ã ROUGE ã pre-train æã®ã©ãã«ãšããŠå ããªãã»ããã¹ã³ã¢ãäžããããšãåãã£ã
- multilingual NLG ããBLUERT ãçµã¿åããã hybrid ãªææ³ã¯ Future work.
6. 次ã«èªãã¹ãè«æã¯ïŒ
- BERTScore: Evaluating Text Generation with BERT
- https://arxiv.org/abs/1904.09675
- ããŒã¹ã©ã€ã³ã¢ãã«ãšããŠæ¯èŒãããŠããææ³ããã¹ãŠã®å®éšã§æ¯èŒãããŠããã
- BLUE ã® hard-ngram overlap ã BERT ã® embedding ã䜿ã£ãŠçœ®ãæãããã®ãšã®ããš
- MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance
- https://arxiv.org/abs/1909.02622
- ããŒã¹ã©ã€ã³ã¢ãã«ãšããŠæ¯èŒãããŠããææ³ãBERT ã® embedding ã«å¯Ÿã㊠Earth Mover Distance ã§é¡äŒŒåºŠãå³ãã
[2020] Towards a Human-like Open-Domain Chatbot
0. è«æ
ã¿ã€ãã«
Towards a Human-like Open-Domain Chatbot
ãªã³ã¯
https://arxiv.org/abs/2001.09977
èè
Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, Quoc V. Le
æçš¿æ¥ä»
2020/01/27
1. ã©ããªãã®ïŒ
人éã«ããäŒè©±è©äŸ¡ã®åæãè¡ãã人éè©äŸ¡çšã®ææšã§ãã SSA (Sensibleness and Specificity Average) ãææ¡ãããããã¢ãã«ã® perplexity ãšçžé¢ãããããšãå®éšã«ãã確èªããã
perplexity ã«ã€ããŠæé©åãè¡ã£ãã¢ãã«ãšããŠããã«ãã¿ãŒã³ã®éè« chatbot ã§ãã Meena ãææ¡ãææ¡ææšã§ãã SSA ã§æ¢åã®éè«ã¢ãã«ã倧ããäžåãã¹ã³ã¢(72%)ãšãªã£ãã
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
倧èŠæš¡ããŒã¿(çŽ341GBã®ããã¹ã)ã»å€§èŠæš¡ã¢ãã«(2.6B ã®ãã©ã¡ãŒã¿ãæ〠Evolved Transformer)ã§ã人éã«ããäŒè©±è©äŸ¡ã®åæã«åºã¥ããŠè¡ã£ãåŠç¿ã¢ãã«ã§æ¢åã¹ã³ã¢ã倧ããæŽæ°ããã
perplexity ã®äœãã¢ãã«ãè¯ã人éè©äŸ¡ã«çžé¢ããããšãå®éšã«ãã瀺ããã
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
- 人éè©äŸ¡ã®åæãã SSA ã®ææ¡ãåã³çžé¢ããèªåè©äŸ¡ææšãšã㊠perplexity ã®éžæã
- Enc 1å±€ãDec 13局㮠Evolved Transformer ã䜿çšããŠãè¶ å€§èŠæš¡åŠç¿ (åŸè¿°) ãè¡ã£ãã¢ãã«ã
- decoding ã工倫ãæ¢åã§äž»èŠã«äœ¿ãããŠãã beam search ã§ã¯ãªããã·ã³ãã«ãª sampling-and-rerank ã®ææ³ã§è¡ãã
- softmax èšç®æã« temperature T ãå ããŠååžã調æŽããN件ã®åè£ããµã³ããªã³ã°åŸãã¹ã³ã¢ãªã©ã§ re-rank ãè¡ã
- Meena (full) ãšããŠæŽã« decoding ãæ¹åããµã³ããªã³ã°ã« top-k ã䜿çšããtemperature ã調æŽããäžã§ãåããããªåãç¹°ãè¿ã(cross-turn repetition) ã軜æžããããã«ã«ãŒã«ã§åè£ãåé€ã
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
- æ¢å chatbot ã¢ãã«ãšäžç·ã«äººéã«ããè©äŸ¡ãè¡ããMeena ãæãé«ã SSA ãç²åŸã人éã¹ã³ã¢ã§ãã 86% ã«å¯Ÿã㊠Meena (base) ã 72%, Meena (full) ã 79% ãšãããªã人éã«è¿ãã¹ã³ã¢ãšãªã£ãã
5. è°è«ã¯ããïŒ
- 劥åœãªãã¹ãã«ãªã£ãŠãããã©ãããstatic test ãš interactive test ãè¡ã£ãããã©ã¡ãããŸã 課é¡ãããã
- static test 㯠context ã 1-3 ãšå°ãã
- interactive test ã¯ä»ã¢ãã«ãšã®æ¯èŒããã¥ããããŸã14-28ã¿ãŒã³ã¯ãŸã å°ãã (æšæ¶ãäžéšãå ããã®ã§ãæ·±ããããã¯ã«ã€ããŠã¯è©±ããªã)
- ææšãšã㊠Sensibleness ãš Specificity ã«ãŸãšããããããããã¹ããšããããã§ã¯ãªã
- ä»ã«ãå€ãã®ææšãèãããã (humor, empathy, ...)
- Sensibleness ãå解ããŠãã£ãšçŽ°ããèŠãããšãã§ãã (logical, personal consistency, ...)
6. 次ã«èªãã¹ãè«æã¯ïŒ
- Evolved Transformer
- Top K ãªã© decoding strategy åšãã®è«æ
[2020] MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance
0. è«æ
ã¿ã€ãã«
MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance
ãªã³ã¯
https://arxiv.org/abs/1909.02622
https://github.com/AIPHES/emnlp19-moverscore
èè
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, Steffen Eger
æçš¿æ¥ä»
EMNLP 2019
1. ã©ããªãã®ïŒ
ããã¹ãçæã®ããã®æ°ããªè©äŸ¡ææšã§ãã MoverScore ãææ¡ã
çŸåšããèªåè©äŸ¡ææšã§ãããåèªã n-gram ã§äžèŽãããã©ãããšãã£ã BLEU ã ROUGE ã¯è¡šé¢çãªé¡äŒŒåºŠã«åºã¥ããŠããããå®éã¯æå³çãªæ¯èŒãããã¹ããšããèãã§ãåææã«åºããã BERTScore ãšåãã BERT ãªã©ã® contextualized embedding ãçšããäžæ¹ã§ãã®ææ³ãäžè¬åããé¡äŒŒåºŠèšç®ã« Earth Mover Distance ãçšããã
4ã¿ã¹ã¯ã§ã®å®éšã§æ¢åã®èªåè©äŸ¡ææšãš BERTScore ãšã®æ¯èŒãè¡ããSoTA ãããã¯ã»ãŒããããšåãã¹ã³ã¢ãç²åŸããŠããã
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
Earth Mover Distance ãããã¹ãå¿çšãã WMD (Word Mover Distance) ãšãBERT ãå§ããšãã contextualized embedding ãçµã¿åãããŠããã¹ãçæã®è©äŸ¡ææšãšããã
ã©ã®ç²åºŠã§é¡äŒŒåºŠãåãããªã©åºç¯å²ã«å®éšã
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
Word Mover Distance
åé¡ã次ã®ããã«å®åŒåã
x^n, y^n ãããããã®æç« ã® n-gram, f_{x^n}, f_{y^n} ãããããã®æã® n-gram ã®éã¿ (idx ã䜿ã£ãŠãã) ãšããŠã
ãã㧠C 㯠d ãé©åœãªè·é¢é¢æ°ãšã㊠ã§äžããããè¡åã§ã
ãC, Fã
㯠C ãš F ã®èŠçŽ ããšã®ç©ã®ããã¹ãŠã®èŠçŽ ã®åã
ãªã®ã§ããã¹ãŠã®åèªã® n-gram ã®çµã¿åããã§è·é¢ãèšç®ããéã¿ä»ããããã®ã®åèšã WMD ã
distance
ããã§ã¯ E(x)
ã BERT ãªã©ã® embeddings ãšããŠã次ã®ããã«ãŠãŒã¯ãªããè·é¢ã䜿ã£ãŠããã
n-gram ã®å Žå㯠idf ã§éã¿ä»ãå¹³åãåã£ãŠããã
n-gram ã® WMD ã®ç¹æ®äŸãšããŠãn = sentence length ã®å Žåã® SMD (Sentence Mover Distance)ã掟çãšããŠå®çŸ©ããŠæ¯èŒããŠããã
BERTScore ãšã®éã
Word Mover Distance ã®å®çŸ©ã«ãããŠã C ã BERT ããŒã¹ã® cos sim. ã«åºã¥ããã®ã«ãFãæé©ååé¡ã解ããã«äžæ§ååžã«ãããš BERTScore ã«ãªãã
èè ã㯠BERTScore ãšã®éããšããŠãHard / Soft Alignments ãäžããŠãããBERTScore 㯠candidate æã®åèªãã reference æã§æãé¡äŒŒåºŠã®é«ãåèªã«ã€ããŠã®ã¹ã³ã¢ã䜿ãã®ã«å¯ŸããMoverScore ã¯å šãŠã®çµã¿åããããè·é¢ãèŠãŠãããããã«ããæå³çã«è¿ãåèªãããŸãçŽä»ãããããšããŠããã
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
4ã€ã®ã¿ã¹ã¯(machine translation, èŠçŽ, image captioning, data-to-text)ã§ä»ã®ããŒã¹ã©ã€ã³ææšãå«ããŠäººéè©äŸ¡ãšã®çžé¢ãæ¯èŒã
å
šãŠã¿ã¹ã¯ã§ãMNLI 㧠finetune ãã BERT ã䜿ã£ããã®ã SoTA ãããã¯ããã«è¿ãæ§èœãšãªã£ãã
BERTScore ãšã®æ¯èŒã§ãåãæ¡ä»¶ã§ (power means ãªã©ã䜿ããã«) æ¯èŒãè¡ã£ãŠãã(äžèšè¡š)ã
IMO: WMD-UNIGRAM ãæ£åœãªæ¯èŒã ãšèãããšã»ãšãã©å€ãããªãããã«èŠãã
5. è°è«ã¯ããïŒ
soft-alignment (MoverScore) ã hard-alignment (BERTScore) ãäžåã£ãŠãããšäž»åŒµããŠããã
IMO: bigram ãçšããå Žåã®è©±ãªã®ã§ãäžæŠã«æ¯èŒã§ããªãæ°ãããŠãã
future work ãšããŠã人éè©äŸ¡ã®ã³ã¹ãã®ããããªã evaluation system ãèšç»ããŠãããšã®ããšã
6. 次ã«èªãã¹ãè«æã¯ïŒ
- BERTScore: Evaluating Text Generation with BERT
ð[2019] XLNet: Generalized Autoregressive Pretraining for Language Understanding
0. è«æ
ã¿ã€ãã«
XLNet: Generalized Autoregressive Pretraining for Language Understanding
è«æãªã³ã¯
https://arxiv.org/abs/1906.08237
èè
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, Quoc V. Le
æçš¿æ¥ä»
2019/06/19
1. ã©ããªãã®ïŒ
BERT ãå§ããšãã AutoEncoder (AE) ç³» pre-train ã¢ãã«ãšãBERT 以åã® AutoRegressive (AR) 系㮠pre-train ã¢ãã«ã«ã€ããŠåæããããããã®é·æãåãå
¥ãã€ã€æ¬ ç¹ãè£ããã㪠AR åã® pre-train ã¢ãã«ã§ãã XLNet ãææ¡ã
BERT 㯠[MASK]
ããŒã¯ã³ã䜿ãããšã§ bidirectional ãªæ
å ±ãåãå
¥ããããšãåºæ¥ãããåæã«æšè«æã«ã¯ [MASK]
ããŒã¯ã³ã¯ååšããªãããã GAP ãããã
ãŸããå
¥åæãã©ã³ãã ã«ãã¹ã¯ãããŠããããšãããBERT ã¯å³å¯ã«åæååžãèšç®ã§ãããããããã®ãã¹ã¯ããŒã¯ã³ã¯ç¬ç«ãšããä»®å®ã§æšè«ãããŠããã
äžæ¹ã§ãAR åã® pre-train ã¢ãã«ã§ã¯ããã®å®çŸ©äž bidirectional ãªæ
å ±ãåãããšãåºæ¥ãªããããè¡šçŸèœåã§å£ãã
XLNet ã§ã¯ãAR åã®ã¢ãã«ã䜿ãã€ã€ãããŒã¯ã³ã®é çªãã©ã³ãã ã«ãã Permutation Language Modeling ã䜿ã£ãŠåŠç¿ãè¡ãããšã§ãAR åã¢ãã«ã®ã¡ãªãããæ®ãããŸãŸ ( = [MASK]
ã䜿ããªããŸãŸ) bidirectional ãªæ
å ±ãçšããããšãã§ããã
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
BERT ãªã©ã® AE å pre-train ã¢ãã«ãšãèšèªã¢ãã«ãªã©ã® AR å pre-train ã¢ãã«ã®ããããã®æ¬ ç¹ãè£ãåŠç¿æ¹æ³ã§ãã Permutation Language Modeling ãææ¡ããããã§åŠç¿ãã XLNet ã§çŸåšã® SOTA ã§ãã BERT ãæ§ã
ãªã¿ã¹ã¯ã§äžåã£ãã
èšèªã¢ãã«ã®æ§è³ªãæ®ãããŸãŸ bidirectional ã«æ
å ±ãåãå
¥ããããèšç·Žæ¹æ³ã¯åã
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
AE åãš AR åã®ããããã®æ¬ ç¹ãè£ãããã® Permutation Language Modeling, åã³ãããå®çŸããããã« Self-Attention ãæ¹è¯ãã Two-Stream Self-Attention ããã€ã³ããšãªãã
Permutation Language Modeling
Two-Stream Self-Attention
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
5. è°è«ã¯ããïŒ
6. 次ã«èªãã¹ãè«æã¯ïŒ
[2016] Large-Margin Softmax Loss for Convolutional Neural Networks
0. è«æ
ã¿ã€ãã«
Large-Margin Softmax Loss for Convolutional Neural Networks
ãªã³ã¯
https://arxiv.org/abs/1612.02295
èè
Weiyang Liu, Yandong Wen, Zhiding Yu, Meng Yang
æçš¿æ¥ä»
ICML 2016
1. ã©ããªãã®ïŒ
Softmax loss ãæ¹è¯ãã Large Marginx softmax (L-Softmax) Loss ãææ¡ãã¯ã©ã¹å
åæ£ãå°ããããã€ã¯ã©ã¹éã§ç¹åŸŽè¡šçŸãããåé¢ãããããã«åŠç¿ãè¡ãã
ãããã SGD ã§åŠç¿ã§ããéåŠç¿ãåé¿ããããšãåºæ¥ãã
4ã€ã®ãã³ãããŒã¯ã§å®éšãè¡ããããããæ§èœã§ããããšã瀺ããã
äžèšå³ã¯éåžžã® softmax loss (å·Šå³) ãšææ¡ææ³ (å³ã®3ã€ã®å³) ããåé¡çŽåã®ã¬ã€ã€ã®æ¬¡å
ã2次å
ã«ããŠå¯èŠåãããã®(äžã training data, äžã test data)ã
èŠãŠåããéããéåžžã® softmax ã«æ¯ã¹ãŠåã¯ã©ã¹å
ã§å°ããåãŸã£ãŠãããã¯ã©ã¹éã®åé¢ã倧ãããªã£ãŠããã
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
å
è¡ç 究ã§ã¯ contrastive loss ã triplet loss ãªã©ãè€æ°ã®ãµã³ãã«ã®ãã¢ãåæã«å
¥ããŠåŠç¿ãè¡ãããšã§ãµã³ãã«éã®è·é¢ã調æŽããŠããããææ¡ææ³ã¯éåžžã® softmax loss ã«ããŒãžã³ãå ããã ãã§åæ§ã®åŠç¿ãã§ããã
éåžžã® softmax loss ã§ãããããä»ã®ææ³ãšçµåãå¯èœã
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
- éåžžã® softmax loss ã¯ãæåŸã® dense layer (åé¡ã¬ã€ã€) ã®ãã©ã¡ãŒã¿ W ã«ãã£ãŠåŸãããç¹åŸŽ f ãšãããšã
- f ã¯æçµå±€ã®ãã©ã¡ãŒã¿ãšå ¥åã®å ç©ãªã®ã§ãcos ã§æžãæãããš
- ããã®æ£è§£ã©ãã«ã«ã®ã¿ããŒãžã³ m ãè§åºŠã«ããããã®ãææ¡ææ³
- ãã®ãŸãŸã ãš if æãå¿ èŠã§åŸ®åäžå¯èœã«ãªãã®ã§ããããé£ç¶ã«ãããã®ãΚãšããŠä»æ§
- ããŒãžã³ä»ãã§åŠç¿ããããšã§ãã¯ã©ã¹éè·é¢ã倧ããããã¯ã©ã¹å
åæ£ãå°ããããŠãã (äžèšå³åç
§)
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
visual classification ãš face verification ã®ã¿ã¹ã¯ã§å®éšãvisual classification 㯠MNIST, CIFAR10, CIFAR100 ã§ã face verification ã¯LFW dataset ã䜿çšã
visual classification ã§ã¯ã©ã®ã¿ã¹ã¯ã§ã SoTA ãèšé²ãããæ·±ãã¢ãã«ã䜿ã£ã CNN ã«ãåããšãå£ããªãã¹ã³ã¢ã«ãªã£ãŠããã
äžèšã®è¡šã®å³äžã face verification, ä»ã visual classification.
face verification ã§ãåæ§ã«ãå ¬éããŒã¿ã䜿ã£ãäžã§ã¯ SoTA ãèšé²ããprivate dataset ã䜿ã£ãã¹ã³ã¢ã«ãããªãè¿ã¥ããŠããã
5. è°è«ã¯ããïŒ
ç¹ã«ãªã
6. 次ã«èªãã¹ãè«æã¯ïŒ
- Dimensionality Reduction by Learning an Invariant Mapping
- http://yann.lecun.com/exdb/publis/pdf/hadsell-chopra-lecun-06.pdf
- Raia Hadsell, Sumit Chopra, Yann LeCun
- CVPR 2006
- Contrastive Loss ã®ææ¡è«æ
- FaceNet: A Unified Embedding for Face Recognition and Clustering
- https://arxiv.org/abs/1503.03832
- Florian Schroff, Dmitry Kalenichenko, James Philbin
- CVPR 2015
- Triplet Loss ã®ææ¡è«æ
ð[2019] Lifelong and Interactive Learning of Factual Knowledge in Dialogues
0. è«æ
ã¿ã€ãã«
Lifelong and Interactive Learning of Factual Knowledge in Dialogues
ãªã³ã¯
https://arxiv.org/abs/1907.13295
èè
Sahisnu Mazumder, Bing Liu, Shuai Wang, Nianzu Ma
æçš¿æ¥ä»
SIGDIAL 2019
1. ã©ããªãã®ïŒ
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
5. è°è«ã¯ããïŒ
6. 次ã«èªãã¹ãè«æã¯ïŒ
[2019] Dialogue Natural Language Inference
0. è«æ
ã¿ã€ãã«
Dialogue Natural Language Inference
ãªã³ã¯
https://arxiv.org/abs/1811.00671
ããŒã¿ã»ãã: https://wellecks.github.io/dialogue_nli/
èè
Sean Welleck, Jason Weston, Arthur Szlam, Kyunghyun Cho (FAIR)
æçš¿æ¥ä»
ACL 2019
arxiv: 2018/11/01
1. ã©ããªãã®ïŒ
察話ã¢ãã«ã«ãããäžè²«æ§ (consistency) ã®èª²é¡ã« Natural Language Inference ã®èŠ³ç¹ããåãçµãã ç 究ã
NLI ã¿ã¹ã¯ãšããŠäžè²«æ§ã®åºŠåããå®çŸ©ããæ°ããªããŒã¿ã»ãã Dialogue NLI ã䜿çšã㊠NLI ã¢ãã«ãåŠç¿ããããããçšãã察話㮠re-rank ææ³ãææ¡ãã察話ãšãŒãžã§ã³ãã®æã€ãã«ãœããšããã®çºè©±ã®éã®äžè²«æ§ãæ¹åããã
â» Natural Language Inference: 2ã€ã®ã»ã³ãã³ã¹ã®é¢ä¿ã®åé¡åé¡ã2ã€ã®ã»ã³ãã³ã¹ã«å¯ŸããŠãå«æ (entailment)ãäžç« (neutral)ãççŸ (contradiction) ããéžæããã
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
察話ã®äžè²«æ§ã NLI ã¿ã¹ã¯ãšããŠåå®çŸ©ãã察話ã¢ãã«ã®äžè²«æ§ã®æ¹åã«å¿çšããã
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
- Dialogue NLI Dataset
- æãšãã«ãœãæã®ã㢠(ãããã¯ãã«ãœãæã®ãã¢) ãšã©ãã« (E: entail, N: neutral, C: contradiction) ã®ããŒã¿ã»ãã
- å ããŒã¿ã¯ Persona-chat ã®ããŒã¿ã䜿ã£ãŠãã
- å šãŠã®çºè©±ããã«ãœãã«å¯ŸããŠããã®æã瀺ã triplet ãäœæ
- 2ã€ã®æã® triplet ãå ã«ãã©ãã«ãä»äž (e.g. åã triplet ãæã£ãŠããã° E: entail)
- Consistent Dialogue Agent via NLI
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
- ããŒã¹ã©ã€ã³ã¢ãã« (KV-Memory Network) ã«å¯ŸããŠå¯Ÿè©±çæãããã®åŸã«ææ¡ææ³ã«ãã re-rank ããããã®ãšæ¯èŒãã3çš®é¡ã® Hits@k ã«ãã㊠re-rank ããªããã®ã«å¯ŸããŠã©ãã倧ããæ¹åããã
- 人éã«ããè©äŸ¡ã§ãåæ§ã«ã consisitency, contradiction ã«ã€ããŠã®è©äŸ¡ã§æ¢åã¢ãã«ãæ¹åããã
5. è°è«ã¯ããïŒ
- æ°ããã察話ã«ãã NLI ã¢ãã«ãä»ã®ã¿ã¹ã¯ã«ã䜿ããã
- NLI ãä»ã®ã¿ã¹ã¯ã«ãŸã ãŸã åºããããã
- ããŒãŠãŸã ãŸã ã¹ã³ã¢äœã
6. 次ã«èªãã¹ãè«æã¯ïŒ
- PersonaChat
- nlp-survey-kb-response/nlp-survey-kb-response#7
- ããŒã¿ã»ããã®å ã«ãªã£ãŠãã
[2019] DyKgChat: Benchmarking Dialogue Generation Grounding on Dynamic Knowledge Graphs
0. è«æ
ã¿ã€ãã«
DyKgChat: Benchmarking Dialogue Generation Grounding on Dynamic Knowledge Graphs
ãªã³ã¯
https://arxiv.org/abs/1910.00610
èè
Yi-Lin Tuan, Yun-Nung Chen, Hung-yi Lee
æçš¿æ¥ä»
EMNLP 2019 (Submitted on 1 Oct 2019)
1. ã©ããªãã®ïŒ
åçã«å€ãããã㪠Knowledge Graph ãçšãã察話ã·ã¹ãã ã®å¿
èŠæ§ãããæ°ããªã¿ã¹ã¯ãšã㊠dynamic knowledge-grounded conversation generation ãææ¡, ãã³ãããŒã¯ããŒã¿ã»ãããšã㊠TV ãã©ãããäœæãã DyKgChat ãå
¬éããã
ããŒã¹ã©ã€ã³ã¢ãã«ãšã㊠Qadpt ãšãã GRU ãšã³ããŒã¡ã«ããºã ãçµã¿èŸŒãã ã¢ãã«ãäœæãæ¢åã¢ãã«ãšã®æ¯èŒå®éšãè¡ã£ãã
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
Dynamic ã«ããããã Knowledge Graph ã« zero-shot ã§ã¢ãã«ãé©çšããããããªã¿ã¹ã¯ã»ãã³ãããŒã¯ããŒã¿ã»ãããææ¡ã
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
Dynamic 㪠knowledge graph ãçšãã察話çæãæ°ããã¿ã¹ã¯ãšããŠææ¡ããŠããã
æ¢åææ³ã§ã KG ãçšãã察話çæã¯ååšãããããããã¯ã°ã©ãããåŸãããç¹åŸŽãªã©ãå
ã«æšè«ãããŠãããKG èªäœãæŽæ°ããããªã©ã§å€åããå Žåã«ã©ã®ãããªåãããããåãããªã課é¡ã®ããã®ã¿ã¹ã¯ã
Knowledge Entity ãããŸãã¢ããªã³ã°ã§ããŠãããã Knowledge Graph ã®å€åã«å¯Ÿå¿ã§ããŠããããã§ããããè©äŸ¡ææšãææ¡ããŠããã
- Knowledge Entity Modeling (Entity ã掻çšã§ããŠããã)
- Knowledge word accuracy: decoder ã®åºåãã KG ã® entity ãæ£ããäºæž¬ãããŠããã
- Knowledge and generic word classification: knowledge graph ã® entity ãåŠãã®åé¡åé¡ã® recall/precision
- Generated knowledge words: çæããã knowledge entity ã®recall/precision
- Adaptation of Changed Knowledge Graph (ã°ã©ãã®å€åã«é©å¿ã§ããŠããã)
- Change rate: åã® KG ããè¿çãã©ã®ãããå€åããã
- Accurate Change rate: KG ãå ¥ãæ¿ããå Žåã® accuracy ã®å€å
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
ããŒã¹ã©ã€ã³ã¢ãã«ãšæ¢åã¢ãã«ã§ DyKgChat ã§æ¯èŒå®éšã
Table 4 ã adaptation ã®è©äŸ¡ãtable 3 ã knowledge entity ã®è©äŸ¡ã
All/Last1/Last2 ã¯ããããã KG ãã©ãã ãã©ã³ãã ã«å
¥ãæ¿ããããAll ã ãšæ¢åã¢ãã«ã§ãããããã ããLast1/Last2 ãªã©å°ãã ãå
ã® KG ãæŽæ°ãããã®ã«å¯ŸããŠå€§ããæ§èœå·®ãããã
ããææšã®æå³ãåããã¥ãã(accuracy changed rate ã¯å€ååãªãå€åããªããŠãè¯ãã®ã§ã¯)ãã©ã®ãããè¯ããªã£ãã®ããããã¥ããã
人éã«ããè©äŸ¡ãè¡ã£ãŠãããFluency ãš Information ã®èŠ³ç¹ã§äžéšãé€ããŠæ¢åã¢ãã«ãäžåã£ãŠããã
5. è°è«ã¯ããïŒ
- MemNet, TAware 㯠KG ã倧ããå€åãããšãã® zero-shot é©çšãããçšåºŠã§ãããã§ã¯ãã£ã
- ææ¡ã¢ãã«ã§ãã Qadpt 㯠KG ã埮åŠã«å€åãããšã察å¿èœåããããã
6. 次ã«èªãã¹ãè«æã¯ïŒ
[2019] Natural Questions: a Benchmark for Question Answering Research
0. è«æ
ã¿ã€ãã«
Natural Questions: a Benchmark for Question Answering Research
ãªã³ã¯
https://ai.google/research/pubs/pub47761
èè
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, Slav Petrov
(Google Research)
æçš¿æ¥ä»
2019
1. ã©ããªãã®ïŒ
QA ã®ããã®ããŒã¿ã»ãã Natural Questions (NQ) ãå
¬éãGoogle ã®æ€çŽ¢ãã°ããåŸããã質åãšã1ã€ã® wikipedia èšäºã§1ããŒã¿ãšãªã£ãŠãããèšäºå
ãã Long Answer (ãã©ã°ã©ããè¡šãªã©)ãš Short Answer (æ°åèª) ãåœãŠãã¿ã¹ã¯ãšãªã£ãŠããã
è«æå
ã§ã¯ãå®éšã«ããããŒã¿ã®è³ªãæ€èšŒã25 way annotation ã§äººéã® annotation ã®å€åã調æ»ãQA ã®ã¿ã¹ã¯ã«é©å㪠robust ãªææšãå°å
¥ãææ°ææ³ã§ã®ããŒã¹ã©ã€ã³äœæãªã©ãè¡ã£ãŠããã
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
5. è°è«ã¯ããïŒ
6. 次ã«èªãã¹ãè«æã¯ïŒ
[2019] Read + Verify: Machine Reading Comprehension with Unanswerable Questions
0. è«æ
ã¿ã€ãã«
Read + Verify: Machine Reading Comprehension with Unanswerable Questions
ãªã³ã¯
https://arxiv.org/abs/1808.05759
èè
Minghao Hu, Furu Wei, Yuxing Peng, Zhen Huang, Nan Yang, Dongsheng Li
æçš¿æ¥ä»
AAAI-19
1. ã©ããªãã®ïŒ
MRC ã¿ã¹ã¯ã«ãããŠãããŒã¿äžã«å«ãŸããçããããªã質å (Unanswerable Questions) ã«ã€ããŠãæ¹åã«åãçµãã ç 究ãæ¢åç 究ã§ã¯è¶³ããŠããªãåçã®åŠ¥åœæ§ãèæ ®ããããã«ã read-and-verify ã·ã¹ãã ãææ¡ãreader ã«ãã£ãŠåŸãããåçãå ¥åã«å¯ŸããŠåŠ¥åœã調ã¹ã verifier ãšã2ã€ã®è£å©æ倱ãå ããåŠç¿ã§2019/08æç¹ã® SQuAD 2.0 ã® SoTA ãæŽæ°ããã
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
æ瀺çã«åçã®åŠ¥åœæ§ãæ€èšŒãã answer verifier ãã¢ãã«ã«çµ±åããã
åçã®æœåºãš no-answer ãªè³ªåã®æ€åºãåããŠããŸãæ±ãããã«2ã€ã®è£å©æ倱ãææ¡ããã
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
- 2ã€ã® auxiliary loss
- Independent Span Loss
- Independent No-Answer Loss
- Answer verifier
- åçãæœåºããŠãããæœåºããåçãšå ¥åãšãªã質åæãã³ã³ããã¹ããå床ã¢ãã«ã«å ¥å
- 3çš®é¡ã®ã¢ãã«ã§åçã劥åœãã©ããæ€èšŒãã
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
No-answer ãªããŒã¿ãå«ãŸãã SQuAD 2.0 㧠2018/08æç¹ã§ SoTA ãæŽæ°ã
Ablation Study ãå§ããšããŠæ§ã
ãªåæãè¡ã£ãŠããã
5. è°è«ã¯ããïŒ
6. 次ã«èªãã¹ãè«æã¯ïŒ
[2018] ArcFace: Additive Angular Margin Loss for Deep Face Recognition
0. è«æ
ã¿ã€ãã«
ArcFace: Additive Angular Margin Loss for Deep Face Recognition
ãªã³ã¯
https://arxiv.org/abs/1801.07698
èè
Jiankang Deng, Jia Guo, Niannan Xue, Stefanos Zafeiriou
æçš¿æ¥ä»
CVPR 2019
arxiv: 20180123
1. ã©ããªãã®ïŒ
è¡šçŸåŠç¿(è·é¢åŠç¿)ã®åéã«ãããŠããèå¥æ§èœãäžããæ倱é¢æ°ãšã㊠ArcFace ãææ¡ã
æ¢åç 究ã§ãã SphereFace ã CosFace ãšæ¯ã¹ãŠãangular ã®ç©ºéã§ããŒãžã³ãå ããããšã§ãããå®è£
ãç°¡åã§åŠç¿ãããããªã£ã(åŸè¿°)ã
2. å è¡ç 究ãšæ¯ã¹ãŠã©ãããããïŒ
æ¢åç 究ã«æ¯ã¹ãŠå®è£
ãç°¡åã§åŠç¿ãè¡ãããããæŽã«å¹æã®é«ãæ倱é¢æ°ãææ¡ããã
æ¢åç 究ãšã®å·®ã¯ã次ã®ç»åã§æ±ºå®å¢çã®å·®ãšããŠæãããŠããã
3. æè¡ãææ³ã®ãã¢ã¯ã©ãïŒ
æ¢åç 究ã§ãã SphereFace ã CosFace ã§ããããã softmax loss ã«ããŒãžã³ãå ¥ããã¢ãããŒããææ¡ãããŠããããArcFace ã§ã¯ cos ã® angular (è§åºŠ) ã®ç©ºéã§ããŒãžã³ã足ãããšã§ããèšç®/å®è£ ãç°¡åã«ããã€å¹æçãªããŒãžã³ãåãå ¥ããŠããã
ãã®3ã€ã®ææ³ã¯äžã€ã®åŒã§ãŸãšããŠè¡šãããšãã§ãã次ã®ããã«ãªãã
ã³ãµã€ã³éšåã« ã®ãããªåœ¢ã§ããŒãžã³ãå
¥ããŠãããm_1, m_2, m_3 ããããã SphereFace, ArcFace, CosFace ã§ã®ææ¡ãšãªã£ãŠããã
æ¢åã® SphereFace, CosFace ãããæå¹ã§ããäžã«å®è£ ãç°¡åã§åŠç¿ãããããããšãã¡ã€ã³ã®è²¢ç®ããšæããããããªãæ¢åææ³ããæå¹ãªã®ãã¯ãããã¯åãããªãã®ã§æéãå²ããããã
4. ã©ããã£ãŠæå¹ã ãšæ€èšŒããïŒ
ããªãå€ãã®é¡èªèããŒã¿ã»ãããçšããŠå®éšãããã®ãã¹ãŠã§æ¢åææ³ã«æ¯ã¹ãŠæãè¯ãçµæãšãªã£ãã
ArcFace + SphereFace + CosFace ãªã©ã®è€å loss ã§ããArcFace åäœã®æ¹ãããçµæã«ã
ãŸããæ瀺çã« Intra Loss, Inter Loss ãå®çŸ©ãããã®ãTriplet Loss ã«ãããã®ãšãæ¯èŒããArcFace ãæããã©ã³ã¹ã®åããçµæã«ãªãããšãå®éšã«ãã瀺ããŠããããããã® loss ãš ArcFace ãçµåããŠããæ§èœã¯äžããããArcFace ã ã㧠Intra loss ã Inter loss ã«ã€ããŠæé©åã§ããŠããããšãåããã
5. è°è«ã¯ããïŒ
6. 次ã«èªãã¹ãè«æã¯ïŒ
- SphereFace: Deep Hypersphere Embedding for Face Recognition
- https://arxiv.org/abs/1704.08063
- Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, Le Song
- CVPR 2017
- SphereFace ææ¡è«æ
- CosFace: Large Margin Cosine Loss for Deep Face Recognition
- https://arxiv.org/abs/1801.09414
- Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, Wei Liu
- CVPR 2018
- CosFace ææ¡è«æãã®1
- Additive Margin Softmax for Face Verification
- https://arxiv.org/abs/1801.05599
- Feng Wang, Weiyang Liu, Haijun Liu, Jian Cheng
- CosFace ææ¡è«æãã®2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
ð Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ððð
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google â€ïž Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.