Comments (6)
This is my result (score is BLEU score on dev set):
experiment 1 (VNMT):
INFO:tensorflow:Best score at step 5000: 0.041646
INFO:tensorflow:Best score at step 10000: 0.089593
INFO:tensorflow:Best score at step 15000: 0.120238
INFO:tensorflow:Best score at step 20000: 0.157707
INFO:tensorflow:Best score at step 25000: 0.162056
INFO:tensorflow:Best score at step 30000: 0.204992
INFO:tensorflow:Best score at step 35000: 0.206457
INFO:tensorflow:Best score at step 40000: 0.219764
INFO:tensorflow:Best score at step 45000: 0.219764
INFO:tensorflow:Best score at step 50000: 0.219764
INFO:tensorflow:Best score at step 55000: 0.239175
INFO:tensorflow:Best score at step 60000: 0.241595
INFO:tensorflow:Best score at step 65000: 0.241595
INFO:tensorflow:Best score at step 70000: 0.241641
INFO:tensorflow:Best score at step 75000: 0.250475
experiment 2 (VNMT w/o KL):
INFO:tensorflow:Best score at step 5000: 0.064169
INFO:tensorflow:Best score at step 10000: 0.083137
INFO:tensorflow:Best score at step 15000: 0.126807
INFO:tensorflow:Best score at step 20000: 0.145696
INFO:tensorflow:Best score at step 25000: 0.169591
INFO:tensorflow:Best score at step 30000: 0.201930
INFO:tensorflow:Best score at step 35000: 0.201930
INFO:tensorflow:Best score at step 40000: 0.201930
INFO:tensorflow:Best score at step 45000: 0.220347
INFO:tensorflow:Best score at step 50000: 0.220347
INFO:tensorflow:Best score at step 55000: 0.234179
INFO:tensorflow:Best score at step 60000: 0.234179
INFO:tensorflow:Best score at step 65000: 0.235006
INFO:tensorflow:Best score at step 70000: 0.235006
INFO:tensorflow:Best score at step 75000: 0.242985
experiment 3 (VNMT w/o KL && set h'_e to zero vector, thus equivalent to RNNSearch):
INFO:tensorflow:Best score at step 5000: 0.195378
INFO:tensorflow:Best score at step 10000: 0.229090
INFO:tensorflow:Best score at step 15000: 0.245143
INFO:tensorflow:Best score at step 20000: 0.249300
INFO:tensorflow:Best score at step 25000: 0.263898
INFO:tensorflow:Best score at step 30000: 0.282799
INFO:tensorflow:Best score at step 35000: 0.285575
INFO:tensorflow:Best score at step 40000: 0.288144
INFO:tensorflow:Best score at step 45000: 0.288144
INFO:tensorflow:Best score at step 50000: 0.295313
INFO:tensorflow:Best score at step 55000: 0.302703
INFO:tensorflow:Best score at step 60000: 0.306085
INFO:tensorflow:Best score at step 65000: 0.306085
INFO:tensorflow:Best score at step 70000: 0.306247
INFO:tensorflow:Best score at step 75000: 0.308485
It's easy to see VNMT is better than VNMT w/o KL, but none of them can match performance of RNNSearch. It seems to be VNMT hasn't converged, and I keep training VNMT to 230k steps and it now achieves a BLEU score of 27 on dev set (still not good enough but better than that in 75k steps).
Has this slow convergence observed in your experiment? Thanks.
from vnmt.
@soloice Yes, training VNMT from scratch without any other optimzation tricks usually results in undesirable convergence and relatively worse performance. We also observed this phenomena in our empirical experiments, and finally we chose to train VNMT based on a well-pretrained NMT model.
I'm not sure whether your experimental design is correct, but I believe there are something wrong with your VNMT w/o KL. This is because VNMT w/o KL just extends NMT with a global source representation. Generally, you can treat it as a combination of the vanilla seq2seq model and the attentional seq2seq model. So, it's performance should be near that of NMT. Based on your results, I guess that after removing the KL term, maybe you forgot to change the reparameterization part where the noise should be dropped.
If you prefer training VNMT from the very begining, I suggest you borrow some idea from the weight annealing on the KL term. However, in my experience, selecting the optimal strategy for the annealing is , to some extent, also task- and data- dependent. You can find more details in this paper.
from vnmt.
Your suggestion is very informative!
I'll check my implementation to see if one of the above issues exists, especially my VNMT w/o KL implementation.
from vnmt.
Hi, I checked my implementation and I don't think my VNMT w/o KL
model is wrong.
My code can be found here, which is based on XMUNMT. The relevant part is in file vnmt.py in L94-95, L109-121, L167 and L277-311.
I use params.enable_KL to switch between VNMT
and VNMT w/o KL
, and disabling params.enable_KL && setting h_e_prime
to zero vector gives RNNSearch.
(Btw, XMUNMT is also from XMU, and I don't know if you know that work, lol~ It has several differences from the original RNNSearch, e.g.: initialization, optimizer, etc.)
I tuned my hyperparameters (mainly learning rate and initializer) and made VNMT w/o KL
converges better (BLEU achieves 0.265 on dev set at step 75k), but still not good enough:
INFO:tensorflow:Best score at step 2500: 0.007113
INFO:tensorflow:Best score at step 5000: 0.028652
INFO:tensorflow:Best score at step 7500: 0.062944
INFO:tensorflow:Best score at step 10000: 0.104961
INFO:tensorflow:Best score at step 12500: 0.122892
INFO:tensorflow:Best score at step 15000: 0.152752
INFO:tensorflow:Best score at step 17500: 0.167090
INFO:tensorflow:Best score at step 20000: 0.183927
INFO:tensorflow:Best score at step 22500: 0.195720
INFO:tensorflow:Best score at step 25000: 0.200146
INFO:tensorflow:Best score at step 27500: 0.209534
INFO:tensorflow:Best score at step 30000: 0.217454
INFO:tensorflow:Best score at step 32500: 0.220276
INFO:tensorflow:Best score at step 35000: 0.227976
INFO:tensorflow:Best score at step 37500: 0.230734
INFO:tensorflow:Best score at step 40000: 0.230734
INFO:tensorflow:Best score at step 42500: 0.238548
INFO:tensorflow:Best score at step 45000: 0.239537
INFO:tensorflow:Best score at step 47500: 0.241912
INFO:tensorflow:Best score at step 50000: 0.247325
INFO:tensorflow:Best score at step 52500: 0.253986
INFO:tensorflow:Best score at step 55000: 0.258078
INFO:tensorflow:Best score at step 57500: 0.258078
INFO:tensorflow:Best score at step 60000: 0.258078
INFO:tensorflow:Best score at step 62500: 0.258346
INFO:tensorflow:Best score at step 65000: 0.259014
INFO:tensorflow:Best score at step 67500: 0.264653
INFO:tensorflow:Best score at step 70000: 0.265454
INFO:tensorflow:Best score at step 72500: 0.265454
INFO:tensorflow:Best score at step 75000: 0.265454
I guess I should try fine-tuning instead of insisting training VNMT
or VNMT w/o KL
from scratch= =
But I still don't understand the hyper-parameters which work well for RNNSearch
failed to do so for VNMT w/o KL
, sad~
from vnmt.
Hi @soloice , XMUNMT is an excellent open-source toolkit. It's glad to see more users.
I glanced over your code for VNMT w/o KL, and I think your implementation is ok. But your training result is different from ours. For this model, we just observe very similar performance as the NMT. And I also do not know where is the problem.
from vnmt.
It's very kind of you to review my code.
Since the reproduction in XMUNMT is slightly different from that of RNNSearch, a possible reason is that AdaDelta optimizer or different initialization causes this phenomenon.
Now I'm working on finetune VNMT given a well trained RNNSearch model, and or course, I'll release my TF code if I can make it work.
from vnmt.
Related Issues (5)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from vnmt.