Comments (2)
Hi @lovodkin93,
if I understand you correctly, you're asking why self-attention for padded tokens (those beyond end-of-sentence) is not fully disabled, and whether this will affect the model parameters.
The short answer is that the padded states are masked further down in the computation path, and thus have no effect on the loss (there is masking in both the self-attention and the cross-attention that prevents attention to them, and on the decoder side, the cross-entropy loss itself is also masked beyond the end-of-sentence). Consequently, the error signal that is being backpropagated through these connections is 0, and they will not affect the model parameters.
from nematus.
great, I figured as much, but didn't find the relevant further masking, and wanted to make sure and not just assume.
Thank you!
from nematus.
Related Issues (20)
- Docker: No module named 'rescore' HOT 1
- Throws error when running with rnn_dropout_embedding: true parameter HOT 5
- Load model from Opus-MT HOT 6
- how to map --external_validation_script option to the current nematus with tensorflow? HOT 4
- reimplement rnn of nematus
- How to continue training from the last checkpoint HOT 5
- Why use tie_encoder_decoder_embeddings instead of tie_decoder_embeddings in line 131 of transformer.py? HOT 3
- decoding in the Transformer-based model HOT 12
- Training stops after few epochs HOT 1
- beam search HOT 1
- translation_maxlen HOT 2
- Performance issues in the difinition of generate_initial_memories, nematus/transformer_inference.py HOT 1
- OP_REQUIRES failed at strided_slice_op.cc:108 : Invalid argument: slice index -1 of dimension 1 out of bounds. HOT 2
- Config mrt_ml_mix seems to be unused HOT 1
- TensorArray Not Used on line 240 of mrt_utils.py
- multiple models (with same vocabulary) for ensemble decoding HOT 2
- None exception HOT 4
- Question about running on GPU HOT 3
- A request for system version updates HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nematus.