Out of vocabulary word vectors

[Henry-E]

Out of vocabulary word vectors

[abisee]

@Henry-E re: distinguishing between unique OOV words, also check out the discussion in the comments started by RobinChen here.

[abisee]

Hi @Henry-E,

Have you read the paper? This is a question that would be easier to answer by looking at the paper than the code.

For the pointer-generator model, OOV words in the source text are represented by the word vector for the UNK token (because as they're OOV, there is no word vector to use). In the paper and the code, we refer to e.g. "50k + number of unknown words in the article" as the extended vocabulary because that's the set of words that can be produced by the decoder. This doesn't mean that all the words in the extended vocabulary have word vectors, though. Only the words in the original vocabulary have word vectors.

Hope this answers your question.

[Henry-E]

Yep that sounds good. I was just trying to figure out how the probability distribution for the extended vocabulary was obtained in practice. For some reason I thought in order to attend properly the attention mechanism needed to be able to distinguish between unique OOV words.

[Henry-E]

Ah ok, and the final bit that I also was confused about, for anyone who comes across this later, was how the loss is calculated. I subsequently thought that maybe the copy mechanism was only applied at test time but it appears that the loss is calculated based on the extended vocabulary.

The loss function is as described in equations (6) and (7), but with respect to our modified probability distribution P(w) given in equation (9).