Question about the paper

[tebesu]

Question about the paper

[tebesu]

1. The matrix C is the external memory matrix as you mentioned. It is a separate learnable parameter however, it could be tied to be equal to M (I have not tried this). If we follow the traditional memory network view: M are the keys and C are the values. We query the keys with the embedding and extract the corresponding memory values in C.

2. Its possible we could use M to aggregate the attention scores that is setting C=M and tying the weights as I mentioned earlier. If I remember correctly an issue is the key values get very large since they compete to be selected in the attention mechanism. So having separate parameters may alleviate this problem. I suspect that the softmax function is not the correct function to represent attention over a large neighborhood.

Hope this helps and good luck in your research.