Inconsistencies in unsqueeze operation description in the book and in notebook and its necessity (3.6.2 Implementing multi-head attention with weight splits)

[rasbt]

Inconsistencies in unsqueeze operation description in the book and in notebook and its necessity (3.6.2 Implementing multi-head attention with weight splits)

[rasbt]

Nice, it turns out you were right, the .unsqueeze(0) was indeed redundant. Love it, it makes the code even simpler and more readable!

[labdmitriy]

Also I have a question - could you please explain why do we need to call contiguous() in the following line in MultiHeadAttention class:

context_vec = context_vec.contiguous().view(b, num_tokens, self.d_out)

[rasbt]

mask_unsqueezed = mask_bool.unsqueeze(0).unsqueeze(0)

Ah yes, this was unnecessary so I updated it to just mask_bool.unsqueeze(0) a while back. I will look into whether I can remove it altogether like you suggest. Thanks!

Also I have a question - could you please explain why do we need to call contiguous() in the following line in MultiHeadAttention class:

context_vec = context_vec.contiguous().view(b, num_tokens, self.d_out)

Good question. This is because the way the memory is organized in this tensor; the .view() would raise an error. What you could do is

context_vec = context_vec.reshape(b, num_tokens, self.d_out)

This this is because (quoting from the documentation):

When possible, the returned tensor will be a view of input. Otherwise, it will be a copy. Contiguous inputs and inputs with compatible strides can be reshaped without copying, but you should not depend on the copying vs. viewing behavior.

However, I haven't used .reshape elsewhere in this book so I wanted to stick with .view for consistency.

[labdmitriy]

Sebastian, thanks a lot for your response,

Good question. This is because the way the memory is organized in this tensor; the .view() would raise an error

Yes, this question was asked because when I deleted .contiguous():

context_vec = context_vec.view(b, num_tokens, self.d_out)

I didn't have any errors and get the same results.

Only one another reason to convert to contiguous tensor that I found here was the following:

This create issues with parallel computations.

But I didn't find more detailed explanation.
Could you please share your thoughts about it?

Thank you.