Clarifications on Combined Representation

[tech-srl]

Clarifications on Combined Representation

[urialon]

Hi @hsellik ,
No problem, feel free to ask questions.

Yes! this is similar to the dense layer of code2vec.

1. The reason for 2d_path is that the path is encoded using a bidirectional LSTM, so we get two output vectors for the path, "forward" and "backwards".

Now, for the question of whether the matrix is ((2d_path+2d_token) × d_hidden) or (d_hidden x (2d_path+2d_token)) - you are right. I think it should be as you say, (d_hidden x (2d_path+2d_token)) (i.e., d_hidden is first).

Thank you for noticing.

2. Yeah, I agree, it is a bit confusing. v_1 and v_l are indeed the terminals, which come from the limited set of AST nodes. For example, such terminals are often: "NameExpr", "StringExpr", "IntegerExpr". When we write that we "encode" the path, we mean that we use these symbols and the nonterminals between them. I.e., the path does contain the terminal. When we call encode_token, we take the "value" of these terminals, which is a user-defined value. For example, by a "value" I refer to the number 3, which can be the value of a terminal of type IntegerExpr. A possible value of a terminal of type NameExpr can be the value mySqlConnection.

So E^{nodes} contain terminal types like NameExpr and IntExpr, along with nonterminal types like BlockStatement and ForeachStatement.
The other embedding matrix, E^{subtokens}, contains the subtokens "3", "my", "sql" and "connection".

Does that make sense?

3. Regarding "encoding" and "embedding":

I usually refer to "embedding" as the vectors that are learned for each symbol, the learned vectors themselves; and I usually refer to "encoding" as something that requires some computation to represent. So, encoding a token can be performed by summing its subtoken embeddings.
encoding a path can be performed by LSTM-ing its node type embeddings.

Does that make sense?

Regarding your chart - it looks accurate! But the best is to look at the code. If there is something you can't find in the code feel free to ask and I'll guide you.

[hsellik]

Thank you for the answers! They all make sense. 😄 💥

I do agree, looking at the code is the best source.

Got to finish off those Andrew Ng courses before tackling the full logic behind Code2Seq attention properly. 😅 On the bright side, Code2Vec model and the attention made a lot of sense!