Question about positional embedding?

[facebookresearch]

Question about positional embedding?

[KaimingHe]

Good catch. It looks like the namings of emb_h and emb_w in get_2d_sincos_pos_embed_from_grid are not ideal (need to verify):

mae/util/pos_embed.py

Lines 38 to 46 in 6a2ba40

	def get_2d_sincos_pos_embed_from_grid(embed_dim, grid):
	assert embed_dim % 2 == 0
	# use half of dimensions to encode grid_h
	emb_h = get_1d_sincos_pos_embed_from_grid(embed_dim // 2, grid[0]) # (H*W, D/2)
	emb_w = get_1d_sincos_pos_embed_from_grid(embed_dim // 2, grid[1]) # (H*W, D/2)
	emb = np.concatenate([emb_h, emb_w], axis=1) # (H*W, D)
	return emb

It should be named as:

    emb_w = get_1d_sincos_pos_embed_from_grid(embed_dim // 2, grid[0])  # (H*W, D/2)
    emb_h = get_1d_sincos_pos_embed_from_grid(embed_dim // 2, grid[1])  # (H*W, D/2)

    emb = np.concatenate([emb_w, emb_h], axis=1) # (H*W, D)

This produces exactly the same emb as the original code.

Note that if you only swap grid[0] and grid[1], it does not produce the same emb. If so, it should still work if you do pre-training using the modified code, but it would break our pre-trained checkpoints as emb was changed.