Implement RotatE

[accenture]

Implement RotatE

[sumitpai]

Committed under feature/8 branch db83441. The model must be benchmarked before merging with develop.

[Chen-Cai-OSU]

Hi,

Thanks for implementing RotatE. I ran the doctest but get a different result:

model = RotatE(batches_count=1, seed=555, epochs=20, k=10,
loss = 'pairwise', loss_params = {'margin': 1},
regularizer = 'LP', regularizer_params = {'lambda': 0.1, 'p': 2})
X = np.array([['a', 'y', 'b'],
['b', 'y', 'a'],
['a', 'y', 'c'],
['c', 'y', 'a'],
['a', 'y', 'd'],
['c', 'y', 'd'],
['b', 'y', 'c'],
['f', 'y', 'e']])
model.fit(X)

print(model.predict(np.array([['f', 'y', 'e'], ['b', 'y', 'd']])))
# ([-0.5440256, -2.1364462], [3, 8]) (I got [-1.3982366, -1.6705607])

print(model.get_embeddings(['f','e'], embedding_type='entity'))
# array([[0.21769601, 0.50337195, 0.15759109, 0.47421402, 0.1621065,
#         0.3356035, -0.03751088, 0.04361391, -0.27752247, 0.62098897,
#         0.07596578, 0.0731838, 0.02322292, 0.5670342, 0.550395,
#         0.4143563, -0.5241787, 0.44299904, 0.53574693, -0.50007296],
#        [-0.22395124, 0.09082595, 0.06242271, 0.64942193, 0.35046905,
#         0.6611834, 0.4461781, 0.33218127, -0.3407303, 0.39955628,
#         -0.07239022, 0.67604166, -0.18704332, -0.03574913, 0.20055567,
#         -0.2691682, 0.06541196, 0.46618107, 0.35908118, -0.7174517]],
#       dtype=float32)

I got
[[ 1.7983948e-01 1.7090929e-01 2.5513259e-01 2.2759666e-01
1.7504254e-04 7.4905530e-02 -2.5646752e-01 1.8808967e-02
-1.1450458e-01 4.1813409e-01 -1.1329085e-01 -1.1636985e-01
2.7548817e-01 3.5413542e-01 5.9430939e-01 4.4631422e-02
-3.8878655e-01 1.4761178e-01 5.1016110e-01 -2.7574781e-01]
[-1.9979909e-01 2.0788297e-01 5.1174801e-02 4.4964004e-01
6.8522096e-02 5.8385164e-01 2.9336452e-01 3.2832843e-01
-5.9851337e-02 2.8525338e-01 -2.0095851e-01 5.8154076e-01
-6.5600999e-02 4.1697629e-02 2.7135696e-02 -1.2308941e-01
2.9924816e-01 3.6170337e-01 2.4286729e-01 -5.6315279e-01]]

[Chen-Cai-OSU]

I compare your implementation with the original implementation. It seems that in the original implementation, there is mode called head-batch, and your implementation correspond to the original implementation when the head-batch condition is on. I am not exactly sure why do we need both versions. Aren't they exactly the same based on the equation (7) of the RotatE paper?

    pi = 3.14159265358979323846
    re_head, im_head = torch.chunk(head, 2, dim=2)
    re_tail, im_tail = torch.chunk(tail, 2, dim=2)

    #Make phases of relations uniformly distributed in [-pi, pi]

    phase_relation = relation/(self.embedding_range.item()/pi)

    re_relation = torch.cos(phase_relation)
    im_relation = torch.sin(phase_relation)

    if mode == 'head-batch':
        re_score = re_relation * re_tail + im_relation * im_tail
        im_score = re_relation * im_tail - im_relation * re_tail
        re_score = re_score - re_head
        im_score = im_score - im_head
    else:
        re_score = re_head * re_relation - im_head * im_relation
        im_score = re_head * im_relation + im_head * re_relation
        re_score = re_score - re_tail
        im_score = im_score - im_tail

    score = torch.stack([re_score, im_score], dim = 0)
    score = score.norm(dim = 0)

    score = self.gamma.item() - score.sum(dim = 2)
    return score

[sumitpai]

Hi. The doc string is not updated. Sorry for that. It will be updated before merging with master.

The implementation of RotatE is not fully complete as we couldn't reproduce the results mentioned in the paper with our current code. Hence it is not merged with the master branch. However, we could get better results with adversarial loss on complEx and other models.

As for the second query, even I am not sure why that has been done in their implementation. If you simplify the two, it boils down to the same equation.

[Chen-Cai-OSU]

Thank you very much for the clarification. I will be waiting for the RotatE to be fully implemented.

[tabacof]

I have updated the feature branch https://github.com/Accenture/AmpliGraph/tree/feature/8 with the following changes:

1. Merging code from current develop
2. Fixed the norm calculation in the RotatE metric
3. Added the embedding initialisation and phase normalisation based on their code -- (note that we raised two issues there as well: 9 and 10)

Still, the algorithm fails to work in practice. I suspect that is due to the head- and tail-batch modes, which are intercalated during training on their codebase with different corruption types for each mode.

[Chen-Cai-OSU]

Thanks for the update! May I know how large is the gap between the current performance and the reported performance in the paper?

[tabacof]

@Chen-Cai-OSU I will try a parameter search to be sure it is not related to parameter choices and then I will report back here. It will take a few days at least. Thank you for the patience.

[Chen-Cai-OSU]

Hello,

Is the implementation of RotatE still ongoing?

[ChrisDelClea]

+1 , is it implemented yet?