shehzaadzd/minerva

Meandering In Networks of Entities to Reach Verisimilar Answers

minerva's Issues

On comparison of MINERVA with other KBE baselines

Hi, thanks for releasing the code! I have some questions on comparison of MINERVA with other baselines such as ComplEx and DistMult. In Section 4.1, it is mentioned that for FB15K-237, "The performance of ConvE and ComplEx are taken from Dettmers et al. (2017)". In the codebase, you have considered link prediction queries (h, r, ?) i.e. to predict the tail entity given head and the relation. Whereas, in most other KBE works, they average the results for (h, r, ?) and (?, r, t) while reporting the results.

Ques:

1. In Table 6 of your paper, does your result correspond to just (h, r, ?) or the average?
2. For other baselines, do you report the metrics for (h, r, ?) or the average? In Dettmers et al. (2017), the Hits@10 numbers reported for ConvE and ComplEx are 0.501 and 0.428 whereas in your paper, it is 0.458 and 0.419 respectively.

Thanks for your time!

Reproducing NELL-995 MAP Results

Thanks very much for releasing the code in accompany with the paper. It definitely makes reproducing the experiments a lot easier. I've been playing with the code base and have some questions on reproducing the NELL-995 experiments.

The codebase does not contain the configuration file for NELL-995 experiments, nor does it contains the evaluation scripts for computing MAP. (Maybe you've missed them from the release?)
I used the hyperparameters reported in "Experimental Details, section 2.3" and the appendix section 8.1 of the paper, which results in the following configuration file:

data_input_dir="datasets/data_preprocessed/nell-995/"
vocab_dir="datasets/data_preprocessed/nell-995/vocab"
total_iterations=1000
path_length=3
hidden_size=400
embedding_size=200
batch_size=64
beta=0.05
Lambda=0.02
use_entity_embeddings=1
train_entity_embeddings=1
train_relation_embeddings=1
base_output_dir="output/nell-995/"
load_model=1
model_load_dir="saved_models/nell-995/model.ckpt"

I run train & test as specified in the README, and evaluate the decoding results using the MAP computation script produced by the DeepPath paper. (I assumed that the experiment setup is exactly the same as the DeepPath paper since you compared head-to-head with them.)

However, the MAP results I obtained this way is significantly lower compared to the reported results.

MINERVA concept_athleteplaysinleague MAP: 0.810746658312 (380 queries evaluated)
MINERVA concept_athleteplaysforteam MAP: 0.649309434089 (386 queries evaluated)
MINERVA concept_organizationheadquarteredincity MAP: 0.944878371403 (246 queries evaluated)
MINERVA concept_athleteplayssport MAP: 0.919186046512 (602 queries evaluated)
MINERVA concept_personborninlocation MAP: 0.775690686628 (192 queries evaluated)
MINERVA concept_teamplayssport MAP: 0.762183612184 (111 queries evaluated)
MINERVA concept_athletehomestadium MAP: 0.519108225108 (200 queries evaluated)
MINERVA concept_worksfor MAP: 0.663530575465 (420 queries evaluated)

I did a few variation on embedding dimensions and also tried to freeze entity embeddings, yet none of the trials produced numbers close to the results tabulated in the MINERVA paper.

Would you please clarify the experiment setup for computing MAP?
I want to make sure I did set the hyperparameters to the correct value. Besides, the DeepPath paper used a relation-dependent underlying graph per relation during inference. Did you also vary the graph per relation or used a base graph for all relations like you did for other datasets?

Many thanks.

Question in code/model/agent.py

action = tf.to_int32(tf.multinomial(logits=scores, num_samples=1))  # [B, 1]

# loss
# 5a.
label_action =  tf.squeeze(action, axis=1)
loss = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=scores, labels=label_action)  # [B,]

the above code confused me, action is sampled from scores, and loss is cross entropy of action and scores, what's the meaning of the loss?

thank you!

Table 4 experiments

Hi @rajarshd, @shehzaadzd,

For all the baseline methods, do you train on train.txt or on both train.txt and graph.txt?
Do you have setting to reproduce the results? (e.g., Neural LP result is different from the ones in their paper for FB15k-237)

Thanks!

Reproduce WN18RR results

I am trying to reproduce the results of WN18RR.
I obtain much lower results despite using the predefined configs.

INFO:root:Hits@1: 0.3500
07/26/2018 02:40:17 PM: [ Hits@1: 0.3500 ]
INFO:root:Hits@3: 0.4346
07/26/2018 02:40:17 PM: [ Hits@3: 0.4346 ]

In the paper Hits@1 is 0.413 and Hits@3 0.456

Moreover, in a new paper they are claiming a sota result compared to these lower results which makes this very confusing...
https://arxiv.org/pdf/1802.04394.pdf

grid-world dataset access

Hi @rajarshd, @shehzaadzd,

When I access the grid world data, I got the following error. I am using git-lfs version 1.2.

git lfs fetch --all
Scanning for all objects ever referenced...
* 26 objects found
Fetching objects...
Git LFS: (0 of 26 files, 26 skipped) 0 B / 6.34 MB, 6.34 MB skipped
[404] Object does not exist on the server
[e16e1d42fbd288e20d73337bbce88b29e1d889cfb409de87b87a7d3d3b3c06e5] Object does not exist on the server 

Thanks!

Dataset Inconsistency

This dataset contains two files for training and the #lines doesn't add up.

https://github.com/shehzaadzd/MINERVA/tree/master/datasets/data_preprocessed/countries_S3

Can you please replace it with origin dataset or just leave a link/path to the origin?

Thank you.

BTW, great work!

Entity and relation embedding when use_entity_embeddings=0

Thanks for publishing the codebase. I have a question about pre-trained embeddings you mentioned in your paper.
I was reading through your code and could not find the pre-trained embeddings. Do you use the result of Xavier initialization as embeddings when use_entity_embeddings flag is 0? It's curious that you achieve a high performance in this setting. Have you tried to compare xavier initialization with embeddings from TransE? What embeddings did you use for the results in your paper?

Encode the relation and entity ?

Hello, I read your paper and codes and there is a confusion.

About encoding relation and entity, here is my understanding.
You don't encode them as vectors, instead using embedding matrices r and e and looking up in the embedding matrices according to their ids. And tf.nn.embedding_lookup() function could train the parameters in the embedding matrices.

Is that right?

Do you think KG can solve this kind of question?

1,
A lives in C. (A, liveInCity, C)
B lives in C. (B, liveInCity, C)
Does A and B live in the same city?

2,
A is 180 cm height. (A, height, 180)
B is 175 cm height. (B, height, 175)
Is B taller than A?

Thank you very much.

is the config file you provide the optimal parameter??

Thanks for reading my question!
is the config file you provide the optimal parameter?? I run a experiment on the dataset FB15K-237 with the default config file your provide . the results seems a lot different from the result in your paper?This is my result:
INFO:root:Hits@1: 0.1069
04/29/2018 12:23:52 AM: [ Hits@1: 0.1069 ]
INFO:root:Hits@3: 0.1904
04/29/2018 12:23:52 AM: [ Hits@3: 0.1904 ]
INFO:root:Hits@5: 0.2314
04/29/2018 12:23:52 AM: [ Hits@5: 0.2314 ]
INFO:root:Hits@10: 0.2864
04/29/2018 12:23:52 AM: [ Hits@10: 0.2864 ]
INFO:root:Hits@20: 0.3413
04/29/2018 12:23:52 AM: [ Hits@20: 0.3413 ]
INFO:root:auc: 0.1655
04/29/2018 12:23:52 AM: [ auc: 0.1655 ]
Could you help me to reproduce the result of your paper??

parameter settings used in paper

Hello thank you for your code, it works smoothly without any problem. I found parameter settings in your code are different than ones in your paper and MAP results were much lower. Would you please post parameter settings used for experiments in your paper? I really appreciate it if you could provide config.sh file for paper experiments. For example, batch size, beta, lambda total_iterations, entity_embeddings, and train_entity_embeddings so on:

data_input_dir="datasets/data_preprocessed/athleteplaysinleague/"
vocab_dir="datasets/data_preprocessed/athleteplaysinleague/vocab"
total_iterations=120
path_length=3
hidden_size=400
embedding_size=200
batch_size=128
beta=0.05
Lambda=0.05
use_entity_embeddings=1
train_entity_embeddings=1
train_relation_embeddings=1
base_output_dir="output/athleteplaysinleague/"
load_model=0
model_load_dir="/home/sdhuliawala/logs/RL-Path-RNN/wn18rrr/edb6_3_0.05_10_0.05/model/model.ckpt"
nell_evaluation=1

train/dev/test split

Why your dev triples are included in training data?

code/data/preprocessing_scripts/nell.py:
out_file.write(e1+'\t'+r+'\t'+e2+'\n')
if np.random.normal() > 0.2:
----dev.write(e1+'\t'+r+'\t'+e2+'\n')

Theoretically you are supposed to split it into 2 datasets (train/test) or 3 (train/dev/test) without overlaps. Please explain the reason behind this. Thank you.

meaning of last-step in return_next_actions

May I know what the meaning of this part is?

num_rollouts

Hi,
I'm getting the following error:

File ".../grapher.py", line 65, in return_next_actions if entities[j] in all_correct_answers[i/rollouts] and entities[j] != correct_e2: TypeError: list indices must be integers or slices, not float

This function is called in environment.py:

next_actions = self.grapher.return_next_actions(self.current_entities, self.start_entities, self.query_relation,self.end_entities, self.all_answers, self.current_hop == self.path_len - 1,self.num_rollouts)

I wonder what the value of num_rollouts should be to prevent i/rollouts to become a float (as far as I see, i is already an integer.

Is this code an official implementation to the method in paper "Neuro-Symbolic Inductive Logic Programming with Logical Neural Networks"?

On the "papers with code" website, this github repo is labeled as the official implementation of a paper named "Neuro-Symbolic Inductive Logic Programming with Logical Neural Networks" (https://paperswithcode.com/paper/neuro-symbolic-inductive-logic-programming), which seems to be a mistake.

the accuracy can be better after fine tune the function `calc_cum_discounted_reward ` as follows

the code is at

MINERVA/code/model/trainer.py

Lines 182 to 183 in d2f44ad

	running_add = self.gamma * running_add + cum_disc_reward[:, t]
	cum_disc_reward[:, t] = running_add

I modified the code as a normal way to calculate returns(just change the time t), it becomes better.
that is from :

for t in reversed(range(self.path_length)):
            running_add = self.gamma * running_add + cum_disc_reward[:, t]
            cum_disc_reward[:, t] = running_add

to :

for t in reversed(range(1, self.path_length)):
            running_add = self.gamma * running_add + cum_disc_reward[:, t]
            cum_disc_reward[:, t-1] = running_add

Thank you! I very appreciate you hard works, so I learned it line by line.

Question in code/model/trainer.py

Hi, I read your paper and codes and there is a confusion.
in trainer.py, What does ”new_scores = test_scores + beam_probs“ mean?
Why not use 'test_scores' or 'beam_probs' directly?

Thanks~

MINERVA/code/model/trainer.py

Line 338 in f706081

new_scores = test_scores + beam_probs

Reproduce Kinship, FB15k-237 results

Hi @rajarshd, @shehzaadzd,

Using the config for kinship and fb15k-237, we only get the following results, which are quite different from the ones in the paper.

FB15k-237
[ Hits@1:  0.1511 ]
[ Hits@3:  0.2183 ]
[ Hits@5:  0.2620 ]
[ Hits@10:  0.3275 ]
[ Hits@20:  0.3898 ]
[ auc:  0.2048 ] 

Kinship
[ Hits@1:  0.2607 ]
[ Hits@3:  0.4981 ]
[ Hits@5:  0.5819 ]
[ Hits@10:  0.6853 ]
[ Hits@20:  0.7551 ]
[ auc:  0.4048 ]

Related to #10, is there any fixes to reproduce the results?

Unable to load the saved_model

I am running the model for countries_s1 model but getting the following error:

Executing python code/model/trainer.py --base_output_dir output/countries_s1/ --path_length 2 --hidden_size 25 --embedding_size 25     --batch_size 256 --beta 0.05 --Lambda 0.05 --use_entity_embeddings 1     --train_entity_embeddings 1 --train_relation_embeddings 1     --data_input_dir datasets/data_preprocessed/countries_s1/ --vocab_dir datasets/data_preprocessed/countries_s1/vocab --model_load_dir saved_models/countries_s1/model.ckpt --load_model 1 --total_iterations 1000 --nell_evaluation 0
HEY
Arguments:
                         LSTM_layers : 1
                              Lambda : 0.05
                     base_output_dir : output/countries_s1/
                          batch_size : 256
                                beta : 0.05
                        create_vocab : 0
                      data_input_dir : datasets/data_preprocessed/countries_s1/
                      embedding_size : 25
                          eval_every : 100
                               gamma : 1
                      grad_clip_norm : 5
                         hidden_size : 25
                          input_file : train.txt
                         input_files : ['datasets/data_preprocessed/countries_s1//train.txt']
                        l2_reg_const : 0.01
                       learning_rate : 0.001
                          load_model : True
                             log_dir : ./logs/
                       log_file_name : output/countries_s1//3f71_2_0.05_100_0.05/log.txt
                     max_num_actions : 200
                           model_dir : output/countries_s1//3f71_2_0.05_100_0.05/model/
                      model_load_dir : saved_models/countries_s1/model.ckpt
                     negative_reward : 0
                     nell_evaluation : 0
                        num_rollouts : 20
                          output_dir : output/countries_s1//3f71_2_0.05_100_0.05
                         output_file :
                         path_length : 2
                    path_logger_file : output/countries_s1//3f71_2_0.05_100_0.05
                                pool : max
                     positive_reward : 1.0
        pretrained_embeddings_action :
        pretrained_embeddings_entity :
                       test_rollouts : 100
                    total_iterations : 1000
             train_entity_embeddings : True
           train_relation_embeddings : True
               use_entity_embeddings : True
                           vocab_dir : datasets/data_preprocessed/countries_s1/vocab
INFO:root:reading vocab files...
01/28/2019 01:56:20 AM: [ reading vocab files... ]
INFO:root:Reading mid to name map
01/28/2019 01:56:20 AM: [ Reading mid to name map ]
INFO:root:Done..
01/28/2019 01:56:20 AM: [ Done.. ]
INFO:root:Total number of entities 273
01/28/2019 01:56:20 AM: [ Total number of entities 273 ]
INFO:root:Total number of relations 6
01/28/2019 01:56:20 AM: [ Total number of relations 6 ]
INFO:root:Skipping training
01/28/2019 01:56:20 AM: [ Skipping training ]
INFO:root:Loading model from saved_models/countries_s1/model.ckpt
01/28/2019 01:56:20 AM: [ Loading model from saved_models/countries_s1/model.ckpt ]
Reading vocab...
batcher loaded
KG constructed
Reading vocab...
Contains full graph
batcher loaded
KG constructed
Reading vocab...
Contains full graph
batcher loaded
KG constructed
2019-01-28 01:56:21.112771: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2019-01-28 01:56:21.112794: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2019-01-28 01:56:21.112801: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2019-01-28 01:56:21.112806: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2019-01-28 01:56:21.112812: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
2019-01-28 01:56:21.988648: I tensorflow/core/common_runtime/gpu/gpu_device.cc:955] Found device 0 with properties:
name: GeForce GTX 1080 Ti
major: 6 minor: 1 memoryClockRate (GHz) 1.582
pciBusID 0000:06:00.0
Total memory: 10.91GiB
Free memory: 10.75GiB
2019-01-28 01:56:21.988740: I tensorflow/core/common_runtime/gpu/gpu_device.cc:976] DMA: 0
2019-01-28 01:56:21.988757: I tensorflow/core/common_runtime/gpu/gpu_device.cc:986] 0:   Y
2019-01-28 01:56:21.988799: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:06:00.0)
INFO:root:Creating TF graph...
01/28/2019 01:56:22 AM: [ Creating TF graph... ]
INFO:root:TF Graph creation done..
01/28/2019 01:56:23 AM: [ TF Graph creation done.. ]
INFO:tensorflow:Restoring parameters from saved_models/countries_s1/model.ckpt
INFO:tensorflow:Restoring parameters from saved_models/countries_s1/model.ckpt
01/28/2019 01:56:23 AM: [ Restoring parameters from saved_models/countries_s1/model.ckpt ]
2019-01-28 01:56:23.830867: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
2019-01-28 01:56:23.846931: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847248: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847249: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847266: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847290: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847248: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847342: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847358: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847437: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847529: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847785: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847753: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847603: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847794: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847881: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847816: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.848095: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.848317: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.848375: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.854893: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.854890: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.854966: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.854933: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.855353: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.855395: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam not found in checkpoint
2019-01-28 01:56:23.855675: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.855750: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.855791: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.855820: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.855785: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.856273: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.856468: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.856486: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.856592: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.856623: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.856677: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
Traceback (most recent call last):
  File "code/model/trainer.py", line 573, in <module>
    trainer.initialize(restore=save_path, sess=sess)
  File "code/model/trainer.py", line 143, in initialize
    return  self.model_saver.restore(sess, restore)
  File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1560, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 895, in run
    run_metadata_ptr)
  File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1124, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1321, in _do_run
    options, run_metadata)
  File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1340, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
         [[Node: save/RestoreV2_17/_3 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_80_save/RestoreV2_17", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]

Caused by op u'save/RestoreV2_9', defined at:
  File "code/model/trainer.py", line 573, in <module>
    trainer.initialize(restore=save_path, sess=sess)
  File "code/model/trainer.py", line 137, in initialize
    self.model_saver = tf.train.Saver(max_to_keep=2)
  File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1140, in __init__
    self.build()
  File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1172, in build
    filename=self._filename)
  File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 688, in build
    restore_sequentially, reshape)
  File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 407, in _AddRestoreOps
    tensors = self.restore_op(filename_tensor, saveable, preferred_shard)
  File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 247, in restore_op
    [spec.tensor.dtype])[0])
  File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/ops/gen_io_ops.py", line 663, in restore_v2
    dtypes=dtypes, name=name)
  File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
    op_def=op_def)
  File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2630, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1204, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

NotFoundError (see above for traceback): Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
         [[Node: save/RestoreV2_17/_3 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_80_save/RestoreV2_17", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]

A whole solution for question-answering?

I have not found the natural language questions in the datasets.
So I think I should turn the natural language questions to logic form like (e1,r,?) first.
Am I right? Thank you very much!

sharing the pre-trained embeddings learned by Dismult

I saw that your baseline such as Dismult achieved very promising results and can be used to initialize other models. Would you mind sharing the pre-trained embeddings learned by Dismult as presented in Table 4?

Dataset definition

Hi, I noticed in your code, the dataset contains five files.

full_graph.txt, graph.txt, train/dev/test.txt

I was wondering what's those files for respectively? I know that train/dev/test.txt is for train/dev/test, but what is graph.txt/full_graph.txt used for? The length of full_graph.txt file isn't equal to the sum of train/dev/test files' length. And what's the difference between full_graph.txt and graph.txt?

Thanks!

How do you extract the general rules from the trained policy(like the (i) example in table 8)？

Thanks for your code! But I have some doubts about your displayed general rules.
Is it just some similar result in test set or you come up with some guidelines for induction?

Pytorch implementation for MINERVA

Thanks for your code.

I want to implement MINERVA with pytorch further research. I implemented my own version according to your source code, but unfortunately I couldn't achieve the similar results as yours.

My implementation is available at https://github.com/ty4b112/pytorch_MINERVA. It will be very helpful if someone use my code and find why my implementation is worse than the original version or provide another pytorch implementation with better performance.

How to process partially structured question？

The Table.6 proposed experiment on WikiQA, but I don't see the implement of this part in the code. Could you give me some instruction on how to process these questions with the code？

Export the mode using `tf.saved_model` and deploy it to production environment

Sorry to bother.
When I'm trying to export the model using tf.saved_model to deploy it to a production environment (tf serving in docker), there are some problems.
The exported graph format requires pure inputs and outputs signature, but in our model,the agent needs to interact with an environment. But all of the codes are written in numpy rather than tensorflow except the agent.py. So if I export the agent only(maybe failed), how the agent interacts with the environment?
My idea: rewrite the environment in TF... but this is going to be a lot of work.

Thanks in advance.

Testing problem

Thanks for publishing the codebase.I have a question about testing.I set it up exactly as you mentioned in the README documentation.The rest arguments of testing are same as training.For example,'load_model=1' and 'model_load_dir="/output/worksfor/b2b4_3_0.05_100_0.05/model/model.ckpt"'.But I got the following error:

INFO:tensorflow:Restoring parameters from /output/worksfor/b2b4_3_0.05_100_0.05/model/model.ckpt
INFO:tensorflow:Restoring parameters from /output/worksfor/b2b4_3_0.05_100_0.05/model/model.ckpt
07/06/2019 03:42:44 PM: [ Restoring parameters from /output/worksfor/b2b4_3_0.05_100_0.05/model/model.ckpt ]
2019-07-06 15:42:44.906846: W tensorflow/core/framework/op_kernel.cc:1318] OP_REQUIRES failed at save_restore_tensor.cc:170 : Invalid argument: Unsuccessful TensorSliceReader constructor: Failed to get matching files on /output/worksfor/b2b4_3_0.05_100_0.05/model/model.ckpt: Not found: /output/worksfor/b2b4_3_0.05_100_0.05/model; No such file or directory
Traceback (most recent call last):
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1322, in _do_call
    return fn(*args)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Unsuccessful TensorSliceReader constructor: Failed to get matching files on /output/worksfor/b2b4_3_0.05_100_0.05/model/model.ckpt: Not found: /output/worksfor/b2b4_3_0.05_100_0.05/model; No such file or directory
	 [[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "code/model/trainer.py", line 574, in <module>
    trainer.initialize(restore=save_path, sess=sess)
  File "code/model/trainer.py", line 144, in initialize
    return  self.model_saver.restore(sess, restore)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1802, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 900, in run
    run_metadata_ptr)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1135, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1316, in _do_run
    run_metadata)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1335, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Unsuccessful TensorSliceReader constructor: Failed to get matching files on /output/worksfor/b2b4_3_0.05_100_0.05/model/model.ckpt: Not found: /output/worksfor/b2b4_3_0.05_100_0.05/model; No such file or directory
	 [[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

Caused by op 'save/RestoreV2', defined at:
  File "code/model/trainer.py", line 574, in <module>
    trainer.initialize(restore=save_path, sess=sess)
  File "code/model/trainer.py", line 138, in initialize
    self.model_saver = tf.train.Saver(max_to_keep=2)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1338, in __init__
    self.build()
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1347, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1384, in _build
    build_save=build_save, build_restore=build_restore)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 835, in _build_internal
    restore_sequentially, reshape)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 472, in _AddRestoreOps
    restore_sequentially)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 886, in bulk_restore
    return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/ops/gen_io_ops.py", line 1463, in restore_v2
    shape_and_slices=shape_and_slices, dtypes=dtypes, name=name)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3392, in create_op
    op_def=op_def)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1718, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): Unsuccessful TensorSliceReader constructor: Failed to get matching files on /output/worksfor/b2b4_3_0.05_100_0.05/model/model.ckpt: Not found: /output/worksfor/b2b4_3_0.05_100_0.05/model; No such file or directory
	 [[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]