bxshi/conmask

ConMask model described in paper Open-world Knowledge Graph Completion.

conmask's Issues

Some questions about the paper

I get stuck in the ConMask Model illustration in the paper Figure 1. what dose w1,w2 .. w7 mean?

I think the w is a similarity score between two vectors. Am I right?

libsampling.so ??? No such file or directory

tensorflow.python.framework.errors_impl.NotFoundError: /home/ftlxj/XiaojunLiu/ConMask/ConMask/ndkgc/ops/./__sampling/libsampling.so: cannot open shared object file: No such file or directory

Error When running Closed/Open World Evaluation

Hi! Thanks for your code! I am trying to run the Closed and Open world evaluation on the Dbpedia50 dataset, following the instructions in README. However, I run into an error on running the command:
tensorflow.python.framework.errors_impl.NotFoundError: /home/bhattacharyya/code/ConMask/ndkgc/ops/./__sampling/libsampling.so: undefined symbol: _ZN10tensorflow8internal21CheckOpMessageBuilder9NewStringB5cxx11Ev

I followed a similar closed issue and saw that the CMake file I am using already has the change suggested in that. Additional info:
I was able to compile the pairwise sampler successfully.
I am using Python 3.6, tf 1.3.
Pasting the screenshot when running the Open world evaluation.

Would be very helpful if anything can be suggested! Thanks!

How to create dataset

If I want to create a new dataset. How can I get those files, such as vocab.txt, train.tails.values and eval.tails.values.closed?

some question about DKRL

hello , bxshi:
could you share the running time about the model of dkrl in your code, and i notice that in your paper, the statistic of dkrl is different from the origin paper. I am confusing .

Looking forward to hearing from you

Error when I run the codes

after I run the cmakeList to generate the file,libsampling.so,I try to run the command you give in the site.But A error happened and I can't find a correct way to fix it,could you help me ?

libsampling.so: undefined symbol: _ZTIN10tensorflow8OpKernelE

My running environment is :python 3.6.5; tensorlfow 1.9.0; gcc 5.5
I heve already tried countless ways on the web ,Hope you can help me soon

Error when I run the dkrl

Hi bxshi, when I run the dkrl.py, always raise
ValueError: Cannot feed value of shape (2, 14904) for Tensor 'static/content_matrix/initial_value:0', which has shape '(14904,)'

https://github.com/bxshi/ConMask/blob/master/ndkgc/models/dkrl.py#L202

I can't find a correct way to fix it,could you help me ?

Questions about datasets and code

Hi, thanks for your code. When I read your code, I run into some troubles.

1. How you get train.tails.idx? The length of train.tails.idx is not equal to train.txt. So as train.heads.idx.
2. How you get vocab.txt? I find not all words in descriptions exsists in vocab.txt.
3. I run the code on 8G2 GPUs, OOM(out of memories) in No.1 global step.
   On 16G4 GPUs, OOM(out of memories) in No.4848 step.
   I use
   python3 -m ndkgc.models.fcn_model_v2 checkpoint_result data/dbpedia50 --force_eval --layer 3 --conv 2 --lr 1e-2 --keep_prob 0.5 --max_content 512 --pos 1 --neg 4 --open --neval 5000 --filter
   Look forward to your reply.

How preprosessing of dbpedia was done?

Thanks for uploading code. I'm really interested in your work.
My question is how to preprocess dbpedia datasets, because descriptions.txt contains, something like
Paris,_Texas 2056 paris texas city county seat lamar county texas united states 2010 census population city 25 171 situated northeast texas western edge piney woods northeast dallas–fort worth metroplex ...

and I can't find out where the sentence boundary is. Also, I'd like to replace lemmatized token to original one.

If you'd know how the preprocessing of dbpedia was done, I'd really appreciate it.
Thanks.

stucker in codes

lab@matrix4:~/lyz$ git clone https://github.com/bxshi/ConMask.git
Cloning into 'ConMask'...
remote: Counting objects: 117, done.
remote: Total 117 (delta 0), reused 0 (delta 0), pack-reused 117
Receiving objects: 100% (117/117), 85.09 KiB | 49.00 KiB/s, done.
Resolving deltas: 100% (49/49), done.
Checking connectivity... done.
lab@matrix4:~/lyz$ cd ConMask/
lab@matrix4:~/lyz/ConMask$ ls
ndkgc  README.md  tools
lab@matrix4:~/lyz/ConMask$ mkdir data
lab@matrix4:~/lyz/ConMask$ ls
data  ndkgc  README.md  tools
lab@matrix4:~/lyz/ConMask$ cd ../
lab@matrix4:~/lyz$ mv dbpedia50 dbpedia500 ./ConMask/data
lab@matrix4:~/lyz$ mv checkpoint_db50_v2_dr_uniweight_2 ./ConMask/
lab@matrix4:~/lyz$ cd ConMask/
lab@matrix4:~/lyz/ConMask$ ls
checkpoint_db50_v2_dr_uniweight_2  data  ndkgc  README.md  tools
lab@matrix4:~/lyz/ConMask$ cd ndkgc/
lab@matrix4:~/lyz/ConMask/ndkgc$ cd ops/
lab@matrix4:~/lyz/ConMask/ndkgc/ops$ ls
content.py     __init__.py  multigpu.py  sampling.py
corruption.py  lookup.py    __sampling
lab@matrix4:~/lyz/ConMask/ndkgc/ops$ cd __sampling/
lab@matrix4:~/lyz/ConMask/ndkgc/ops/__sampling$ ls
CMakeLists.txt  pairwise_sampler.cpp
lab@matrix4:~/lyz/ConMask/ndkgc/ops/__sampling$ cmake .
-- The C compiler identification is GNU 5.4.0
-- The CXX compiler identification is GNU 5.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- OS Type: Linux
-- TensorFlow include dir: /home/lab/anaconda3/lib/python3.6/site-packages/tensorflow/include
-- Configuring done
-- Generating done
-- Build files have been written to: /home/lab/lyz/ConMask/ndkgc/ops/__sampling
lab@matrix4:~/lyz/ConMask/ndkgc/ops/__sampling$ cmake --build .
Scanning dependencies of target sampling
[ 50%] Building CXX object CMakeFiles/sampling.dir/pairwise_sampler.cpp.o
[100%] Linking CXX shared library libsampling.so
[100%] Built target sampling
lab@matrix4:~/lyz/ConMask/ndkgc/ops/__sampling$ cd ../../
lab@matrix4:~/lyz/ConMask/ndkgc$ cd ../
lab@matrix4:~/lyz/ConMask$ source activate root
(base) lab@matrix4:~/lyz/ConMask$ python3 -m ndkgc.models.fcn_model_v2 checkpoint_db50_v2_dr_uniweight_2 data/dbpedia50 --force_eval --layer 3 --conv 2 --lr 1e-2 --keep_prob 0.5 --max_content 512 --pos 1 --neg 4 --open --neval 5000 --eval --filter
Traceback (most recent call last):
  File "/home/lab/anaconda3/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/lab/anaconda3/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/lab/lyz/ConMask/ndkgc/models/fcn_model_v2.py", line 6, in <module>
    from ndkgc.models.content_model import ContentModel
  File "/home/lab/lyz/ConMask/ndkgc/models/content_model.py", line 6, in <module>
    from ndkgc.ops import *
  File "/home/lab/lyz/ConMask/ndkgc/ops/__init__.py", line 1, in <module>
    from ndkgc.ops.corruption import *
  File "/home/lab/lyz/ConMask/ndkgc/ops/corruption.py", line 6, in <module>
    from ndkgc.ops.sampling import single_negative_sampling, multiple_negative_sampling
  File "/home/lab/lyz/ConMask/ndkgc/ops/sampling.py", line 5, in <module>
    __lib = tf.load_op_library(os.path.join(__so_dir, './__sampling/libsampling.so'))
  File "/home/lab/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/load_library.py", line 64, in load_op_library
    None, None, error_msg, error_code)
tensorflow.python.framework.errors_impl.NotFoundError: /home/lab/lyz/ConMask/ndkgc/ops/./__sampling/libsampling.so: undefined symbol: _ZN10tensorflow7strings6StrCatB5cxx11ERKNS0_8AlphaNumE

Can I just use 2 GPU?OR reduce the batch size?

Can I just use 2 GPU?OR reduce the batch size?

How to reduce gpu to one?

Hello, thanks for your code very mush!
I recently use your code to train DKRL model, but it needs 4 GPUs, which my machine just have one. And from Readme I know it can reduce the gpus to one, but I don't konw how to change it. Would you mind to tell me briefly how to change the code?

the datasets about dkrl

DATA_DIR = './data/fb15k/'
model = DKRL(
    entity_file=os.path.join(DATA_DIR, 'entities.txt'),
    relation_file=os.path.join(DATA_DIR, 'relations.txt'),
    vocab_file=os.path.join(DATA_DIR, 'vocab.txt'),
    pretrain_vocab_file=os.path.join(DATA_DIR, 'glove.6B.100d.txt'),
    content_file=os.path.join(DATA_DIR, 'descriptions.txt'),
    train_file=os.path.join(DATA_DIR, 'train.txt'),
    valid_file=os.path.join(DATA_DIR, 'valid.txt'),
    test_file=os.path.join(DATA_DIR, 'test.txt'),
    all_triples_file=os.path.join(DATA_DIR, 'all_triples.txt')
)

Can the above data set be sent to me ?

/mutex.h:25:22: fatal error: nsync_cv.h: No such file or directory

mldl@mldlUB1604:~/ub16_prj/ConMask/ndkgc/ops/__sampling$ cmake --build .
[ 50%] Building CXX object CMakeFiles/sampling.dir/pairwise_sampler.cpp.o
In file included from /usr/local/lib/python3.5/dist-packages/tensorflow/include/tensorflow/core/platform/mutex.h:31:0,
                 from /usr/local/lib/python3.5/dist-packages/tensorflow/include/tensorflow/core/framework/op.h:32,
                 from /home/mldl/ub16_prj/ConMask/ndkgc/ops/__sampling/pairwise_sampler.cpp:7:
/usr/local/lib/python3.5/dist-packages/tensorflow/include/tensorflow/core/platform/default/mutex.h:25:22: fatal error: nsync_cv.h: No such file or directory
compilation terminated.
CMakeFiles/sampling.dir/build.make:62: recipe for target 'CMakeFiles/sampling.dir/pairwise_sampler.cpp.o' failed
make[2]: *** [CMakeFiles/sampling.dir/pairwise_sampler.cpp.o] Error 1
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/sampling.dir/all' failed
make[1]: *** [CMakeFiles/sampling.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2
mldl@mldlUB1604:~/ub16_prj/ConMask/ndkgc/ops/__sampling$