rowanz/neural-motifs

Code for Neural Motifs: Scene Graph Parsing with Global Context (CVPR 2018)

neural-motifs

Like this work, or scene understanding in general? You might be interested in checking out my brand new dataset VCR: Visual Commonsense Reasoning, at visualcommonsense.com!

This repository contains data and code for the paper Neural Motifs: Scene Graph Parsing with Global Context (CVPR 2018) For the project page (as well as links to the baseline checkpoints), check out rowanzellers.com/neuralmotifs. If the paper significantly inspires you, we request that you cite our work:

Bibtex

@inproceedings{zellers2018scenegraphs,
  title={Neural Motifs: Scene Graph Parsing with Global Context},
  author={Zellers, Rowan and Yatskar, Mark and Thomson, Sam and Choi, Yejin},
  booktitle = "Conference on Computer Vision and Pattern Recognition",  
  year={2018}
}

Setup

0. Install python3.6 and pytorch 3. I recommend the Anaconda distribution. To install PyTorch if you haven't already, use conda install pytorch=0.3.0 torchvision=0.2.0 cuda90 -c pytorch.

1. Update the config file with the dataset paths. Specifically:

   • Visual Genome (the VG_100K folder, image_data.json, VG-SGG.h5, and VG-SGG-dicts.json). See data/stanford_filtered/README.md for the steps I used to download these.
   • You'll also need to fix your PYTHONPATH: export PYTHONPATH=/home/rowan/code/scene-graph

2. Compile everything. run make in the main directory: this compiles the Bilinear Interpolation operation for the RoIs as well as the Highway LSTM.

3. Pretrain VG detection. The old version involved pretraining COCO as well, but we got rid of that for simplicity. Run ./scripts/pretrain_detector.sh Note: You might have to modify the learning rate and batch size, particularly if you don't have 3 Titan X GPUs (which is what I used). You can also download the pretrained detector checkpoint here.

4. Train VG scene graph classification: run ./scripts/train_models_sgcls.sh 2 (will run on GPU 2). OR, download the MotifNet-cls checkpoint here: Motifnet-SGCls/PredCls.

5. Refine for detection: run ./scripts/refine_for_detection.sh 2 or download the Motifnet-SGDet checkpoint.

6. Evaluate: Refer to the scripts ./scripts/eval_models_sg[cls/det].sh.

help

Feel free to open an issue if you encounter trouble getting it to work!

neural-motifs's Issues

A small problem about w & h of an image.

Hi, Rowan,
Thanks for sharing your code. I've noticed maybe a small bug. In visual_genome.py, getitem function:

w, h = image_unpadded.size

if h > w:
        im_size = (IM_SCALE, int(w * img_scale_factor), img_scale_factor)
else:
        ...

why the order of w & h is reversed?

I'm not very sure because I've not read the code thoroughly. If this is a bug, will it influence the following codes?

Thanks!

cudaCheckError() failed : invalid device function

I use the cuda8.0 pytroch0.4 and python3.6
how can I do to solve the problem?

Questions about the recalls of Scene Graph Classification

Thanks for your excellent work.
When I evaluated the recalls of Scene Graph Classification using your pretrained sgcls model(vgrel-motifnet-sgcls.tar), I got higher results than your paper's.

======================sgcls============================
R@100: 0.417541  vs 36.5 in your paper
R@50: 0.410185 vs 35.8 in your paper
R@20: 0.384014 vs 32.9 in your paper 

What's the reason? the better detector?

Running for a single image using _visualize.py

Hey thanks for sharing the code..
Running this code ..I have got the model and in rel_model.py getitem when I pass the image tensor to the model ..
Error is forward () takes at least 4 argument (2 given)
In nn.model.py

After changing pytorch 0.4.0 to 0.3.0 glove file should be re-downloaded(solved)

Hi, Roman：
I have change pytorch 0.4.0 to 0.3.0; Under 0.4.0, the code can run the baseline model (./scripts/train_models_sgcls.sh 0); it downloads the glove file for the first time and process it.
However, after changing the pytorch version, the processed files are not compatible with 0.3.0 and the code does not judge it and report errors.
So, the processed files should be removed or remove all the glove file and let the code download and process it under the current pytorch version.

About the usage of flags

Hey! Really appreciate this amazing work.
I have a little question about these four flags: -gt_box, -old_feat, -pass_in_obj_feats_to_decoder, -pass_in_obj_feats_to_edge.
Could you please explain these flags? Thank you a lot.

Btw, I am trying to use MaskRCNN to extract features and predict the relations between instances. Is there any suggestions? Hope to hear from you soon!

Best wishes,
HsuanKung

RuntimeError: cuda runtime error (8) in nms

Hi,
I have encounted the following error when I run the code on two Titan XP:

THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1518238409320/work/torch/lib/THC/generic/THCTensorMathPairwise.cu line=21 error=8 : invalid device function
Traceback (most recent call last):
File "/home/wtliao/.pycharm_helpers/pydev/pydevd.py", line 1668, in
main()
File "/home/wtliao/.pycharm_helpers/pydev/pydevd.py", line 1662, in main
globals = debugger.run(setup['file'], None, None, is_module)
File "/home/wtliao/.pycharm_helpers/pydev/pydevd.py", line 1072, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "/home/wtliao/work_space/neural-motifs-master/models/train_detector.py", line 214, in
rez = train_epoch(epoch)
File "/home/wtliao/work_space/neural-motifs-master/models/train_detector.py", line 70, in train_epoch
tr.append(train_batch(batch))
File "/home/wtliao/work_space/neural-motifs-master/models/train_detector.py", line 103, in train_batch
result = detector[b]
File "/home/wtliao/work_space/neural-motifs-master/lib/object_detector.py", line 418, in getitem
outputs = nn.parallel.parallel_apply(replicas, [batch[i] for i in range(self.num_gpus)])
File "/home/wtliao/anaconda2/envs/pytorch/lib/python2.7/site-packages/torch/nn/parallel/parallel_apply.py", line 67, in parallel_apply
raise output

After debug line by line, I find that this error arises in the operation: keep.append(keep_im + s), line 24 in nms.py

Any idea to solve it? Thanks!

Generating Scene Graphs for random Images

@rowanz ,
I wanted to generate scene graphs for images not belonging to the Visual Genome dataset, using your model. Can I do this using the existing codebase? If yes, can you please guide me through it?

Hi,do you have been balancing image ,because some relations just have a little

can you release your pycocotools code?

model for eval

Hi,
I want to reimplement the baseline evaluation. But I found that, the pretrained model for baseline evaluation is not provided. Could you share them? Thanks!.

1. checkpoints/baseline-sgdet/vgrel-17.tar
2. stanford_sgdet.pkl
3. checkpoints/motifnet-sgdet/vgrel-14.tar
4. checkpoints/baseline-sgcls/vgrel-11.tar
5. checkpoints/motifnet-sgcls/vgrel-7.tar

Question about the code of rel_stanford

Hi~

my problem is about the stanford model of your code. the scripts of train_stanford.sh includes the item of "-ckpt checkpoints/vgdet/vg-24.tar" , but i don't find some explaining or download links about "vgdet/vg-24.tar" .
So, can you tell me the details of this file and how to get it, and more implementation details about standord model [like ..epochs ? ]

I hope you can answer my questions. thanks~~

Why the checkpoint of PRCLS and SGCLS are the same thing?

Thanks for your sharing. I am a freshman in this topic. According to the checkpoint you posted, it seems that two different task using the same checkpoint. Could you explain why we do not need to train different task separately? If what I think is true, Could you share the checkpoint of PRCLS/PRDET with me?

how to run a demo for my image

thanks for your sharing. i want to run a demo for my images and visualize the result. i have already downloaded the models from your website. How should i do? Thanks very much.

some problem about nms

Hi Rowan, thanks for your codes sharing. However, I find nms.nms_apply(keep, boxes_sorted, nms_thresh) in lib/fpn/nms/functions/nms.py would make pytorch break up. Before I execute that line, the operation of Tensor.cuda()+int() is legal, but after I execute that line this operation will raise error: RuntimeError: cuda runtime error (48) : no kernel image is available for execution on the device at /pytorch/torch/lib/THC/generic/THCTensorMathPairwise.cu:21 when it goes on keep_im + s at line 24. Could you please help me? Thanks!

The checkpoint file is damaged?

Hello, I have downloaded the chechpoint file "vgrel-motifnet-sgdet.tar",but i can't uncompress it. Maybe it is damaged? Can you reupload it again?

import issues

while trying to train i'm getting the next error

n.anchor_targets import anchor_target_layer
File "/home/sgalk87/neural-motifs/lib/fpn/anchor_targets.py", line 12, in
from lib.fpn.box_intersections_cpu.bbox import bbox_overlaps
ImportError: No module named 'lib.fpn.box_intersections_cpu.bbox'
seems like a problem with the relative imports, did you solve it?

It is seems that the pretrained detector checkpoint is not available

I have downloaded the pretrained detector checkpoint many times, However, the error shows that the archive is corrupted.

Training sgcls with ResNet ?

Hi!
I have successfully pretrained an object detector with ResNet but I've been unable to to train the rel detection model. I think there is something I do not understand (namely the differences between and use cases of detector.roi_fmap and rel_model.roi_fmap).
I guess the ResNet part of the code is depecrated now, do you have an updated version?
Or some ideas about how to update it?
I would be very grateful :-)

How to implement the "NO SCENE GRAPH CONSTRAINT” on associate embedding?

To my best knowledge, the experiment results in associate embedding (9.7, 11.3..... ) should be under the constraint of the scene graph. However, you have cited the result of associate embedding in the column of non-scene graph constraint. Could you help me to figure it out? Thanks

Training Object Detector and Motifnet jointly

Hi did you experiment unfreezing the detector's weights while training morifnet?

different result on cuda8 and cuda9

Hi Rowan, I'm running your codes now, but I found I'll get totally different result on cuda8 and cuda9 when I train detector(I don't use lstm here). Results from cuda9 are reasonable, while the loss from cuda8 will not decrease after several iterations. Could you please tell me why it happens and I can solve it? I need to put the project to a server with cuda8 and I can't change cuda version for that server. Thank you!

how to load the checkpoints for pretrain detector, MotifNet-cls and Motifnet-SGDet

Hi,
I know it is kind of stupid, but how can I load those three checkpoints? In the bash codes, I dod see something like checkpoints/baseline-sgdet/vgrel-17.tar, but what are these? shall I replace them with the download one?

Thank you!

A problem about nms

Hi Rowan, thanks for your codes sharing. However, I find nms.nms_apply(keep, boxes_sorted, nms_thresh) in lib/fpn/nms/functions/nms.py would make pytorch break up. Before I execute that line, the operation of Tensor.cuda()+int() is legal, but after I execute that line this operation will raise error: RuntimeError: cuda runtime error (48) : no kernel image is available for execution on the device at /pytorch/torch/lib/THC/generic/THCTensorMathPairwise.cu:21 when it goes on keep_im + s at line 24. Could you please help me? Thanks!

Using resnet-152 instead of vgg-16 will get better performance?

how to run the code

Hi, thanks for sharing your impressive work and code. When I try to run the code, I have enountered some problems.

1. when I ran ./scripts/pretrain_detector.sh, the error is
   Traceback (most recent call last):
   File "models/train_detector.py", line 7, in
   from dataloaders.mscoco import CocoDetection, CocoDataLoader
   ModuleNotFoundError: No module named 'dataloaders'

I try to comment the import module, and then the error arises for importing other modules. I try to directly import any of the given module in python. It works. It seems that it can't recognized the module when I try to run the .sh. This error also presents when I run other .sh file. Could you give me a solution that I don't need to run any of them in .sh. Thanks a lot

what's wrong in eval???? VGDataLoader.split ValueError: not enough values to unpack (expected 3, got 2)

I have run the code once, everything goes right. But today, I want to check the result again ,and i run script/eval......sh, and got this :
**ValueError: not enough values to unpack (expected 3, got 2)
**

I recheck the code and error comes from here:
train_loader, val_loader = VGDataLoader.splits(train, val, mode='rel',
batch_size=conf.batch_size,
num_workers=conf.num_workers,
num_gpus=conf.num_gpus)

and the split defined here:
class VGDataLoader(torch.utils.data.DataLoader):
"""
Iterates through the data, filtering out None,
but also loads everything as a (cuda) variable
"""

@classmethod
def splits(cls, train_data, val_data, batch_size=3, num_workers=1, num_gpus=3, mode='det',
           **kwargs):
    assert mode in ('det', 'rel')
    train_load = cls(
        dataset=train_data,
        batch_size=batch_size * num_gpus,
        shuffle=True,
        num_workers=num_workers,
        collate_fn=lambda x: vg_collate(x, mode=mode, num_gpus=num_gpus, is_train=True),
        drop_last=True,
        # pin_memory=True,
        **kwargs,
    )
    val_load = cls(
        dataset=val_data,
        batch_size=batch_size * num_gpus if mode=='det' else num_gpus,
        shuffle=False,
        num_workers=num_workers,
        collate_fn=lambda x: vg_collate(x, mode=mode, num_gpus=num_gpus, is_train=False),
        drop_last=True,
        # pin_memory=True,
        **kwargs,
    )
    return train_load, val_load

miss file "lib/rel_model_baseline.py"

Thanks for your nice code. The file "lib/rel_model_baseline.py" is missing, could you please upload it and its checkpoints, thank you!

How to generate the scene graph of a single image?

Hello,
How to generate the scene graph of a single image, instead of run the evaluation on the whole dataset?
Can you tell me more about that?
Thanks.

Not sure RelModel is learning

while running train_rels.py -m sgcls -model motifnet -order leftright -nl_obj 2 -nl_edge 4 -b 6 -clip 5
-p 100 -hidden_dim 512 -pooling_dim 4096 -lr 1e-3 -ngpu 1 -ckpt checkpoints/vgdet/vg-24.tar
-save_dir checkpoints/motifnet2 -nepoch 50 -use_bias

my rel loss is constantly around 0.15-0.12 is this the loss you got while training , did you see any improvements, do you think the model might not learn like expected?

i'm getting similar results to what you published.

problems with preforming eval

im running the next command
python3 models/eval_rels.py -m sgcls -model motifnet -order leftright -nl_obj 2 -nl_edge 4 -b 6 -clip 5
-p 100 -hidden_dim 512 -pooling_dim 4096 -lr 1e-3 -ngpu 1 -test -ckpt checkpoints/motifnet2/vgrel-10.tar -nepoch 50 -cache motifnet_sgcls -use_bias

0%| | 0/26446 [00:00<?, ?it/s]~~~~~~~~ Hyperparameters used: ~~~~~~~
old_feats : False
pass_in_obj_feats_to_decoder : False
coco : False
rec_dropout : 0.1
batch_size : 6
nl_edge : 4
print_interval : 100
order : leftright
cache : motifnet_sgcls
model : motifnet
val_size : 5000
l2 : 0.0001
mode : sgdet
use_resnet : False
hidden_dim : 512
clip : 5.0
adam : False
use_tanh : False
lr : 0.001
multi_pred : True
use_proposals : False
num_workers : 1
num_gpus : 1
nl_obj : 2
pooling_dim : 4096
num_epochs : 50
pass_in_obj_feats_to_edge : False
test : True
limit_vision : False
det_ckpt :
gt_box : False
ckpt : checkpoints/motifnet2/vgrel-5.tar
save_dir :
use_bias : True
loading word vectors from /home/sgalk87/neural-motifs/data/glove.6B.200d.pt
background -> background
fail on background
loading word vectors from /home/sgalk87/neural-motifs/data/glove.6B.100d.pt
background -> background
fail on background
Unexpected key module.detector.features.0.weight in state_dict with size torch.Size([64, 3, 3, 3])
Unexpected key module.detector.features.0.bias in state_dict with size torch.Size([64])
Unexpected key module.detector.features.2.weight in state_dict with size torch.Size([64, 64, 3, 3])
Unexpected key module.detector.features.2.bias in state_dict with size torch.Size([64])
Unexpected key module.detector.features.5.weight in state_dict with size torch.Size([128, 64, 3, 3])
Unexpected key module.detector.features.5.bias in state_dict with size torch.Size([128])
Unexpected key module.detector.features.7.weight in state_dict with size torch.Size([128, 128, 3, 3])
Unexpected key module.detector.features.7.bias in state_dict with size torch.Size([128])
Unexpected key module.detector.features.10.weight in state_dict with size torch.Size([256, 128, 3, 3])
Unexpected key module.detector.features.10.bias in state_dict with size torch.Size([256])
Unexpected key module.detector.features.12.weight in state_dict with size torch.Size([256, 256, 3, 3])
Unexpected key module.detector.features.12.bias in state_dict with size torch.Size([256])
Unexpected key module.detector.features.14.weight in state_dict with size torch.Size([256, 256, 3, 3])
Unexpected key module.detector.features.14.bias in state_dict with size torch.Size([256])
Unexpected key module.detector.features.17.weight in state_dict with size torch.Size([512, 256, 3, 3])
Unexpected key module.detector.features.17.bias in state_dict with size torch.Size([512])
Unexpected key module.detector.features.19.weight in state_dict with size torch.Size([512, 512, 3, 3])
Unexpected key module.detector.features.19.bias in state_dict with size torch.Size([512])
Unexpected key module.detector.features.21.weight in state_dict with size torch.Size([512, 512, 3, 3])
Unexpected key module.detector.features.21.bias in state_dict with size torch.Size([512])
Unexpected key module.detector.features.24.weight in state_dict with size torch.Size([512, 512, 3, 3])
Unexpected key module.detector.features.24.bias in state_dict with size torch.Size([512])
Unexpected key module.detector.features.26.weight in state_dict with size torch.Size([512, 512, 3, 3])
Unexpected key module.detector.features.26.bias in state_dict with size torch.Size([512])
Unexpected key module.detector.features.28.weight in state_dict with size torch.Size([512, 512, 3, 3])
Unexpected key module.detector.features.28.bias in state_dict with size torch.Size([512])
Unexpected key module.detector.roi_fmap.0.weight in state_dict with size torch.Size([4096, 25088])
Unexpected key module.detector.roi_fmap.0.bias in state_dict with size torch.Size([4096])
Unexpected key module.detector.roi_fmap.3.weight in state_dict with size torch.Size([4096, 4096])
Unexpected key module.detector.roi_fmap.3.bias in state_dict with size torch.Size([4096])
Unexpected key module.detector.score_fc.weight in state_dict with size torch.Size([151, 4096])
Unexpected key module.detector.score_fc.bias in state_dict with size torch.Size([151])
Unexpected key module.detector.bbox_fc.weight in state_dict with size torch.Size([604, 4096])
Unexpected key module.detector.bbox_fc.bias in state_dict with size torch.Size([604])
Unexpected key module.detector.rpn_head.anchors in state_dict with size torch.Size([37, 37, 20, 4])
Unexpected key module.detector.rpn_head.conv.0.weight in state_dict with size torch.Size([512, 512, 3, 3])
Unexpected key module.detector.rpn_head.conv.0.bias in state_dict with size torch.Size([512])
Unexpected key module.detector.rpn_head.conv.2.weight in state_dict with size torch.Size([120, 512, 1, 1])
Unexpected key module.detector.rpn_head.conv.2.bias in state_dict with size torch.Size([120])
Unexpected key module.context.obj_embed.weight in state_dict with size torch.Size([151, 200])
Unexpected key module.context.obj_embed2.weight in state_dict with size torch.Size([151, 200])
Unexpected key module.context.rel_embed.weight in state_dict with size torch.Size([35390, 200])
Unexpected key module.context.pos_embed.0.weight in state_dict with size torch.Size([4])
Unexpected key module.context.pos_embed.0.bias in state_dict with size torch.Size([4])
Unexpected key module.context.pos_embed.0.running_mean in state_dict with size torch.Size([4])
Unexpected key module.context.pos_embed.0.running_var in state_dict with size torch.Size([4])
Unexpected key module.context.pos_embed.1.weight in state_dict with size torch.Size([128, 4])
Unexpected key module.context.pos_embed.1.bias in state_dict with size torch.Size([128])
Unexpected key module.context.obj_ctx_rnn.weight in state_dict with size torch.Size([17784832])
Unexpected key module.context.obj_ctx_rnn.bias in state_dict with size torch.Size([5120])
Unexpected key module.context.decoder_rnn.obj_embed.weight in state_dict with size torch.Size([152, 100])
Unexpected key module.context.decoder_rnn.input_linearity.weight in state_dict with size torch.Size([3072, 612])
Unexpected key module.context.decoder_rnn.input_linearity.bias in state_dict with size torch.Size([3072])
Unexpected key module.context.decoder_rnn.state_linearity.weight in state_dict with size torch.Size([2560, 512])
Unexpected key module.context.decoder_rnn.state_linearity.bias in state_dict with size torch.Size([2560])
Unexpected key module.context.decoder_rnn.out.weight in state_dict with size torch.Size([151, 512])
Unexpected key module.context.decoder_rnn.out.bias in state_dict with size torch.Size([151])
Unexpected key module.context.edge_ctx_rnn.weight in state_dict with size torch.Size([12148736])
Unexpected key module.context.edge_ctx_rnn.bias in state_dict with size torch.Size([10240])
Unexpected key module.union_boxes.conv.0.weight in state_dict with size torch.Size([256, 2, 7, 7])
Unexpected key module.union_boxes.conv.0.bias in state_dict with size torch.Size([256])
Unexpected key module.union_boxes.conv.2.weight in state_dict with size torch.Size([256])
Unexpected key module.union_boxes.conv.2.bias in state_dict with size torch.Size([256])
Unexpected key module.union_boxes.conv.2.running_mean in state_dict with size torch.Size([256])
Unexpected key module.union_boxes.conv.2.running_var in state_dict with size torch.Size([256])
Unexpected key module.union_boxes.conv.4.weight in state_dict with size torch.Size([512, 256, 3, 3])
Unexpected key module.union_boxes.conv.4.bias in state_dict with size torch.Size([512])
Unexpected key module.union_boxes.conv.6.weight in state_dict with size torch.Size([512])
Unexpected key module.union_boxes.conv.6.bias in state_dict with size torch.Size([512])
Unexpected key module.union_boxes.conv.6.running_mean in state_dict with size torch.Size([512])
Unexpected key module.union_boxes.conv.6.running_var in state_dict with size torch.Size([512])
Unexpected key module.roi_fmap.1.0.weight in state_dict with size torch.Size([4096, 25088])
Unexpected key module.roi_fmap.1.0.bias in state_dict with size torch.Size([4096])
Unexpected key module.roi_fmap.1.3.weight in state_dict with size torch.Size([4096, 4096])
Unexpected key module.roi_fmap.1.3.bias in state_dict with size torch.Size([4096])
Unexpected key module.roi_fmap_obj.0.weight in state_dict with size torch.Size([4096, 25088])
Unexpected key module.roi_fmap_obj.0.bias in state_dict with size torch.Size([4096])
Unexpected key module.roi_fmap_obj.3.weight in state_dict with size torch.Size([4096, 4096])
Unexpected key module.roi_fmap_obj.3.bias in state_dict with size torch.Size([4096])
Unexpected key module.post_lstm.weight in state_dict with size torch.Size([8192, 512])
Unexpected key module.post_lstm.bias in state_dict with size torch.Size([8192])
Unexpected key module.rel_compress.weight in state_dict with size torch.Size([51, 4096])
Unexpected key module.rel_compress.bias in state_dict with size torch.Size([51])
Unexpected key module.freq_bias.obj_baseline.weight in state_dict with size torch.Size([22801, 51])
We couldn't find detector.features.10.bias,context.pos_embed.0.weight,detector.features.19.bias,context.decoder_rnn.out.ight,detector.score_fc.bias,detector.rpn_head.conv.2.weight,context.obj_embed2.weight,context.decoder_rnn.state_linearitweight,detector.features.10.weight,detector.features.24.weight,detector.features.26.bias,roi_fmap.1.0.weight,detector.rphead.conv.2.bias,detector.features.14.bias,rel_compress.weight,union_boxes.conv.0.weight,roi_fmap_obj.0.weight,context.doder_rnn.state_linearity.bias,freq_bias.obj_baseline.weight,post_lstm.weight,union_boxes.conv.4.bias,detector.bbox_fc.weht,detector.features.21.bias,detector.features.17.weight,detector.features.26.weight,context.decoder_rnn.input_linearityias,roi_fmap_obj.3.bias,detector.score_fc.weight,detector.features.0.bias,context.pos_embed.0.bias,union_boxes.conv.6.ruing_mean,context.decoder_rnn.obj_embed.weight,detector.features.12.bias,context.obj_embed.weight,roi_fmap.1.3.weight,roimap_obj.0.bias,context.pos_embed.1.bias,detector.features.17.bias,union_boxes.conv.6.running_var,union_boxes.conv.6.biasontext.pos_embed.0.running_mean,detector.features.19.weight,detector.rpn_head.anchors,detector.features.21.weight,union_xes.conv.6.weight,detector.features.7.bias,detector.features.14.weight,roi_fmap.1.0.bias,roi_fmap_obj.3.weight,detector.atures.24.bias,detector.features.28.weight,context.decoder_rnn.input_linearity.weight,context.edge_ctx_rnn.weight,detect.features.7.weight,detector.roi_fmap.0.weight,detector.features.2.weight,detector.roi_fmap.3.bias,union_boxes.conv.2.runng_mean,context.decoder_rnn.out.bias,detector.features.2.bias,detector.roi_fmap.3.weight,union_boxes.conv.2.running_var,i_fmap.1.3.bias,detector.features.28.bias,context.obj_ctx_rnn.weight,detector.roi_fmap.0.bias,context.pos_embed.0.runninvar,detector.features.12.weight,union_boxes.conv.4.weight,post_lstm.bias,detector.rpn_head.conv.0.weight,detector.rpn_he.conv.0.bias,detector.features.5.bias,context.pos_embed.1.weight,context.edge_ctx_rnn.bias,union_boxes.conv.0.bias,detecr.features.5.weight,union_boxes.conv.2.weight,rel_compress.bias,union_boxes.conv.2.bias,detector.bbox_fc.bias,detector.ftures.0.weight,context.obj_ctx_rnn.bias
100%|██████████| 26446/26446 [2:29:23<00:00, 2.95it/s]
======================sgdet============================
R@100: 0.000000
R@50: 0.000000
R@20: 0.000000

cudaCheckError() failed : no kernel image is available...

Was attempting to run: bash scripts/pretrain_detector.sh and rain into the following:
cudaCheckError() failed : no kernel image is available for execution on the device

I'm running on NVIDIA TITAN X on cuda9.0, and I've tried the following:

• in root/Makefile:
  • changing export PATH := /usr/local/cuda-9.0/bin:$(PATH)
• in /lib/lstm/highway_lstm_cuda/make.sh b/lib/lstm/highway_lstm_cuda/make.sh:
  • setting CUDA_MODELS=(61), which seems to correspond to the TITAN X

Any ideas about where else I can look to modify?

training issue

Hi I'm trying to train the model with a subset of the predicates thus i'm doing the next change in train_rels.py

intead of inserting train.ind_to_predicates to the rel model im inserting a subset in the next order

rel_list = sample(train.ind_to_predicates, round(0.75*len(train.ind_to_predicates)))

when training i get the next error

Traceback (most recent call last):
File "./models/train_rels.py", line 197, in
rez = train_epoch(epoch)
File "./models/train_rels.py", line 108, in train_epoch
tr.append(train_batch(batch, verbose=b % (conf.print_interval*10) == 0)) #b == 0))
File "./models/train_rels.py", line 140, in train_batch
result = detector[b]
File "/home/sgalk87/neural-motifs/lib/rel_model.py", line 591, in getitem
return self(*batch[0])
File "/home/sgalk87/neural-motifs/venv/lib/python3.5/site-packages/torch/nn/modules/module.py", line 325, in call
result = self.forward(*input, **kwargs)
File "/home/sgalk87/neural-motifs/lib/rel_model.py", line 568, in forward
), 1))
RuntimeError: The size of tensor a (38) must match the size of tensor b (51) at non-singleton dimension 1

do you have any idea how to fix this problem or is tehre a different approach i should take in order to train with a subset of the predicates.

A question about motifs.

Thanks for sharing code. But my concept of motifs is not very clear. Do you mind telling me which code or pseudocode is generating motifs ?

please can you detail all the dependencies?

I tried building and needed cython for example, i installed cython and still had this error

sgalk87@rishon1:~/neural-motifs$ make
cd lib/draw_rectangles; python setup.py build_ext --inplace
Traceback (most recent call last):
File "setup.py", line 2, in
from Cython.Build import cythonize
ImportError: No module named Cython.Build
Makefile:6: recipe for target 'draw_rectangles' failed
make: *** [draw_rectangles] Error 1

thanks

Is rel_model_stanford.py an implementation of standford paper "Scene Graph Generation by Iterative Message Passing"?

Error: cudaCheckError() failed : no kernel image is available for execution on the device

Hi,
When running the script eval_rels.py to test the baseline model, I get the error cudaCheckError() failed : no kernel image is available for execution on the device, in eval_rels.py/val_batch(...) line det_res = detector[b].

I'm running on a P100 GPU, with python 3.5, CUDA 9.2, PyTorch 0.3.1. I set in lib/lstm/highway_lstm_cuda/make.sh the CUDA_MODELS to 60. In this way, the highway LSTM is compiled with /usr/local/cuda/bin/nvcc -c -o highway_lstm_kernel.cu.o highway_lstm_kernel.cu.o --compiler-options -fPIC -gencode arch=compute_60,code=sm_60. The compilation of the CPython lib seems to work fine.

Do you have an idea what could be the problem? Thanks in advance!

how to re-implement the result in no graph constraints ???

I got the result in graph constraints with given code and parameters, but where is the no constraints part?
How to implement no graph constraints ?

Check relation index (rel_inds in rel_model.py) and make sure every proposal only have one relation to another proposal area ?

faster_rcnn_500k.h5 what is this network?

Hi Rowan,

Many thanks for releasing your work! As I am trying to pretrain the detector, I found the training need to restore a checkpoint called faster_rcnn_500k.h5. If I understand it correctly, this net is a VGG network but weights are trained on some other dataset?

Could you let me know on what dataset is this network trained, for how many epochs/hours and what is its performance on that dataset?

Many thanks!
Chu.

How to visualize the results?

After training and evaluation, how can I visualize the image with boxes and corresponding scene graph?
Also, in the row 320 of lib/object_detector.py, does it mean we need to modify the code if we train models like sgdet? Because when the mode is sgdet, the statement of "if" branch will never run.

Thanks!

how many epoch have you trained the motif net in?

Adaption to small image.

Hi, Rowan,
I'd like to resize the image to 224 × 224 instead of 592 × 592. I think I should change IM_SCALE and ANCHOR_SIZE. Are those all I have to change?

Thanks!

AttributeError: Can't get attribute '_rebuild_tensor_v2' on <module 'torch._utils'

  File "models/rels.py", line 42, in <module>
    limit_vision=conf.limit_vision
  File "/home/satreja0/neural-motifs/lib/rel_model.py", line 354, in __init__
    pass_in_obj_feats_to_edge=pass_in_obj_feats_to_edge)
  File "/home/satreja0/neural-motifs/lib/rel_model.py", line 95, in __init__
    embed_vecs = obj_edge_vectors(self.classes, wv_dim=self.embed_dim)
  File "/home/satreja0/neural-motifs/lib/word_vectors.py", line 18, in obj_edge_vectors
    wv_dict, wv_arr, wv_size = load_word_vectors(wv_dir, wv_type, wv_dim)
  File "/home/satreja0/neural-motifs/lib/word_vectors.py", line 55, in load_word_vectors
    return torch.load(fname_pt)
  File "/home/satreja0/miniconda2/envs/pytorch3/lib/python3.6/site-packages/torch/serialization.py", line 261, in load
    return _load(f, map_location, pickle_module)
  File "/home/satreja0/miniconda2/envs/pytorch3/lib/python3.6/site-packages/torch/serialization.py", line 409, in _load
    result = unpickler.load()
AttributeError: Can't get attribute '_rebuild_tensor_v2' on <module 'torch._utils' from '/home/satreja0/miniconda2/envs/pytorch3/lib/python3.6/site-packages/torch/_utils.py'>

I am using pytorch 3.0 but _rebuild_tensor_v2 comes in verison 4.0
This error comes when a model is tried to load from an older version

train_detector.py loading dataset too slow

Everytime I started ./scripts/pretrain_detector.sh, I need to wait VG.splits a longtime to get dataset loaded.

are there any ways to accelerate data loading?

pytorch 0.4.0 causes maximum recursion error in `block_orthogonal` instantiation

neural-motifs/lib/lstm/highway_lstm_cuda/alternating_highway_lstm.py

Lines 31 to 33 in 15c8754

	if isinstance(tensor, Variable):
	block_orthogonal(tensor.data, split_sizes, gain)
	return tensor

likely because Tensor and Variable are now the same class: https://pytorch.org/2018/04/22/0_4_0-migration-guide.html

training rel detector using multi gpus

Hi,
I have successfully trained the detector using multiple gpus (8). But I have the following issue when training rel detector using more than one GPUs (have tried on 1080 ti, p100 and K40)

Traceback (most recent call last):
  File "/home/wtliao/work_space/neural-motifs-master-backup/models/train_rels.py", line 229, in <module>
    rez = train_epoch(epoch)
  File "/home/wtliao/work_space/neural-motifs-master-backup/models/train_rels.py", line 135, in train_epoch
    tr.append(train_batch(batch, verbose=b % (conf.print_interval*10) == 0)) #b == 0))
  File "/home/wtliao/work_space/neural-motifs-master-backup/models/train_rels.py", line 179, in train_batch
    loss.backward()
  File "/home/wtliao/anaconda2/envs/mofit/lib/python3.6/site-packages/torch/autograd/variable.py", line 167, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, retain_variables)
  File "/home/wtliao/anaconda2/envs/mofit/lib/python3.6/site-packages/torch/autograd/__init__.py", line 99, in backward
    variables, grad_variables, retain_graph)
RuntimeError: narrow is not implemented for type UndefinedType

The code works well for single gpu. I have no idea about that at all and I cant find a sollution by google. Do you have any idea about that? Thanks

Did anyone successfully download the pretrained checkpoints?

I have tried several times to download the pretrained model, but it's a broken file. Did anyone successfully download the pretrained files? Or the uploaded file is broken?

training fails

running the next command 👍
python3 ./models/train_rels.py -m sgcls -model motifnet -order leftright -nl_obj 2 -nl_edge 4 -b 6 -clip 5
-p 100 -hidden_dim 512 -pooling_dim 4096 -lr 1e-3 -ngpu 1 -ckpt checkpoints/vgdet/vg-24.tar
-save_dir checkpoints/motifnet2 -nepoch 50 -use_bias
and getting the next error, did you encounter this error?
Training starts now!
Traceback (most recent call last):
File "./models/train_rels.py", line 199, in
rez = train_epoch(epoch)
File "./models/train_rels.py", line 110, in train_epoch
tr.append(train_batch(batch, verbose=b % (conf.print_interval*10) == 0)) #b == 0))
File "./models/train_rels.py", line 142, in train_batch
result = detector[b]
File "/home/sgalk87/neural-motifs/lib/rel_model.py", line 593, in getitem
return self(*batch[0])
File "/home/sgalk87/neural-motifs/venv/lib/python3.5/site-packages/torch/nn/modules/module.py", line 325, in call
result = self.forward(*input, **kwargs)
File "/home/sgalk87/neural-motifs/lib/rel_model.py", line 538, in forward
boxes.data, result.boxes_all)
File "/home/sgalk87/neural-motifs/venv/lib/python3.5/site-packages/torch/nn/modules/module.py", line 325, in call
result = self.forward(*input, **kwargs)
File "/home/sgalk87/neural-motifs/lib/rel_model.py", line 333, in forward
box_priors=box_priors,
File "/home/sgalk87/neural-motifs/lib/rel_model.py", line 221, in edge_ctx
obj_embed2 = self.obj_embed2(obj_preds)
File "/home/sgalk87/neural-motifs/lib/rel_model.py", line 221, in edge_ctx
obj_embed2 = self.obj_embed2(obj_preds)
File "/usr/lib/python3.5/bdb.py", line 48, in trace_dispatch
return self.dispatch_line(frame)
File "/home/sgalk87/neural-motifs/venv/lib/python3.5/site-packages/pudb/debugger.py", line 187, in dispatch_line
self.user_line(frame)
File "/home/sgalk87/neural-motifs/venv/lib/python3.5/site-packages/pudb/debugger.py", line 408, in user_line
self.interaction(frame)
File "/home/sgalk87/neural-motifs/venv/lib/python3.5/site-packages/pudb/debugger.py", line 376, in interaction
show_exc_dialog=show_exc_dialog)
File "/home/sgalk87/neural-motifs/venv/lib/python3.5/site-packages/pudb/debugger.py", line 2118, in call_with_ui
return f(*args, **kwargs)
File "/home/sgalk87/neural-motifs/venv/lib/python3.5/site-packages/pudb/debugger.py", line 2362, in interaction
self.event_loop()
File "/home/sgalk87/neural-motifs/venv/lib/python3.5/site-packages/pudb/debugger.py", line 2322, in event_loop
keys = self.screen.get_input()
File "/home/sgalk87/neural-motifs/venv/lib/python3.5/site-packages/urwid/raw_display.py", line 336, in get_input
keys, raw = self.parse_input(None, None, self.get_available_raw_input())
File "/home/sgalk87/neural-motifs/venv/lib/python3.5/site-packages/urwid/raw_display.py", line 436, in get_available_raw_input
codes = self._get_gpm_codes() + self._get_keyboard_codes()
File "/home/sgalk87/neural-motifs/venv/lib/python3.5/site-packages/urwid/raw_display.py", line 510, in _get_keyboard_codes
code = self._getch_nodelay()
File "/home/sgalk87/neural-motifs/venv/lib/python3.5/site-packages/urwid/raw_display.py", line 644, in _getch_nodelay
return self._getch(0)
File "/home/sgalk87/neural-motifs/venv/lib/python3.5/site-packages/urwid/raw_display.py", line 554, in _getch
return ord(os.read(self._term_input_file.fileno(), 1))
TypeError: ord() expected a character, but string of length 0 found

Why feeding roi feature to vgg again?

In rel_model.py, obj_feature_map(), you feed the pooled features(feature_pool) from RoIAlign layer to vgg Net again to generate roi features. Why not returning feature_pool directly?

related codes :

self.roi_fmap_obj = load_vgg(pretrained=False).classifier

def obj_feature_map(self, features, rois):
    """
    Gets the ROI features
    :param features: [batch_size, dim, IM_SIZE/4, IM_SIZE/4] (features at level p2)
    :param rois: [num_rois, 5] array of [img_num, x0, y0, x1, y1].
    :return: [num_rois, #dim] array
    """
    feature_pool = \
RoIAlignFunction(self.pooling_size, self.pooling_size, spatial_scale=1 / 16)(features, rois)
    return self.roi_fmap_obj(feature_pool.view(rois.size(0), -1))

Questions about RoIAlign

When I read your code, I am confused by the calculation of RoI align. In RoIAlignFunction, the normalized rois are computed as follows:
rois_normalized[:, 1] /= width and width=(data_width - 1) / saptial_scale which should be the width of the input image, so the normalized rois are in [0, 1], is this correct? From the paper of msak-rcnn, I think rois_normalized[:, 1] *= spatial_scale.