yueliao/ppdm

Code for "PPDM: Parallel Point Detection and Matching for Real-time Human-Object Interaction Detection".

ppdm's Issues

no code here?

Can you help me to prepare custom dataset?

I've downloaded HOI dataset.

Depth image :

Label:

it's look like this..

Is it correct dataset or else can you share me any useful resources.

That would help me to prepare my custom dataset..

Thanks in advance..

DDP和AMP对最终精度有影响吗

请问使用了ddp和amp在相同参数情况下训练结果怎么样呢，能保持原有精度吗

Question about HICO mAP

The mAP result get by src/lib/eval/hico_eval.py is the same with the default mode result of HICO evaluation?

test_time for DLA-34

I test the DLAbackbone but get the test time about 480ms per image(But the paper said 27ms)

Category List of HICO-DET dataset annotation

Hi, thx for your great work!
I note that there are 'category_id' in your HICO-DET annotation file 'trainval_hico.json'. But i have no idea about what is exactly those numbers mean. Because It is not as same as the category id in original datasets annotations. Is it a new category mapping? And is there a list about the category_id and its corresponding category for obj/verb?

Weird evaluation results

Hi Yue,

Thanks for you nice work! I tried to run the evaluation codes on your shared model (model_last.pth of dla34 in https://drive.google.com/drive/folders/1K0H05nSUOCq939tmvBRJjskdPSLy1U-U, I renamed it model_last.dla34.pth), however i failed to reproduce the numbers. Below are the command i run and stdout/stderr i got (file path masked). Do you have some thoughts on what might be wrong? There seems to be some parameters missing in the .pth file, is it desired?

$ python test_hoi.py hoidet --exp_id hoidet_hico_dla --load_model ../models/model_last.dla34.pth  --gpus 0 --dataset hico --image_dir images/test2015 --test_with_eval
Fix size testing.
training chunk_sizes: [32]
The output will be saved to  [MYFOLDER]/PPDM/src/lib/../../exp/hoidet/hoidet_hico_dla
heads {'sub_offset': 2, 'wh': 2, 'hm_rel': 117, 'hm': 80, 'obj_offset': 2, 'hm_human': 1, 'reg': 2}
Namespace(K=100, arch='dla_34', batch_size=32, cat_spec_wh=False, chunk_sizes=[32], data_dir='[MYFOLDER]/PPDM/src/lib/../../data', dataset='hico', debug=0, debug_dir='[MYFOLDER]/PPDM/src/lib/../../exp/hoidet/hoidet_hico_dla/de
bug', debugger_theme='white', demo='', dense_wh=False, down_ratio=4, exp_dir='[MYFOLDER]/PPDM/src/lib/../../exp/hoidet', exp_id='hoidet_hico_dla', fix_res=True, flip=0.5, flip_test=False, gpus=[0], gpus_str='0', head_conv=256, heads={'sub_offset': 2, 'wh': 2, 'hm_rel': 1
17, 'hm': 80, 'obj_offset': 2, 'hm_human': 1, 'reg': 2}, hide_data_time=False, hm_weight=1, image_dir='images/test2015', input_h=512, input_res=512, input_w=512, keep_res=False, load_model='../models/model_last.dla34.pth', lr=0.000125, lr_step=[90, 120], master_batch_size=32, mean=array([[[0.40789655, 0.44719303, 0
.47026116]]], dtype=float32), metric='loss', mse_loss=False, nms=False, no_color_aug=False, norm_wh=False, not_cuda_benchmark=False, not_prefetch_test=False, not_rand_crop=False, not_reg_offset=False, num_classes=80, num_classes_verb=117, num_epochs=140, num_iters=-1, num_stacks=1, num_workers=4, off_weight=1, outp
ut_h=128, output_res=128, output_w=128, pad=31, print_iter=0, reg_loss='l1', reg_offset=True, resume=False, root_dir='[MYFOLDER]/PPDM/src/lib/../..', root_path='../Dataset', rotate=0, save_all=False, save_dir='[MYFOLDER]/PPDM/
src/lib/../../exp/hoidet/hoidet_hico_dla', save_predictions=False, save_video='', scale=0.4, seed=317, shift=0.1, std=array([[[0.2886383 , 0.27408165, 0.27809834]]], dtype=float32), task='hoidet', test=False, test_dir='', test_scales=[1.0], test_video=False, test_with_eval=True, trainval=False, use_cos=0, use_verb_
sub=0, val_intervals=100000, vis_thresh=0.3, wh_weight=0.1)
Creating model...
loaded ../models/model_last.dla34.pth, epoch 140
No param hm_human.0.weight.
No param hm_human.0.bias.
No param hm_human.2.weight.
No param hm_human.2.bias.
----epoch :0 -----
hoidet_hico_dla |################################| [9545/9546]|Tot: 0:06:08 |ETA: 0:00:01 |load 0.000s (0.000s) |pre 0.000s (0.000s) |tot 0.031s (0.032s) |merge 0.004s (0.005s) |net 0.019s (0.020s) |dec 0.005s (0.005s) |post 0.002s (0.002s) 
--------------------
mAP: 0.039696969697 mAP rare: 0.0204216073781  mAP nonrare: 0.0454545454545  max recall: 0.0216666666667
--------------------
best model id: 0, best map: 0.039696969697
Traceback (most recent call last):
  File "test_hoi.py", line 112, in <module>
    prefetch_test(opt)
  File "test_hoi.py", line 108, in prefetch_test
    save_json(best_output, model_path, 'best_predictions.json')
UnboundLocalError: local variable 'model_path' referenced before assignment

Inference time on the HICO-Det test set

Excuse me,

1. how is the inference time calculated on the iCAN / TIN, including which parts of the inference time (if divided into pre-processing, HOI inference , post-processing) ?
2. and, for iCAN, do you use the test code of iCAN to calculate the inference time? thanks.

Question about multi-person case

Hi,
the size of object heatmap is [num_class, output_h, output_w] in your code.
Does it mean that the heatmap can not classify different person due to catorgry_id=1?
If not, how to address multi-person case?
e.g. image [COCO_train2014_000000210751.jpg]:

HOI-A dataset

Thank you for sharing your great work! I have been working on human object interaction detection and want to evaluate our algorithm with HOI-A dataset. When are you going to release it? I hope you will be able to provide the information.

Questions about code and dataset

1. PPDM achieves high mAP in HICO and HOI-A dataset which are both big dataset, what about the performance on small dataset like v-coco?
2. What is the effect of the affine_transform when producing labeled heatmap?
   Thanks

Large gap between released mAP and my exp's.

Thanks for your excellent work. I am trying to achieve the released mAP for DLA34 with hoidet_hico_dla.sh, however, after 140 epoch training I only get the result as below:

I run in python 3.6.12 and PyTorch 0.4.1. Would you mind giving me some tips about the reasons for the low performance?

Fix size testing.
training chunk_sizes: [32]
The output will be saved to ../../exp/hoidet/hoidet_hico_dla
heads {'hm': 80, 'wh': 2, 'hm_rel': 117, 'sub_offset': 2, 'obj_offset': 2, 'reg': 2}
Namespace(K=100, arch='dla_34', batch_size=32, cat_spec_wh=False, chunk_sizes=[32], data_dir='../../data', dataset='hico', debug=0, debug_dir='/exp/hoidet/hoidet_hico_dla/debug', debugger_theme='white', demo='', dense_wh=False, down_ratio=4, exp_dir='../../exp/hoid
et', exp_id='hoidet_hico_dla', fix_res=True, flip=0.5, flip_test=False, gpus=[0], gpus_str='0', head_conv=256, heads={'hm': 80, 'wh': 2, 'hm_rel': 117, 'sub_offset': 2, 'obj_offset': 2, 're
g': 2}, hide_data_time=False, hm_weight=1, image_dir='hico_20160224_det/images/test2015', input_h=512, input_res=512, input_w=512, keep_res=False, load_model=''
, lr=0.000125, lr_step=[90, 120], master_batch_size=32, mean=array([[[0.40789655, 0.44719303, 0.47026116]]], dtype=float32), metric='loss', mse_loss=False, nms=False, no_color_aug=False, no
rm_wh=False, not_cuda_benchmark=False, not_prefetch_test=False, not_rand_crop=False, not_reg_offset=False, num_classes=80, num_classes_verb=117, num_epochs=140, num_iters=-1, num_stacks=1,
num_workers=4, off_weight=1, output_h=128, output_res=128, output_w=128, pad=31, print_iter=0, reg_loss='l1', reg_offset=True, resume=False, root_dir='lib/..
/..', root_path='../Dataset', rotate=0, save_all=False, save_dir='../../exp/hoidet/hoidet_hico_dla', save_predictions=False, save_video='', scale=0.4, se
ed=317, shift=0.1, std=array([[[0.2886383 , 0.27408165, 0.27809834]]], dtype=float32), task='hoidet', test=False, test_dir='', test_scales=[1.0], test_video=False, test_with_eval=True, trai
nval=False, use_cos=0, use_verb_sub=0, val_intervals=100000, vis_thresh=0.3, wh_weight=0.1)
Creating model...
loaded ../../exp/hoidet/hoidet_hico_dla/model_140.pth, epoch 140
----epoch :140 -----

mAP: 0.19771157974547146 mAP rare: 0.11901581021109064 mAP nonrare: 0.2212181083076891 max recall: 0.49568942049889964

what are the differences between model_last and model_best?

hello, I have run the code on model_best of hourglass, but the mAP obtained by this model_best is 0.0，and model_last can be used to get a mAP more than 15. And also , the prediction results of the models obtained in the training process are all empty, and the sizes of these models are all 2.18G. Thanks very much for your answer and your excellent idea.

The error of testing the trained model:

File "test_hoi.py", line 114, in
prefetch_test(opt)
File "test_hoi.py", line 99, in prefetch_test
map = hoi_eval.evalution(output_hoi)
File "/home/ll579/PPDM_/src_/lib/eval/hico_eval.py", line 66, in evalution
bbox_pairs, bbox_ov = self.compute_iou_mat(gt_bbox, pred_bbox)
ValueError: not enough values to unpack (expected 2, got 0)

Could you please provide model perfomance on HOIA dataset?

including each verb's AP score?

Question about the result on the HOI-A dataset

Could you please releaze the pretrained model on the HOI-A dataset? Thanks a lot.

How to build my own dataset?

I noticed that you used a self built dataset. I want to build a dataset in this format. What should I do or what tools should I use to mark it? Please give me some advice.

vcoco annotation

hello, could you please release the vcoco dataset, i want to know the format of vcoco in this repo, thanks!

How to train my own dataset？

I want to perform transfer learning to train the model with the pretrained weights on my own small image dataset. My small dataset category is completely different from that of Hoi. Let me know how to go about it.thank you.

Custom Dataset - Meaning of "subject_id" and "hoi_category_id"

Hello,
I would like to train your model with Custom Data. While creating the annotation I came across your annotation files. Here I found the following:
In the field "annotations" the corresponding bounding boxes and a category_id are noted.
The "bbox" probably corresponds to each annotated subject or object.
As I understand it, the category_id for each annotation in the "annotations" field is the word list you posted here.
Can you confirm my assumption here so far?

Furthermore I have a question about the field "hoi_annotation".
What is the meaning of subject_id, object_id and hoi_category_id in each HOI annotation?

I have attached a sample annotation of the trainval_hico.json file. Here I have commented on what is unclear. How these values come about.:

    {
        "file_name": "HICO_train2015_00000009.jpg",
        "img_id": 9,
        "annotations": [
            {  // Index 0
                "bbox": [
                    190,
                    101,
                    290,
                    305
                ],
                "category_id": 1 // 1 = 'person' - The ID comes from the word list
            },
            { // Index 1
                "bbox": [
                    210,
                    99,
                    431,
                    335
                ],
                "category_id": 8 // 8 = 'truck'  - The ID comes from the word list
            },
            { // Index 2
                "bbox": [
                    339,
                    93,
                    597,
                    406
                ],
                "category_id": 1 // 1 = 'person'  - The ID comes from the word list
            }
        ],
        "hoi_annotation": [
            {
                "subject_id": 0, // Reference to the element of "annotations" with index 0
                "object_id": 1,  // Reference to the element of "annotations" with index 1
                "category_id": 53, // Verb category - According to the word list, the verb with ID = 53 = "load"
                "hoi_category_id": 571 // HOI triplet category
            },
            {
                "subject_id": 2, // Reference to the element of "annotations" with index 2
                "object_id": 1, // Reference to the element of "annotations" with index 1
                "category_id": 53, // Verb category - According to the word list, the verb with ID = 53 = "load"
                "hoi_category_id": 571 // HOI triplet category
            }
        ]
    }

The corresponding visualized annotation and image look like this here:

undefined symbol: __cudaRegisterFatBinaryEnd Issue

Hello
I have cuda 7.5 but I get the following error"undefined symbol: __cudaRegisterFatBinaryEnd Issue" when run python main.py Hoidet --batch_size 112 --master_batch 7 --lr 4.5e-4 --gpus 0,1,2,3,4,5,6,7 --num_workers 16 --load_model ../models/ctdet_coco_dla_2x.pth --image_dir images/train2015 --dataset hico --exp_id hoidet_hico_dla

HOI-A dataset

Where I can get the images and annotations of HOI-A dataset?
Thanks.

How to eval trained model in branch pt1?

Hi Yue,

I found that the former scripts from master cannot be used directly in branch pt1. I modified the test_hoi.py to test.py, but it still does not work.

socket.gaierror: [Errno -3] Temporary failure in name resolution

When I run the main.py, I get the error.
How can I fix it? My server cannot connect to the internet.

About the generation of "corre_.npy"

Hi YueLiao,

I'm so sorry for bothering you again. I wondered may I have the "corre_vcoco.npy" file if it's at your convenience? I found that just like the hico dataset, the program also relies on the binary mask to do the validation or testing. Annotation alone cannot bring me there. Thank you in advance!

Bozhou

While building the model it's showing sum error?

I'm using it in google colab..

Thanks in Advance.

CODE:

%cd src/lib/models/networks/DCNv2
!./make.sh

ERROR:

/content/PPDM/src/lib/models/networks/DCNv2
Traceback (most recent call last):
File "build.py", line 3, in
from torch.utils.ffi import create_extension
File "/usr/local/lib/python3.6/dist-packages/torch/utils/ffi/init.py", line 1, in
raise ImportError("torch.utils.ffi is deprecated. Please use cpp extensions instead.")
ImportError: torch.utils.ffi is deprecated. Please use cpp extensions instead.
Traceback (most recent call last):
File "build_double.py", line 3, in
from torch.utils.ffi import create_extension
File "/usr/local/lib/python3.6/dist-packages/torch/utils/ffi/init.py", line 1, in
raise ImportError("torch.utils.ffi is deprecated. Please use cpp extensions instead.")
ImportError: torch.utils.ffi is deprecated. Please use cpp extensions instead.

Question about the verb_name_dict and obj_name_dict for vcoco dataset

Thanks for you great work. I tried to visualize vcoco on your provided annotations by using PIC_HOIW tools. However, I cannot access the corresponding verb_name_dict and obj_name_dict. Could you please share with me them? Thanks a lot.

How many GPUs at least to use when training?

I got OOM when using 2 GPUs(--gpus 2,3), I want to know how many GPUs at least to use when training?
Looking forward to your reply, thanks.

main architecture of network

Hello
thanks for your code
Where is the ground truth point heatmap file?
I dont understand your code. In which file is there Point Detection Branch file and Point matching branch ؟
In which code is the main architecture of your network and ?

HOI-A/HOI-W annotation json files

Hi Yueliao,

I'm running experiments on HOI-A/HOI-W dataset. In the src dataset/hoia.py there are two input json files: train_hoia.json, and test_hoia.json. I can't find them in the current repo.

From the PIC website in the BaiduDrive, is the 'trainval.json' the file for 'train_hoia.json' in hoia.py?

And, where is test_hoia.json? Need it to evaluate the test results. I also checked your PIC-HOIW github, but still cannot find it.

Thank you for your help!

dataset annotations

Professor，can you give a brief introduction of annotations like(test_hico.json,trainval_hico.json,corre_hico.npy)?In test_hico.json,there are no introductions about subject_id、category_id and object_id.This confuses me.No map between .id and name was introduced

runtime error

hello when i run the code, i encounter this problem, cuda9.0 pytorch0.4.1

RuntimeError: cuda runtime error (11) : invalid argument at /opt/conda/conda-bld/pytorch_1535491974311/work/aten/src/THC/THCGeneral.cpp:663

could you give me some suggestions? thanks

Training error

Hello, I built PPDM as it is written in Installation.
But when I launch the training I got this:
What might be the problem ?

I'm using AWS g4dn.xlarge machine.

HOI-A dataset

Thanks for sharing! but, could you share your HOI-A dataset mentioned in you paper?
here is my email: [email protected]

Import Error!

Import Error:

Thanks in advance

Code:

%cd /content/PPDM/src
!python /content/PPDM/src/test_hoi.py --exp_id hoidet_hico_dla --gpus 0 --dataset hico --image_dir images/test2015 --test_with_eval

ERROR:

/content/PPDM/src
Traceback (most recent call last):
File "/content/PPDM/src/test_hoi.py", line 16, in
from detectors.detector_factory import detector_factory
File "/content/PPDM/src/lib/detectors/detector_factory.py", line 5, in
from .hoidet import HoidetDetector
File "/content/PPDM/src/lib/detectors/hoidet.py", line 12, in
from .base_detector import BaseDetector
File "/content/PPDM/src/lib/detectors/base_detector.py", line 11, in
from models.model import create_model, load_model
File "/content/PPDM/src/lib/models/model.py", line 8, in
from .networks.pose_dla_dcn import get_pose_net as get_dla_dcn
File "/content/PPDM/src/lib/models/networks/pose_dla_dcn.py", line 16, in
from .DCNv2.dcn_v2 import DCN
File "/content/PPDM/src/lib/models/networks/DCNv2/dcn_v2.py", line 11, in
from .dcn_v2_func import DCNv2Function
File "/content/PPDM/src/lib/models/networks/DCNv2/dcn_v2_func.py", line 9, in
from ._ext import dcn_v2 as _backend
File "/content/PPDM/src/lib/models/networks/DCNv2/_ext/dcn_v2/init.py", line 3, in
from ._dcn_v2 import lib as _lib, ffi as _ffi
ImportError: /content/PPDM/src/lib/models/networks/DCNv2/_ext/dcn_v2/_dcn_v2.so: undefined symbol: __cudaPopCallConfiguration

HOI-A dataset

Could you publish HOI-A dataset? I see on your paper and github, you already publish your result. But you haven't published your dataset yet.
If you publish dataset, please publish it on google drive/one drive because baidu required sigh up and it doesn't send message to VN.

How could i get the test result in HOI-A/W?

ERROR: cudaRegisterFatBinaryEnd

Hello!
I am a Masters student trying to run your code for my final project (it looks incredibly interesting)!
However, I am running into an issue when I try to train or test it using your instructions.
I am on CUDA 9.0, torch==0.4.0, torchvision==0.2.0. However I have tried a mix of the three to no avail.

Upon running, I get the error: ImportError: /home//PPDM/src/lib/models/networks/DCNv2/_ext/dcn_v2/_dcn_v2.so: undefined symbol: __cudaRegisterFatBinaryEnd

Please do let me know if you have any suggestions to remedy this. Thank you!

HOI-A/HOI-W annotation json files

Hi Yueliao,

I'm running experiments on HOI-A/HOI-W dataset. In the src dataset/hoia.py there are two input json files: train_hoia.json, and test_hoia.json. I can't find them in the current repo. Could you tell me that ?
Thank you for your help!

Test error

Hi, thanks for your work.
There is an error when testing a pre-trained model:

Traceback (most recent call last):
  File "test_hoi.py", line 112, in <module>
    prefetch_test(opt)
  File "test_hoi.py", line 98, in prefetch_test
    map = hoi_eval.evalution(output_hoi)
  File "PPDM/src/lib/eval/hico_eval.py", line 62, in evalution
    bbox_pairs, bbox_ov = self.compute_iou_mat(gt_bbox, pred_bbox)
ValueError: not enough values to unpack (expected 2, got 0)

And it's ok if I test without '--test_with_eval'. Do you have any advice?

test for hoia

Hello! Thx for your work! Could you publicly the test code of hoi-a dataset now? The challenge seems has already closed.

Visualization of Points Heatmaps

@YueLiao
Thanx for ur wonderful work.
Could u please offer me the scripts of "Visualization of Points Heatmaps", which visualize points heatmaps in original images (Figure 6 of the paper)?
I want to learn more from PPDM framework and cite ur paper.

Thanx again.
Waiting for reply : )

how to train on custmer dataset?

Thank you for sharing the code. I trained on my own dataset(two object, six verb), but my output value is like this. Is there something wrong with my Settings?

@YueLiao

Annotation for vCOCO dataset

Hi there,

Amazing work you've done! I have finished reproducing similar mAP via your repository and that's amazingly well. For the next step, I plan to try out your methodology on v-COCO dataset, however I note that you refer to iCAN project for the vcoco part. I'm curious that whether you prepared the annotation for the COCO dataset part?

I navigated to the iCAN project webpage, but found the annotations that they provided is not quite of "json" format and the syntax of which the vcoco_eval cannot even parse.

If you can help, I appreciate it a lot!

question about equation 5 in the paper

Hi, @YueLiao @ZhiyuanChen
I feel confused about equation 5 in the paper,

It seem to be 0 in the equation all the time. If misunderstand, can you tell me what the meaning of equation 5?

vcoco json file and visualizationprogram

hello，i wonder if you could upload the vcoco json file and visualization program file. I'd apprecicate it if you could.

Recurring problem，How to get the accuracy in the paper？

Hello, I used the code you provided to fail to achieve the accuracy in your paper (71.23@mAP) on the HOIA dataset. I am using the hourglass model. Can you provide checkpoints?

train error caused may be cuda

Hi, i try to train the code. And my conda env include:
pytorch0.4.1、cudn9.0.176、gcc6、python3.6.12. and nvcc -v is 9.0 too.
but i also get error "THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1535491974311/work/aten/src/THC/THCGeneral.cpp line=663 error=8 : invalid device function” and RuntimeError: cuda runtime error (8) : invalid device function at /opt/conda/conda-bld/pytorch_1535491974311/work/aten/src/THC/THCGeneral.cpp:663"
Besides，i try to set batchsize to 1 but not help. If you have some ideas, please tell me. Thank you!!Happy New year.

HOI-A/HOI-W annotation json files

Sorry, I also cannot find the annotation file of test set from the HOI-A/HOI-W dataset from official website and your provided link. Since I have no chance to access it, I cannot verify PPDM's performance on HOI-A test set. Could you please help me to access this annotation? Thanks a lot.

the relationship of subject_id and object_id

I can't understand the relationship of this subject_id and object_id in your test_hico.json file. Whether there is a word list, can be one-to-one correspondence.

test_hico.json
what is means in the subject_id , object_id . What do the numbers inside mean?
Thank you

how to get a visualized demo (image or video)

Thanx for ur great job! @YueLiao
I am very interested in this work.
I wonder if u can tell me how to get a visualized image demo (e.g., image from HICO, VCOCO, or own image) or video demo?
I would appreciate it if you can provide related scripts.

Best regards