youngxiao13/posefromshape

(BMVC 2019) PyTorch implementation of Paper "Pose from Shape: Deep Pose Estimation for Arbitrary 3D Objects"

👋 Hi, I’m @YoungXIAO13

posefromshape's Issues

The source of python-blender-2.77-py36_0.tar.bz2

Hi,

Could you please share where/how did you get the file python-blender-2.77-py36_0.tar.bz2?

I have tried to install bpy manually by building the blender with bpy target, but have faced some difficulties ...

TransLightning explanation

Hi,

What is this transformation doing: https://github.com/YoungXIAO13/PoseFromShape/blob/master/auxiliary/dataset.py#L15 ? From the names of the variable, I see that it uses PCA, but don't understand the approach ...

I have tried to google it (application of methods using PCA on image preprocessing), didn't have success ...

Maybe just a link describing this technique would be enough?

Also, I have visualized the images after this transformation, and I see that it adds unreal artifacts to image (small regions of pixels with totally different colors). Also, my visualization was on images that weren't normalized, and you in the code first do the normalization and then apply this transformation (here: https://github.com/YoungXIAO13/PoseFromShape/blob/master/auxiliary/dataset.py#L202). Do you know what's the effect of this transformation on normalized images (since normalized images cannot be visualized)?

shapnetcore error

Traceback (most recent call last):
File "training.py", line 219, in
criterion_azi, criterion_ele, criterion_inp, criterion_reg, optimizer)
File "training.py", line 159, in train
for i, data in enumerate(data_loader):
File "/home/abhishek/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 568, in next
return self._process_next_batch(batch)
File "/home/abhishek/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 608, in _process_next_batch
raise batch.exc_type(batch.exc_msg)
FileNotFoundError: Traceback (most recent call last):
File "/home/abhishek/.local/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 99, in _worker_loop
samples = collate_fn([dataset[i] for i in batch_indices])
File "/home/abhishek/.local/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 99, in
samples = collate_fn([dataset[i] for i in batch_indices])
File "/home/abhishek/UnsupervisedFMapNet/Scr/PoseFromShape/auxiliary/dataset.py", line 435, in getitem
renders = read_multiviwes(self.render_transform, render_example_path, self.render_number, self.tour, mutation)
File "/home/abhishek/UnsupervisedFMapNet/Scr/PoseFromShape/auxiliary/dataset.py", line 92, in read_multiviwes
render_names = [name for name in os.listdir(render_example_path)]
FileNotFoundError: [Errno 2] No such file or directory: 'data/ShapeNetCore/Renders_semi_sphere/02691156/19e531e6d0bd5ddd67c54439d6177032/crop'

Trouble running inference.sh (no CUDA-capable device detected)

Hello,
I'm simply trying to run the inference.sh script with some of my own pictures and models, but when I do so I get this output:

Color management: using fallback mode for management
Namespace(bin_size=15, image_path='data/TaskBoard/img', img_feature_dim=1024, input_dim=224, model='model/ObjectNet3D.pth', obj_path='data/TaskBoard/obj/GearLarge.obj', render_path='data/TaskBoard/mv_gearlarge/crop', shape='MultiView', shape_feature_dim=256, tour=2, view_num=12)
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1549627089062/work/aten/src/THC/THCGeneral.cpp line=51 error=38 : no CUDA-capable device is detected
Traceback (most recent call last):
File "inference.py", line 45, in
model.cuda()
File "/home/pgavriel/anaconda3/envs/PoseFromShape/lib/python3.6/site-packages/torch/nn/modules/module.py", line 260, in cuda
return self._apply(lambda t: t.cuda(device))
File "/home/pgavriel/anaconda3/envs/PoseFromShape/lib/python3.6/site-packages/torch/nn/modules/module.py", line 187, in _apply
module._apply(fn)
File "/home/pgavriel/anaconda3/envs/PoseFromShape/lib/python3.6/site-packages/torch/nn/modules/module.py", line 187, in _apply
module._apply(fn)
File "/home/pgavriel/anaconda3/envs/PoseFromShape/lib/python3.6/site-packages/torch/nn/modules/module.py", line 187, in _apply
module._apply(fn)
File "/home/pgavriel/anaconda3/envs/PoseFromShape/lib/python3.6/site-packages/torch/nn/modules/module.py", line 193, in _apply
param.data = fn(param.data)
File "/home/pgavriel/anaconda3/envs/PoseFromShape/lib/python3.6/site-packages/torch/nn/modules/module.py", line 260, in
return self._apply(lambda t: t.cuda(device))
File "/home/pgavriel/anaconda3/envs/PoseFromShape/lib/python3.6/site-packages/torch/cuda/init.py", line 162, in _lazy_init
torch._C._cuda_init()
RuntimeError: cuda runtime error (38) : no CUDA-capable device is detected at /opt/conda/conda-bld/pytorch_1549627089062/work/aten/src/THC/THCGeneral.cpp:51
Error: Not freed memory blocks: 8, total unfreed memory 0.008392 MB

What I don't understand is, my anaconda installation is in my home directory, and there is no /opt/conda/ folder on my machine, but I don't see where it's being told to look under /opt/conda/ for CUDA devices. Do I need to build CUDA drivers from source? Did I configure something wrong? Any help would be greatly appreciated. Thanks!

conda error 404 matplotlib not found

CondaError: CondaHTTPError: HTTP 404 NOT FOUND for url https://conda.anaconda.org/conda-forge/linux-64/matplotlib-3.0.2-py36h8a2030e_1.tar.bz2
Elapsed: 00:00.317496
CF-RAY: 5038d1c95c3968d6-CDG

Thank you for your help!

error in running train_Pascal3d.sh for PointCloud

Hello,
I ran train_Pascal3d.sh for shape=MultiView. It runs fine without a problem.
But, when I tried Pointcloud it gives the following error :

Namespace(batch_size=16, bin_size=15, dataset='Pascal3D', decrease=200, img_feature_dim=1024, keypoint=False, lr=0.0001, model=None, n_epoch=300, novel=False, print_freq=50, random=True, random_range=0, shape='PointCloud', shape_dir='pointcloud', shape_feature_dim=1024, tour=2, view_num=12, workers=4)
Random Seed: 6555
Traceback (most recent call last):
File "training.py", line 225, in
criterion_azi, criterion_ele, criterion_inp, criterion_reg, optimizer)
File "training.py", line 165, in train
for i, data in enumerate(data_loader):
File "/home/anaconda3/envs/PoseFromShape/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 637, in next
return self._process_next_batch(batch)
File "/home/anaconda3/envs/PoseFromShape/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 658, in _process_next_batch
raise batch.exc_type(batch.exc_msg)
OSError: Traceback (most recent call last):
File "/home/anaconda3/envs/PoseFromShape/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 138, in _worker_loop
samples = collate_fn([dataset[i] for i in batch_indices])
File "/home/anaconda3/envs/PoseFromShape/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 138, in
samples = collate_fn([dataset[i] for i in batch_indices])
File "/home/PoseFromShape/auxiliary/dataset.py", line 298, in getitem
point_cloud = read_pointcloud(example_path, self.point_num, mutation)
File "/home/PoseFromShape/auxiliary/dataset.py", line 129, in read_pointcloud
point_cloud_raw = pymesh.load_mesh(model_path).vertices
File "/home/anaconda3/envs/PoseFromShape/lib/python3.6/site-packages/pymesh/meshio.py", line 21, in load_mesh
raise IOError("File not found: {}".format(filename));
OSError: File not found: data/Pascal3D/pointcloud/boat/03/compressed.ply

I cannot find compressed.ply file myself and getting an error. Please see how it can be resolved.

Different number of classes (bins) for angles error

When trying different number of classes (bins) for angles, I get

/opt/conda/conda-bld/pytorch_1550802451070/work/aten/src/THCUNN/ClassNLLCriterion.cu:105: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [14,0,0] Assertion `t >= 0 && t < n_classes` failed.
Traceback (most recent call last):
  File "training.py", line 278, in <module>
    main()
  File "training.py", line 230, in main
    criterion_azi, criterion_ele, criterion_inp, criterion_reg, optimizer)
  File "training.py", line 187, in train
    loss_reg = criterion_reg(out[3], out[4], out[5], label.float())
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/code/auxiliary/loss.py", line 42, in forward
    return delta_loss(pred_azi, pred_ele, pred_rol, target, self.__bin__)
  File "/code/auxiliary/loss.py", line 28, in delta_loss
    delta_azi = pred_azi[torch.arange(pred_azi.size(0)), target_label[:, 0]].tanh() / 2
RuntimeError: CUDA error: device-side assert triggered

The error happens inside the DeltaLoss class in the delta_loss method in the line:

delta_azi = pred_azi[torch.arange(pred_azi.size(0)), target_label[:, 0]].tanh() / 2

(link to line: https://github.com/YoungXIAO13/PoseFromShape/blob/master/auxiliary/loss.py#L29)

I tried debugging, but wasn't successful. I mostly use TF, so maybe I'm missing something very common in Pytorch?

When I use default number of classes (azimuth=24, elevation=12, in-plane=24), training runs normally and completes.

using GT boxes to crop images

Are you using GT boxes to crop the images in all the experiments in this paper? I am assuming StarMap, as well as all other baselines, do not use the cropped images and that is the reason for their worse performance. They need to simultaneously localize the object as well as estimate the viewpoint. Can you please correct me if I am wrong?

about downloading checkpoints

i can't download your checkpoints from provided links, could you provide other links.
thanks a lot!

Questions about the evaluation metric

Thanks for sharing the codes and it is really helpful! I'm confused about the evaluation metrics. It seems the arccos((Trace(R_pred^T R_gt) - 1) / 2) is used to measure the viewpoint estimation error. However, a more commonly used metric is the geodesic distance |log R_pred^T R_gt|_F / \sqrt{2}. The first one seems to have higher values (i.e. rotation error) than the second one, so the scores of your model might be better if you use the geodesic distance. I'm not sure if I misunderstood anything here. What are your thoughts on this issue?

LINEMOD data error

I downloaded LINEMOD dataset and then ran ply2obj.py create_annotation.py. they both run fine. But while running render.py for the same I get the following error :

Python 3.6.9 |Anaconda, Inc.| (default, Jul 30 2019, 19:07:31)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

import bpy
Traceback (most recent call last):
File "", line 1, in
ImportError: libopenal.so.1: cannot open shared object file: No such file or directory

CGAL_LIBRARIES-NOTFOUND

Hi, I'm following your instructions and try to find the executable_virtual_scanner, but here comes the error. I don't know why CGAL_LIBRARIES cannot be found. I'm working on Ubuntu 20.04

cmake -DCMAKE_BUILD_TYPE=Release ..
-- Found Boost: /usr/lib/x86_64-linux-gnu/cmake/Boost-1.71.0/BoostConfig.cmake (found version "1.71.0") found components: system
-- No preference for use of exported Eigen CMake configuration set, and no hints for include directory provided. Defaulting to preferring an installed/exported Eigen CMake configuration if available.
-- Found installed version of Eigen: /usr/lib/cmake/eigen3
-- Found Boost: /usr/lib/x86_64-linux-gnu/cmake/Boost-1.71.0/BoostConfig.cmake (found version "1.71.0") found components: thread
-- CGAL_INCLUDE_DIR=/usr/include
-- CGAL_LIBRARIES=CGAL_LIBRARIES-NOTFOUND
-- BOOST_THREAD_LIBRARIES=Boost::thread
-- GMP_LIBRARIES=/usr/lib/x86_64-linux-gnu/libgmp.so
-- CGAL not found.
-- Found Boost: /usr/lib/x86_64-linux-gnu/cmake/Boost-1.71.0/BoostConfig.cmake (found version "1.71.0") found components: filesystem
-- Configuring done
CMake Error at cpp/libvirtualscanner/CMakeLists.txt:15 (add_library):
Target "virtualscanner" links to target "CGAL::CGAL" but the target was not found. Perhaps a find_package() call is missing for an IMPORTED target, or an ALIAS target is missing?

CMake Error at cpp/virtualscanner/CMakeLists.txt:5 (add_executable):
Target "virtualscannerexe" links to target "CGAL::CGAL" but the target was not found. Perhaps a find_package() call is missing for an IMPORTED target, or an ALIAS target is missing?

CUDA error on running bash run/train_Pascal3D.sh

I just downloaded Pascal dataset and its pretrained models. I see in auxiliary/spec_file.txt CUDA 8 is being installed with python 3.6.
On running bash run/train_Pascal3D.sh I get CUDA VERSION ERROR
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED
Namespace(batch_size=16, bin_size=15, dataset='Pascal3D', decrease=200, img_feature_dim=1024, keypoint=False, lr=0.0001, model=None, n_epoch=300, novel=False, print_freq=50, random=True, random_range=0, shape='PointCloud', shape_dir='pointcloud', shape_feature_dim=1024, tour=2, view_num=12, workers=4)
Random Seed: 2019
/home/anaconda3/envs/PoseFromShape/lib/python3.6/site-packages/torch/cuda/init.py:115: UserWarning:
Found GPU0 Tesla V100-PCIE-16GB which requires CUDA_VERSION >= 9000 for
optimal performance and fast startup time, but your PyTorch was compiled
with CUDA_VERSION 8000. Please install the correct PyTorch binary
using instructions from https://pytorch.org

Question and errors

Hello,
Thank you for sharing the source code.
I've noticed that all the images used in your article are cropped around the object. Did you try your approach otherwise?

I've also started running the code on my machine but I'm getting some errors when running the script python render.py

Color management: using fallback mode for management
  0%|                                                                                                                                                                                                                                                                                            | 0/12 [00:00<?, ?it/sP(  0.0000 sec |   0.0000 sec) Importing OBJ 'CAD/aeroplane/aeroplane.mat'...                                                                                                                                                                                                                      | 0/1 [00:00<?, ?it/s]
  (  0.0001 sec |   0.0001 sec) Parsing OBJ file...
Progress:   0.00%

Error: Traceback (most recent call last):
  File "/home/jaouadros/anaconda3/envs/PoseFromShape/lib/python3.6/site-packages/2.77/scripts/addons/io_scene_obj/__init__.py", line 147, in execute
    return import_obj.load(context, **keywords)
  File "/home/jaouadros/anaconda3/envs/PoseFromShape/lib/python3.6/site-packages/2.77/scripts/addons/io_scene_obj/import_obj.py", line 1066, in load
    context_multi_line = b'l' if strip_slash(line_split) else b''
  File "/home/jaouadros/anaconda3/envs/PoseFromShape/lib/python3.6/site-packages/2.77/scripts/addons/io_scene_obj/import_obj.py", line 835, in strip_slash
    if line_split[-1][-1] == 92:  # '\' char
IndexError: list index out of range

location: /home/jaouadros/anaconda3/envs/PoseFromShape/lib/python3.6/site-packages/2.77/scripts/modules/bpy/ops.py:189

Do you have an idea about what could be the problem?

PASCAL3D+ data link error

I tried downloading data using data/prepare_data.sh but get the following error as the link is not working:
ftp://cs.stanford.edu/cs/cvgl/PASCAL3D+_release1.1.zip cite cannot be reached and thus I cannot download the data.
Is there a work around for the same ?

Detecting 6D pose of all objects in one image

Hi

I am trying to get 6D pose of all recognized objects in an image using your package. Do you have any codes for that? I mean I need the model to detect all of the objects in the image(for example when it is trained with ObjectNet3D I need it to recognize those objects) and output its location and angle of the object. So I can make a cuboid from that object in the 3D space. Is your algorithm capable of this and Do you have any codes ready for doing this?

Thank You

rotation_acc method for accuracy calculation

Hi Yang,

I have a couple of questions regarding the accuracy calculation.

1. I assume that this line converts the angles to [-180, 180]. But you only convert Elevation and In-plane rotation, and not Azimuth angle? Why?

2. These calculations convert from polar coordinates (Azimut, Elevation, In-plane rotation) to the rotation matrix? How did you arrive at these formulas? I tried it myself, didn't succeed. Also, I have tried to search for it on the web, also didn't find anything like that.

Category agnostic explaination

Can someone explain as to how 3D shape information addition is helpful in generalization. The paper just says it does but does not explain a reason for the same. With respect to deep learning please elaborate on how it is able to generalize by just using multiview?

how to get the rotation matrix according to azimuth, elevation and in plane rotation?

Hi Yang,

Thanks for great work.

May i ask how to get the rotation matrix according to azimuth, elevation and in plane rotation angles?

Thanks,

Yuer

rendering issue

I get this when I use your render scripts. Do you have any idea why this might be happening?

Its related to blender and happens when i import bpy.

Color management: using fallback mode for management
ALSA lib confmisc.c:767:(parse_card) cannot find card '0'
ALSA lib conf.c:4528:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:4528:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name
ALSA lib conf.c:4528:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5007:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2495:(snd_pcm_open_noupdate) Unknown PCM default
AL lib: (EE) ALCplaybackAlsa_open: Could not open playback device 'default': No such file or directory

virtualscanner output

Hi

I am trying to make point clouds from ObjectNet3D dataset, but I am getting an error:

ObjectNet3D/CAD/obj/aeroplane/01.obj done! Time: 1186065
Traceback (most recent call last):

  File "/home/mohammad/anaconda3/envs/PoseFromShape/lib/python3.6/shutil.py", line 550, in move
    os.rename(src, real_dst)

FileNotFoundError: [Errno 2] No such file or directory: 'ObjectNet3D/CAD/obj/aeroplane/01.ply' -> 'ObjectNet3D/pointcloud/aeroplane/01/sampled.ply'


During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/home/mohammad/Mohammad_ws/PoseFromShape/data/point_cloud.py", line 78, in <module>
    ply_path = sample_point_cloud_from_obj(args.virtualscanner, model_path, example_dir)

  File "/home/mohammad/Mohammad_ws/PoseFromShape/data/point_cloud.py", line 23, in sample_point_cloud_from_obj
    shutil.move(ply, ply_dest)

  File "/home/mohammad/anaconda3/envs/PoseFromShape/lib/python3.6/shutil.py", line 564, in move
    copy_function(src, real_dst)

  File "/home/mohammad/anaconda3/envs/PoseFromShape/lib/python3.6/shutil.py", line 263, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)

  File "/home/mohammad/anaconda3/envs/PoseFromShape/lib/python3.6/shutil.py", line 120, in copyfile
    with open(src, 'rb') as fsrc:

FileNotFoundError: [Errno 2] No such file or directory: 'ObjectNet3D/CAD/obj/aeroplane/01.ply'

I tried to find the reason. I think currently virtualscanner outputs file in .points format. But it needs to be .ply format. Is that correct? If yes do you know how can I solve the problem? And if no do you know what might be causing the error?

Thank You

on the point cloud data range

Hi congrats on the nice work! I have a small question on the point cloud data of objectnet3d and pascal3d datasets. It seems that the data value ranges in [0,1], so it is not standard processed like modelnet or shapenet dataset where point cloud samples are scaled to a unit ball?

object angles based on pointcloud model

Hi

Thanks for your great package. It has good features, but I think one thing is missing which can help people use your package much easier and attract more people to it. I need to input an image with only one object in it (such as a can) which has been seen while training and also pointcloud of that object and get the object angles as output. I know you are doing something similar in tesitng.py code for testing objects, but that code is complicated and it can be very useful. Is it possible for you to provide such a code?

Thank You

no training, code finishes in few mins

Namespace(azi_classes=24, batch_size=16, dataset='ObjectNet3D', decrease=200, ele_classes=12, img_feature_dim=1024, inp_classes=24, keypoint=True, lr=0.0001, model=None, mutated=True, n_epoch=300, novel=False, num_render=12, pretrained_resnet=False, print_freq=50, random_range=0, shape='MultiView', shape_feature_dim=256, tour=2, workers=4)

Random Seed: 5015

after running, bash run/train_ObjectNet_3D.sh

Using predict for one image

I am trying to use one of your pre-trained models with an object I will use for my project. However, I couldn't find any instructions about how to use the models with another dataset. Could you please explain how to do that? I'm a little lost in your codes right now.

error with train_pascal3D.sh

Hi,
Any help/suggestion would be very helpful.

File "training.py", line 93, in
train_loader = DataLoader(dataset_train, batch_size=opt.batch_size, shuffle=True, num_workers=opt.workers, drop_last=True)
File "/home/abhishek/.local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 176, in init
sampler = RandomSampler(dataset)
File "/home/abhishek/.local/lib/python3.6/site-packages/torch/utils/data/sampler.py", line 66, in init
"value, but got num_samples={}".format(self.num_samples))
ValueError: num_samples should be a positive integer value, but got num_samples=0

shape and image feature dimension

can you please tell me what are the specific shape and image feature dimensions used for the results in the paper, for different datasets? also can you indicate the training time for each dataset?

How to get the trained model?

Hello, congratulate to you! You did a good job. I want to know how to download the file of the trained model ? I can not download it from the download_models.sh, can you help me? thank you for your help.