[CVPR 2022 Oral, Best Student Paper] EPro-PnP: Generalized End-to-End Probabilistic Perspective-n-Points for Monocular Object Pose Estimation
Home Page: https://www.youtube.com/watch?v=TonBodQ6EUU
License: Apache License 2.0
First of all, I congratulate you on winning the honor of cvpr2022 best student paper.
I recently read your paper and was confused about some of the details:The paper introduces that Epro-PNP output is the probability density distribution of attitude. How can we finally calculate the deterministic attitude information from the density distribution? I didn't understand this point in the paper.
I am confused about the meaning of "x3d", it seems like all the "x3d" of an object are in one plane, how can PnP calculate pose with this 3d points？
here is my test result about x3d in bev (the first picture)
EPro-PnP/EPro-PnP-6DoF/lib/train.py
Line 163 in 2097276
Hi,
thanks for your wonderful paper - It's a pleasure to read!
Is there a way to download the BOP related data without a baidu account? I cannot register to their service because my cellphone number does not conform to the required format. If not, could you maybe describe the data layout in more detail, so that one assemble the data oneself ?
Hi, Very nice work to follow. I wonder how to visualize the orientation like this figure?
Thank you so much for your contributions.
Hi, authors!
Recently I've been following your work EPro-PnP-6DoF.
I have a problem with sampling points during training.
I've noticed that you randomly sampled 512 points in the output x2d/x3d images (code here). But there might be the issue with sampling some background points and disturbing the procedure of estimating object pose.
Have you ever considered this or conducted any experiments?
When I run the test, it show file not found error.
so where can i get the models_info.
Hi, when I read the paper, I got intuition saying: pose ambiguity can be handled by multiple modes learned by the model. However, right after that intuition, you said that: empirically, Dirac-like distribution works best, resulting in the simplified KL-divergence as eq (5).
How to understand the intuition and the empirical finding?
Thanks,
Hello, I want to know how to use EPro-PnP-6DOF to get the detailed position and pose infomation(like x y z r p y) instead of distribution map？
Thank you！
I always have problems with the code.
usage: main.py [-h] --cfg CFG [--load_model LOAD_MODEL] [--debug] [--test]
main.py: error: the following arguments are required: --cfg
I downloaded the data set to train the model. But the Main file has this problem.
May I ask what kind of adjustment I should make for the above problems?
Hello,
Firstly i wanna thank you for the excellent work! I have three questions regarding the norm factor(Global scaling) learned in EPro-PnP-6DoF's rotation head.
Thank you very much!
hello, thanks for your source code firstly.
i want to know what is the difference between "pose_opt" and "pose_opt_plus"
if i want to use pose in my own loss, which one should i pick ?
thanks for your answer !
return pose_opt, cost, pose_opt_plus, pose_samples, pose_sample_logweights, cost_init
Hi,
Thanks for your great job!
I have some questions about the performance of EPro-PnP, can I use EPro-PnP as a faster PnP solver to replace the traditional PnP(e.g. EPnP with Ransec by cv2.solvePnPRansac)?.
I try to use EPro-PnP to replace the EPnP as a post-processing module to solve the 2D-3D registration problem(image to point cloud) , but I found that EPro-PnP performs well in easy mode but will cause large errors in difficult mode (with some false correspondences) while EPnP (with Ransec) can handle these false correspondences more robust. Can you give me some suggestions on how to avoid large errors? Thanks a lot.
The class name 'DefromPnPHead' would be 'DeformPnPHead', is that correct? :-P although it doesn't really affect anything at all...
Dear authors,
Thanks for releasing your excellent work.
I'm training EPro-PnP-6DoF following your instructions provided. I noticed that loss_mc sometimes becomes negative, resulting in the overall loss being negative as well. I believe this is caused by the logarithm in the computation for L_pred, as the log of the number between 0 and 1 is negative. Is this behavior expected? Could you please elaborate a little on the logic behind it?
Much appreciated your reply in advance!
楼主您好，这篇论文的思路太惊艳了，我非常想在自己的数据集上测试一下，但是环境配置了好几天都没能正常运行demo，能不能麻烦您帮忙开放了一个docker，万分感谢！！！
Hi there,
I tried the pertained model and it works well. I am currently reading the code and got confused by the following code:
where it says
# get image projection
cam_points = (lidar_points - cam_info['sensor2lidar_translation']
) @ cam_info['sensor2lidar_rotation']
Why do you use cam_info['sensor2lidar_rotation'] here? It seems that cam_info['sensor2lidar_rotation'] means rotation from sensor to lidar and so if you would like to transform from lidar, you may need to use inverse like np.linalg.inv(cam_info['sensor2lidar_rotation']). why not use the inverse? Thank you.
hello, i use your trained model to test 6D pose estimate in linemod dataset, but i find all the accuracies are zeros and none. what's wrong with my code?
Thanks for sharing your work.
Dear Author
Thank you very much for your great and amazing work, I have got the following error when I run this command line
pip install -v -e . .please help me in this case.
Thanks in advanced
The environment is in the following:
Installing collected packages: epropnp-det
Running setup.py develop for epropnp-det
Running command python setup.py develop
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda-11.3'
running develop
running egg_info
writing epropnp_det.egg-info/PKG-INFO
writing dependency_links to epropnp_det.egg-info/dependency_links.txt
writing requirements to epropnp_det.egg-info/requires.txt
writing top-level names to epropnp_det.egg-info/top_level.txt
reading manifest file 'epropnp_det.egg-info/SOURCES.txt'
writing manifest file 'epropnp_det.egg-info/SOURCES.txt'
running build_ext
building 'epropnp_det.ops.iou3d.iou3d_cuda' extension
gcc -pthread -B /home/koosha/anaconda3/envs/epropnp_det/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DWITH_CUDA -I/home/koosha/anaconda3/envs/epropnp_det/lib/python3.7/site-packages/torch/include -I/home/koosha/anaconda3/envs/epropnp_det/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/koosha/anaconda3/envs/epropnp_det/lib/python3.7/site-packages/torch/include/TH -I/home/koosha/anaconda3/envs/epropnp_det/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda-11.3/include -I/home/koosha/anaconda3/envs/epropnp_det/include/python3.7m -c epropnp_det/ops/iou3d/src/iou3d.cpp -o build/temp.linux-x86_64-cpython-37/epropnp_det/ops/iou3d/src/iou3d.o -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -DTORCH_EXTENSION_NAME=iou3d_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
/home/koosha/anaconda3/envs/epropnp_det/lib/python3.7/site-packages/setuptools/command/easy_install.py:147: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
EasyInstallDeprecationWarning,
/home/koosha/anaconda3/envs/epropnp_det/lib/python3.7/site-packages/setuptools/command/install.py:37: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
setuptools.SetuptoolsDeprecationWarning,
/home/koosha/anaconda3/envs/epropnp_det/lib/python3.7/site-packages/torch/utils/cpp_extension.py:381: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
warnings.warn(msg.format('we could not find ninja.'))
Traceback (most recent call last):
File "", line 36, in
File "", line 34, in
File "/home/koosha/anaconda3/EPro-PnP/EPro-PnP-Det/setup.py", line 143, in
zip_safe=False)
File "/home/koosha/anaconda3/envs/epropnp_det/lib/python3.7/site-packages/setuptools/init.py", line 87, in setup
return distutils.core.setup(**attrs)
File "/home/koosha/anaconda3/envs/epropnp_det/lib/python3.7/site-packages/setuptools/_distutils/core.py", line 185, in setup
return run_commands(dist)
File "/home/koosha/anaconda3/envs/epropnp_det/lib/python3.7/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
dist.run_commands()
File "/home/koosha/anaconda3/envs/epropnp_det/lib/python3.7/site-packages/setuptools/_distutils/dist.py", line 968, in run_commands
self.run_command(cmd)
File "/home/koosha/anaconda3/envs/epropnp_det/lib/python3.7/site-packages/setuptools/dist.py", line 1217, in run_command
super().run_command(command)
File "/home/koosha/anaconda3/envs/epropnp_det/lib/python3.7/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
cmd_obj.run()
File "/home/koosha/anaconda3/envs/epropnp_det/lib/python3.7/site-packages/setuptools/command/develop.py", line 34, in run
self.install_for_development()
File "/home/koosha/anaconda3/envs/epropnp_det/lib/python3.7/site-packages/setuptools/command/develop.py", line 114, in install_for_development
self.run_command('build_ext')
File "/home/koosha/anaconda3/envs/epropnp_det/lib/python3.7/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
self.distribution.run_command(command)
File "/home/koosha/anaconda3/envs/epropnp_det/lib/python3.7/site-packages/setuptools/dist.py", line 1217, in run_command
super().run_command(command)
File "/home/koosha/anaconda3/envs/epropnp_det/lib/python3.7/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
cmd_obj.run()
File "/home/koosha/anaconda3/envs/epropnp_det/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 84, in run
_build_ext.run(self)
File "/home/koosha/anaconda3/envs/epropnp_det/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 346, in run
self.build_extensions()
File "/home/koosha/anaconda3/envs/epropnp_det/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 735, in build_extensions
build_ext.build_extensions(self)
File "/home/koosha/anaconda3/envs/epropnp_det/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 466, in build_extensions
self._build_extensions_serial()
File "/home/koosha/anaconda3/envs/epropnp_det/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 492, in _build_extensions_serial
self.build_extension(ext)
File "/home/koosha/anaconda3/envs/epropnp_det/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 246, in build_extension
_build_ext.build_extension(self, ext)
File "/home/koosha/anaconda3/envs/epropnp_det/lib/python3.7/site-packages/setuptools/_distutils/command/build_ext.py", line 554, in build_extension
depends=ext.depends,
File "/home/koosha/anaconda3/envs/epropnp_det/lib/python3.7/site-packages/setuptools/_distutils/ccompiler.py", line 599, in compile
self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
File "/home/koosha/anaconda3/envs/epropnp_det/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 483, in unix_wrap_single_compile
cflags = unix_cuda_flags(cflags)
File "/home/koosha/anaconda3/envs/epropnp_det/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 450, in unix_cuda_flags
cflags + _get_cuda_arch_flags(cflags))
File "/home/koosha/anaconda3/envs/epropnp_det/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1606, in _get_cuda_arch_flags
arch_list[-1] += '+PTX'
IndexError: list index out of range
error: subprocess-exited-with-error
python setup.py develop did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
full command: /home/koosha/anaconda3/envs/epropnp_det/bin/python -c '
exec(compile('"'"''"'"''"'"'
# This is <pip-setuptools-caller> -- a caller that pip uses to run setup.py
#
# - It imports setuptools before invoking setup.py, to enable projects that directly
# import from `distutils.core` to work with newer packaging standards.
# - It provides a clear error message when setuptools is not installed.
# - It sets `sys.argv[0]` to the underlying `setup.py`, when invoking `setup.py` so
# setuptools doesn'"'"'t think the script is `-c`. This avoids the following warning:
# manifest_maker: standard file '"'"'-c'"'"' not found".
# - It generates a shim setup.py, for handling setup.cfg-only projects.
import os, sys, tokenize
try:
import setuptools
except ImportError as error:
print(
"ERROR: Can not execute `setup.py` since setuptools is not available in "
"the build environment.",
file=sys.stderr,
)
sys.exit(1)
__file__ = %r
sys.argv[0] = __file__
if os.path.exists(__file__):
filename = __file__
with tokenize.open(__file__) as f:
setup_py_code = f.read()
else:
filename = "<auto-generated setuptools caller>"
setup_py_code = "from setuptools import setup; setup()"
exec(compile(setup_py_code, filename, "exec"))
'"'"''"'"''"'"' % ('"'"'/home/koosha/anaconda3/EPro-PnP/EPro-PnP-Det/setup.py'"'"',), "<pip-setuptools-caller>", "exec"))' develop --no-deps
cwd: /home/koosha/anaconda3/EPro-PnP/EPro-PnP-Det/
error: subprocess-exited-with-error
python setup.py develop did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
Hello, thank you for your great work!
Could you tell me the different between EPro-PnP-Det v2 and EPro-PnP-Det v1 in nuscense leader board?
Hi, this is really a fantastic work. I tried 4DoF model and it works seamlessly especially when the car are nearby and not truncated. However, I noticed that for objects that are a little far from the came (e.g. > 80 meters), the translation becomes unstable enough. For example, in frame n-1 the object might be 85 meters away but in frame n it may be 80 or 90 meters away. I understand that the relative precision might still acceptable but the absolute error should not take into account. So I'm wondering is there any approach to eliminate the error for far objects (both training or inferring are fine)?
Hello!For 6D pose estimation,if number of 2D-3D pairs is 8 , can I use EPro-Pnp ?Due to certain reasons，dense 2D-3D correspondse is unavailable.
Hi, thanks for the repo and paper, seriously good work!
I trained the network on my own data and maybe you have an idea where to look to improve performance, judged on these results. Especially ADD and ARP are super low. Any idea? I suspect that I probably did something wrong with data preparation since the training on the LM data works great.
[0708 15:57:00 @eval.py:75] ------------ x -----------
[0708 15:57:00 @eval.py:76] [rot_thresh, trans_thresh: RotAcc, TraAcc, SpcAcc
[0708 15:57:00 @eval.py:77] average_accuracy[-1, -1.00]: 74.14, 80.30, 69.60
[0708 15:57:00 @eval.py:84] average_accuracy[ 2, 0.02]: 48.48, 70.71, 43.43
[0708 15:57:00 @eval.py:84] average_accuracy[ 5, 0.05]: 82.83, 83.84, 78.79
[0708 15:57:00 @eval.py:84] average_accuracy[10, 0.10]: 90.91, 91.92, 88.89
[0708 15:57:00 @eval.py:92] ---------- performance over 1 classes -----------
[0708 15:57:00 @eval.py:93] [rot_thresh, trans_thresh: RotAcc, TraAcc, SpcAcc
[0708 15:57:00 @eval.py:95] average_accuracy[-1, -1.00]: 74.14, 80.30, 69.60
[0708 15:57:00 @eval.py:104] average_accuracy[ 2, 0.02]: 48.48, 70.71, 43.43
[0708 15:57:00 @eval.py:104] average_accuracy[ 5, 0.05]: 82.83, 83.84, 78.79
[0708 15:57:00 @eval.py:104] average_accuracy[10, 0.10]: 90.91, 91.92, 88.89
[0708 15:57:00 @eval.py:120]
Is something wrong in my Installation or other problem? Thanks a lot ~
I get a wrong result by forward_test.
The result will be correct when I reverse the y coordinate
Hi, first congratulate and respect to your great work!
I find the dataset structure in your "data preparation" part is different from that of the official BOP dataset, i.e. the official BOP dataset may not contain VOC2012 as background and the content in LM is different from yours. So I wonder how to test on BOP dataset (using BOP toolkit) with your EPro-pnp. Is there any suggestions or existed work? Thanks a lot!
The 6DoF asks for "/dataset/lm/models/models_info.txt" but I couldnt find this file!
File "lm.py", line 30, in __init__ self.infos = self.load_lm_model_info(ref.lm_model_info_pth) File "lm.py", line 143, in load_lm_model_info with open(info_pth, 'r') as f:
老师，您好，有一个问题，困惑了我好长时间了，今天特意来此处，请教一下您，渴望能够得到您的解答哟，先感谢一个哟。
我们的应用场景是用机械手去抓取一个指定的目标对象，因此，我们需要准确的解析出目标相对于相机坐标系的位姿。我们的图像是通过“Creator建模+Vega仿真”的技术途径来生成的，也就是说，所有的图像都是仿真生成的，另外，在仿真的过程中，我们还可以同步的输出目标在当前观测视点下的位姿（可以转换成含有旋转和平移分量的尺寸为4×4的齐次变换矩阵）。在制作LineMOD样式的自定义数据集时，我们遇到了如下的问题：
怎么生成“lm\real_train\ape\000005-coor.pkl”样式的数据呢？或者说，“000005-coor.pkl”样式的数据，是怎么制作出来的呢？
我仔细的分析了一下，“000005-coor.pkl”这个文件当中的coor，其形状为(57, 44, 3)，该尺寸跟同一路径下的“000005-box.txt”里面的数据是相吻合的：
紧接着，我们统计了一下coor当中的非零行的数量（感觉这个数据想要表达的是目标的顶点数量），发现它跟同一路径下的“000005-label.png”当中的非零像素的总量，并不相等：
基于此差异，我做出了进一步的推断：我们可能需要根据目标的姿态信息（正如同一路径下的“000005-pose.txt”所反映出来的那样）来变换原始的模型文件（models\ape\ape.ply），以便生成跟“000005-label.png”等组合数据相对应的coor.pkl文件，为此，我尝试了一下Matlab当中的pctransform函数，遗憾的是，目前对于此函数的调用，还没有成功。
又或者是coor.pkl这个文件跟原始的模型文件ape.ply之间，并没有显示的对应关系呢？
老师，不知道我的分析思路是否正确呢？或者说，我到底应该采取什么样的解析途径，才能将coor.pkl这个数据的制作规律，给摸索清楚呢？可否请您指点一下迷津呢？万分感谢哟。
First of all, I congratulate you on winning the honor of cvpr2022 best student paper.
I am very confused about generating the probability density, whether the peak of the probability density is the coordinates of the object, I made some annotations in the figure below.
DEPro-PnP-main\EPro-PnP-6DoF\dataset\lm\real_test\cat\000001-coor.pkl.
I want to ask what the meaning of "coor" in the pkl file. And how is it generated?
How to get Eq.(7) from Eq.(5) in the paper?
Thank you for sharing your code. During inference, when we obtain the pose distribution p(y|X) by the probabilistic PnP layer, how to use the p(y|X) to obtain the y*? We can not understand the Section 3.4.
During execute process, it was found that init_detector, inference_detector, show_result file were missing
你好，感谢开源。
请教一个论文中3.1节公式3的问题
根据条件概率公式展开后，为什么分子和分母中的 p(y) 可以隐去？
Hi HanSheng, very excellent work!
In equation (2), likelihood fuction p(X|y) is defined as .
We can view p(X|y) as the joint probability of p(X1|y), ... , p(X2|y). So .
||fi(y)|| is reprojective error and its value from 0 to infinity. Let x be ||fi(y)||, . So p(Xi|y) is not a pdf.
Is my induction correct? If my induction is correct, statement p(X|y) is not proper here. And following equation can't use Bayes theorem to get p(y|X).
Perhaps i'm splitting hairs, but it really confused me.
Hi, I encounter an error when training using cdpn init config.
_, R_vector, T_vector = cv2.solvePnP(
IndexError: boolean index did not match indexed array along dimension 1; dimension is 64 but corresponding boolean dimension is 2
The error codes are
_, R_vector, T_vector = cv2.solvePnP( x3d_np_[mask_np_], x2d_np_[mask_np_], cam_intrinsic_np, dist_coeffs, flags=cv2.SOLVEPNP_EPNP)
Specifically:
x3d_np_[mask_np_]
I downloaded dataset sub-datasets "full dataset(v1.0)" and "Teaser dataset (v0.1) -- Deprecated" in "nuScenes" to pre-process the data.
When I run the follow code:"python tools/data_converter/nuscenes_converter.py data/nuscenes --version v1.0-trainval"
An error:"FileNotFoundError: file "data/nuscenes/samples/LIDAR_TOP/n015-2018-08-02-17-16-37+0800__LIDAR_TOP__1533201470448696.pcd.bin" does not exist
"
Please tell me how to slove it ? Thanks!
Can you tell me how to use EPro-pnp in https://github.com/sjtuytc/segmentation-driven-pose this method?
Hello, I want to ask a question.
Hi, after running the two programs, where is the pose data stored, and I only see the visualization results, Thank you.
Hi,
I would like to ask how many frames per second you reach on your reported results on the LineMOD Benchmark. Thx!
我想把EPro-PnP-Det训练的模型转成onnx格式,但中途遇到了很多问题,希望能提供转换脚本帮助我快速找到问题所在,谢谢
Hi, I run the codes with config epropnp_basic.yaml, but the results seems to be lower than reported. I post log here.
Hi, after running the two programs, where is the pose data stored, and I only see the visualization results, Thank you.
IndexError: boolean index did not match indexed array along dimension 1; dimension is 64 but corresponding boolean dimension is 2.
I found that dim=1 in pred_conf_np = w2d.mean(dim=1).cpu().numpy() needs to be modified to -1.
But after that there is a new error as follow.
Traceback (most recent call last):
File "main.py", line 108, in
main()
File "main.py", line 77, in main
_, preds = test(0, cfg, test_loader, network, obj_vtx, obj_info, criterions)
File "/home/raid/james/EPro-PnP-main/EPro-PnP-6DoF/tools/../lib/test.py", line 223, in test
pose_init=pose_opt, force_init_solve=False, fast_mode=True)
File "/home/raid/james/EPro-PnP-main/EPro-PnP-6DoF/tools/../lib/ops/pnp/epropnp.py", line 144, in monte_carlo_forward
new_trans_distr, new_rot_distr = self.gen_new_distr(i, *distr_params)
File "/home/raid/james/EPro-PnP-main/EPro-PnP-6DoF/tools/../lib/ops/pnp/epropnp.py", line 306, in gen_new_distr
new_trans_distr = MultivariateStudentT(3, trans_mode[iter_id], trans_cov_tril[iter_id])
File "/home/ubuntu/.local/lib/python3.6/site-packages/pyro/distributions/distribution.py", line 18, in call
return super().call(*args, **kwargs)
File "/home/ubuntu/.local/lib/python3.6/site-packages/pyro/distributions/multivariate_studentt.py", line 45, in init
super().init(batch_shape, event_shape, validate_args=validate_args)
File "/home/ubuntu/.local/lib/python3.6/site-packages/torch/distributions/distribution.py", line 53, in init
raise ValueError("The parameter {} has invalid values".format(param))
ValueError: The parameter loc has invalid values
Hi, I wonder where is the codes for equation (8)?
I cannot find it in your code.
Thanks.
Congratulations on winning the honor of cvpr2022 best student paper and thanks for sharing your work!
However, the computation of the Jacobian matrix confused me a lot. In your code,
Line 121 in 1545131
When the dof equals to 6,
d_x3dcam_d_rot = skew(x3d_rot * 2)
jac = torch.cat((d_x2d_d_x3dcam, d_x2d_d_x3dcam @ d_x3dcam_d_rot), dim=-1)
Could you please tell me how do you get the Jacobian matrix? It’s a little difficult for me to understand the d_x3dcam_d_rot, and I wish your reply could help me to figure out why the x3d_rot should be multiplied by 2, and why only the rotation is considered.
Additionally, I referred to some materials when trying to compute the Jacobian matrix, for example, in this article https://zhuanlan.zhihu.com/p/482540286, they compute the Jacobian matrix of the reprojection errors in this way:
I think if I follow this equation, the code should be
x3d_cam = x3d @ quaternion_to_rot_mat(pose[..., 3:]).transpose(-1, -2)+ pose[..., None, :3]
jac = torch.cat((d_x2d_d_x3dcam, d_x2d_d_x3dcam @ skew(x3d_cam)), dim=-1)
Which is different from your algorithm, and it puzzles me a lot. I am looking forward to your reply, and thank you again for your great job.
Hi~
Thanks for your wonderful work.
I have a question about your paper. Why you choose a mixture of von Mises and uniform distribution rather than t-distribution or just uniform distribution to be the proposal distribution of 1D orientation?
Could you please give me some suggestions about how to choose the proposal distribution?
Thanks~
Another question is about the Jacobian J. Since your final optimization target is changed from equation 1 to equation 5, why J still equals to ∂fi(y)/∂y? (The Lpred in equation 5 is not considered in this Jacobian J I think.)
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.