oneeyedeagle/eagle-texturemapping

Patch-Based Optimization for Image-Based Texture Mapping

EAGLE-TextureMapping

This is an attempt for implementation of the paper "Patch-Based Optimization for Image-Based Texture Mapping".

Before use

1. Make sure you have installed Boost, Eigen3, VTK 6.3, PCL 1.9, OpenCV 4, and ImageMagick. (I'm using Ubuntu 16.04, but the project works if you successfully install these things on Windows.)

2. Open the project with Qt creator (be sure to set the Make path as same as the project path), then you need to edit the INCLUDE paths in eagle_textureMapping.pro file to fit in your envirenment. (If necessary, recompile ./lib/Eagle_Utils.cpp with g++ -shared -o libEagle_Utils.so Eagle_Utils.cpp , if the OpenCV4 is in the system path.)

3. [UPDATE in 2020.1.9] I upgrade a mini-version of TextureMapping codes, which only include things relating to Patch-Based Optimization. The old version can still be found on branch Full-Ver.

4. [UPDATE in 2020.9.7] I added some settings in getalignresults.cpp file's header, check their comments for more information. Also I editted the obj generation function for much better result, now you could edit the function pointProjectionValidMesh to change the condition when generating the obj. It's recommanded to set a more rigid condition then that when iterating, to avoid reprojecting which is on the boundary.

How to use

All parameters are in the settings.h file, change them to your own data's info as the comment says. (The data in the project has been configed on init_zhou() function.)

• If you have a PLY file:

  1. It's recommended to choose some keyframes manually for better results, also to delete the keyframe which shifts too much among others. (what the kfIndexs variable does.) (That is to say, these frames' camera matrixs are inaccurate or with large errors. It's really difficult for my code to make these frames fit others. So ghosting areas remain at such situation.)

  2. Change settings.h, especially checking for kfCameraTxtFile and plyFile under the keyFramesPath.

  3. Run the project.

About settings

• The patchStep variable controls the patch numbers when voting. If it's set to 1, the lamda variable needs to be large enough to make Targets not as same as Sources. (I explore patchStep = patchWidth / 2 makes lamda = 0.2 ~ 0.4 effectively work, but larger lamda makes the result fall into local minima too early, while smaller lamda makes no sense as Textures will do nothing to Targets.)

• The scaleInitH variable sets the first scale resolution's height. I find if it's set to 64, it's quite difficult to store minor textures at finer scales in my datas. So I set it to originImgH / 4 which is better than too small.

• [UPDATE in 2020.9.7] I added some limits when remapping (when pixels on the image A will be projected to the world, then projected to the image B), which could be found in the function pointProjectionValid.

  1. I make sure the depth is close to the origin on image B, to avoid occlusion.

  2. As pixels on the boundary (which means its neighbors may be on background.) will make mistakes, I disgard all pixels that projected to the boundary of the object.

  3. While the angle between ray (which is a line from the pixel's world position to the mesh) and the mesh is calculated, I add a condition that the angle should be larger than 90°, then the mesh is more close to the parallel state with the image plane.

eagle-texturemapping's Issues

Problems encountered during environment configuration

When I imported the project into Qt creator, when configuring the Kit, I should choose which version of MinGW. He kept giving me the error that the Kit was not configured, but I couldn't choose the Kit on the left either.

Issue compiling the project

g++ -c -pipe -std=gnu++14 -fopenmp -O2 -Wall -W -D_REENTRANT -fPIC -DQT_NO_DEBUG -DQT_GUI_LIB -DQT_CORE_LIB -I. -Irayint -Ilib -isystem /usr/include/eigen3 -isystem /usr/local/include/opencv -isystem /usr/include/ni -I/usr/local/cuda/include -isystem /usr/local/include/vtk-7.1 -isystem /usr/local/include/pcl-1.12 -isystem /usr/include/x86_64-linux-gnu/qt5 -isystem /usr/include/x86_64-linux-gnu/qt5/QtGui -isystem /usr/include/x86_64-linux-gnu/qt5/QtCore -I. -isystem /usr/include/libdrm -I/usr/lib/x86_64-linux-gnu/qt5/mkspecs/linux-g++ -o lib/getalignresults.o getalignresults.cpp
getalignresults.cpp: In member function ‘void getAlignResults::generateTexturedOBJ(std::__cxx11::string, std::__cxx11::string, std::__cxx11::string)’:
getalignresults.cpp:1214:67: error: no matching function for call to ‘getAlignResults::checkMeshMapImg(size_t&, size_t&, std::vector<cv::Point_<int> >&, float&)’
             if (checkMeshMapImg(mesh_i, img_i, v_uv_tmp, img_score) == true) {
                                                                   ^
In file included from getalignresults.cpp:1:0:
getalignresults.h:123:10: note: candidate: bool getAlignResults::checkMeshMapImg(size_t, size_t, std::vector<cv::Point_<int> >&)
     bool checkMeshMapImg(size_t mesh_i, size_t img_i, std::vector<cv::Point2i> &v_uv);
          ^~~~~~~~~~~~~~~
getalignresults.h:123:10: note:   candidate expects 3 arguments, 4 provided
getalignresults.cpp:1254:5: error: ‘writeCameraTraj’ was not declared in this scope
     writeCameraTraj(resultImgNamePattern);
     ^~~~~~~~~~~~~~~
getalignresults.cpp:1254:5: note: suggested alternative: ‘readCameraTraj’
     writeCameraTraj(resultImgNamePattern);
     ^~~~~~~~~~~~~~~
     readCameraTraj
getalignresults.cpp: At global scope:
getalignresults.cpp:1257:6: error: prototype for ‘bool getAlignResults::checkMeshMapImg(size_t, size_t, std::vector<cv::Point_<int> >&, float&)’ does not match any in class ‘getAlignResults’
 bool getAlignResults::checkMeshMapImg(size_t mesh_i, size_t img_i, std::vector<cv::Point2i> &v_uv, float &score)
      ^~~~~~~~~~~~~~~
In file included from getalignresults.cpp:1:0:
getalignresults.h:123:10: error: candidate is: bool getAlignResults::checkMeshMapImg(size_t, size_t, std::vector<cv::Point_<int> >&)
     bool checkMeshMapImg(size_t mesh_i, size_t img_i, std::vector<cv::Point2i> &v_uv);

More Discussion

@RatTac
I think it's necessary to make a new issue to test if you can get the notification...

I am quite curious about your results with the method that generating a depth map for every image. Could you push some images here?

I think I am still wrong with the remapping step. As the mesh are tiny on a image's resolution, many holes exist for the round(pixel.x) and round(pixel.y).

Here are results with only 3 input images, index of 0,4,6.

By merging all images to View i with weights (blurry results) :

By remapping others to View i with weights (many small black holes are visible):

After iteration of my code with alpha=1, lamda=2 (too much wraps are generated), the new Textures, and they are what the texture is on the model :

Try this program on stereo-matching based depths

Hi,
Thanks for your excellent recurrence. I wanna try this program on stereo-matching based depths. But I don't know the following parameters means. Following in first line red marked, does it mean this is a relative pose between camera 0 and camera 1? Are next 2-5 lines quaternion matrix or some other format parameters. And furthermore, do you have any suggestions that what I should care if I use this program on stereo-matching based depths, maybe it's not easy to get a enough good result on stereo-matching based depths and I should do some filtering on depth maps? I'm new in this field, please don't mind for any bad requests and questions.

Thank you very much.

Run the project without depth image

Dear author,

I want to run the project without depth image , just with obj file and images and camera pose. With these information, can I reconstruct the texture ?

Thanks so much!