happylun/sketchmodeling

Source code for the Sketch Modeling project: reconstruct a 3D shape from line drawing sketches.

3D Shape Reconstruction from Sketches via Multi-view Convolutional Networks

---

Project Page

Introduction

This archive contains source code for training / testing our algorithm for converting 2D sketches into 3D shapes. The code base contains two parts: the Network Part and the Fusion Part.

Network Part

The network part contains Python code for predicting depth and normal maps from input sketch images. The code uses TensorFlow framework to train and test the deep neural networks. Please read the README file within the Network folder for more details.

Fusion Part

The fusion part contains C++ code for fusing depth and normal maps to 3D shapes. Visual Studio is required to compile and run our code. Please read the README file within the Fusion folder for more details.

Other Notes

• Our code is released under GPL v3 license.

• If you would like to use our code, please cite the following paper:

  Zhaoliang Lun, Matheus Gadelha, Evangelos Kalogerakis, Subhransu Maji, Rui Wang, "3D Shape Reconstruction from Sketches via Multi-view Convolutional Networks", Proceedings of the International Conference on 3D Vision (3DV) 2017

• For any questions or comments, please contact Zhaoliang Lun ([email protected])

sketchmodeling's Issues

Error using 3 points of view

I have an sketch with 3 points of view... so i used --sketch_views SFT to use all of them, but i end up with this error

InvalidArgumentError (see above for traceback): Assign requires shapes of both tensors to match. lhs shape= [4,4,6,64] rhs shape= [4,4,4,64]
	 [[Node: save/Assign_393 = Assign[T=DT_FLOAT, _class=["loc:@monnet/G_net/e1/weights"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](monnet/G_net/e1/weights, save/RestoreV2:393)]]

Do i have to train the MonterNet again to use 3 points of view ?

Full traceback:

start running...
Loading testing data...
Building network...
2018-06-04 10:00:04.261194: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
Testing...
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1322, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [4,4,6,64] rhs shape= [4,4,4,64]
	 [[Node: save/Assign_393 = Assign[T=DT_FLOAT, _class=["loc:@monnet/G_net/e1/weights"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](monnet/G_net/e1/weights, save/RestoreV2:393)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 153, in <module>
    tf.app.run()
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/platform/app.py", line 126, in run
    _sys.exit(main(argv))
  File "main.py", line 142, in main
    monnet.test(sess, views, num_test_shapes)
  File "/home/cadar/Documents/GitHub/SketchModeling/Network/code/MonsterNet/monnet.py", line 445, in test
    self.saver.restore(sess, ckpt.model_checkpoint_path)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 1802, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 900, in run
    run_metadata_ptr)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1135, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1316, in _do_run
    run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1335, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [4,4,6,64] rhs shape= [4,4,4,64]
	 [[Node: save/Assign_393 = Assign[T=DT_FLOAT, _class=["loc:@monnet/G_net/e1/weights"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](monnet/G_net/e1/weights, save/RestoreV2:393)]]

Caused by op 'save/Assign_393', defined at:
  File "main.py", line 153, in <module>
    tf.app.run()
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/platform/app.py", line 126, in run
    _sys.exit(main(argv))
  File "main.py", line 142, in main
    monnet.test(sess, views, num_test_shapes)
  File "/home/cadar/Documents/GitHub/SketchModeling/Network/code/MonsterNet/monnet.py", line 442, in test
    self.saver = tf.train.Saver()
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 1338, in __init__
    self.build()
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 1347, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 1384, in _build
    build_save=build_save, build_restore=build_restore)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 835, in _build_internal
    restore_sequentially, reshape)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 494, in _AddRestoreOps
    assign_ops.append(saveable.restore(saveable_tensors, shapes))
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 185, in restore
    self.op.get_shape().is_fully_defined())
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/state_ops.py", line 283, in assign
    validate_shape=validate_shape)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_state_ops.py", line 60, in assign
    use_locking=use_locking, name=name)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 3392, in create_op
    op_def=op_def)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1718, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

ReadAccessViolation error

When I run the fusion part of the program, it throw up a ReadAccessViolation error. This leads me to believe that a string is being passed without reference somewhere in the code, I have followed the instructions in the ReadMe docs, and the only thing I have changed is setting the 'ReconstructMesh' part of the solution to the StartUp project in visual studio. Has anyone encountered this problem before, and if so, how would I go about fixing it?

Running on my own depth maps

Hi @happylun I have tested the model on my own depth maps, but resulted nothing. I am using this image

I have found the depths in your dataset are in 3-color-channel.

Can you forward the code for conversion of my depth image into your format.

Thanks in advance!!

Could you release the methods which you used to render ground truth normal maps and depth ?

Hi, I'm recently reading your papers. Very great job.
I'm curious about the way you get your ground truth images in your training dataset.
Could you release the methods which you used to render ground truth normal maps and depth ?
As well as, the way your fuse these 2 kinds of images.
Thanks a lot!

Fusion part

Hi @happylun, I am testing this code using another dataset. I have completed the Network part. I have doubt in the Fusion part.

ReconstructMesh.exe 1 FS
./CharacterDraw/hires/m1/
./CharacterDraw/output/images/m1/
./CharacterDraw/output/reconstruct/m1/
./CharacterDraw/view/view.off
What do each bold line indicate? I have assumed and replaced them with my dataset accordingly. But cannot understand the second line.
ReconstructMesh.exe 1 FS
**./SketchModeling/zhaoliang_lun_trainingdata/Airplane/sketch/02691156/1a04e3eab45ca15dd86060f189eb133

./SketchModeling/zhaoliang_lun_trainingdata/Airplane/dn/02691156/1a04e3eab45ca15dd86060f189eb133
./SketchModeling/zhaoliang_lun_trainingdata/Airplane/view/view.off**

Please guide. Thanks!!

The method of rendering the ground truth of normal/depth map

Hello, I have used your TestingData to predict normal/depth maps and fused them together to make mesh model. Absolutely, it works well.
Now, I am trying to training the network with my own input and ground truth(normal/depth map). I used blender to render the ground truth, but the result is different with your dataset(as below pictures). What's more, they makes no sense in the fusing part(as below pictures).
I'd like to ask you about the method of your rendering the groundtruth. Or can you share me the type or criteria(to some degree, what I mean is the direction of every color meaning) of your normal graph and I am going to map my normal pictures to yours.
Extremely thank you!!!

The testing data files can not be opened

The URL"Download testing data files here" can not be opened!
"The requested URL /~zlun/papers/SketchModeling/TestingData.7z was not found on this server."

running the reconstruction part with my own depth/normal maps

Hi, I am trying to use the mesh reconstruction from depth/normal maps part of your pipeline but with my own depth and normal maps, but I keep getting image format error in the C++ code (wrong byte per pixel amount) how do I have to save images for them to be computed by your C++ code?

Thank you

Generating training sketchs

Hi @happylun Can you please share the code for creation of synthetic line drawings.

problem in loadSketch (vector subscript out of range)

After some time i finnaly fixed issue with "vector subscript out of range", maybe it can helped anyone. If you enter parametrs witch not ended "/" slash - you got exception: "vector subscript out of range". Just add slash on end of path and all will be fine!

Dataset

Hi @happylun Can you please share the Training dataset link? The link you shared in README is not working.

开始迭代后，loss打印nan值，CUDA_ERROR_OUT_OF_MEMORY

环境： win11 + tensorflow-1.14.0

请问这个是正常的嘛，试着跑了几个epoch，validate输出的depth map都是没内容的。。。

Error in testing part

@happylun I have installed libraries according to your README. I get this error when I test the main.py on chair data. I have tried some attempts for solving this error but those doesn't work. Please help.

(SKetchModelling) C:\GenAssi\SketchModeling\Network\code\MonsterNet>python main.py --test --data_dir C:\GenAssi\SketchModeling\zhaoliang_lun_trainingdata\Chair\ --train_dir C:\GenAssi\SketchModeling\Checkpoint\Chair\ --test_dir C:\GenAssi\SketchModeling\chair_results
C:\Users\dhornala.bharadwaj\AppData\Local\Continuum\anaconda3\envs\SKetchModelling\lib\site-packages\tensorflow\python\framework\dtypes.py:455: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
C:\Users\dhornala.bharadwaj\AppData\Local\Continuum\anaconda3\envs\SKetchModelling\lib\site-packages\tensorflow\python\framework\dtypes.py:456: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
C:\Users\dhornala.bharadwaj\AppData\Local\Continuum\anaconda3\envs\SKetchModelling\lib\site-packages\tensorflow\python\framework\dtypes.py:457: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
C:\Users\dhornala.bharadwaj\AppData\Local\Continuum\anaconda3\envs\SKetchModelling\lib\site-packages\tensorflow\python\framework\dtypes.py:458: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
C:\Users\dhornala.bharadwaj\AppData\Local\Continuum\anaconda3\envs\SKetchModelling\lib\site-packages\tensorflow\python\framework\dtypes.py:459: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
start running...
Loading testing data...
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "BestSplits" device_type: "CPU"') for unknown op: BestSplits
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "CountExtremelyRandomStats" device_type: "CPU"') for unknown op: CountExtremelyRandomStats
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "FinishedNodes" device_type: "CPU"') for unknown op: FinishedNodes
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "GrowTree" device_type: "CPU"') for unknown op: GrowTree
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "ReinterpretStringToFloat" device_type: "CPU"') for unknown op: ReinterpretStringToFloat
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "SampleInputs" device_type: "CPU"') for unknown op: SampleInputs
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "ScatterAddNdim" device_type: "CPU"') for unknown op: ScatterAddNdim
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "TopNInsert" device_type: "CPU"') for unknown op: TopNInsert
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "TopNRemove" device_type: "CPU"') for unknown op: TopNRemove
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "TreePredictions" device_type: "CPU"') for unknown op: TreePredictions
E c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:943] OpKernel ('op: "UpdateFertileSlots" device_type: "CPU"') for unknown op: UpdateFertileSlots
Building network...
Testing...

_W c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:993] Not found: Can not get size for: C:\GenAssi\SketchModeling\zhaoliang_lun_trainingdata\Chair\sketch/03001627/264322794651490ec0d3c02f7e255b2b/sketch-S-0.png : The system cannot find the file specified.

W c:\tf_jenkins\home\workspace\release-win\device\cpu\os\windows\tensorflow\core\framework\op_kernel.cc:993] Not found: Can not get size for: C:\GenAssi\SketchModeling\zhaoliang_lun_trainingdata\Chair\sketch/03001627/264322794651490ec0d3c02f7e255b2b/sketch-F-0.png : The system cannot find the file specified._

Traceback (most recent call last):
File "C:\Users\dhornala.bharadwaj\AppData\Local\Continuum\anaconda3\envs\SKetchModelling\lib\site-packages\tensorflow\python\client\session.py", line 1022, in _do_call
return fn(*args)
File "C:\Users\dhornala.bharadwaj\AppData\Local\Continuum\anaconda3\envs\SKetchModelling\lib\site-packages\tensorflow\python\client\session.py", line 1004, in _run_fn
status, run_metadata)
File "C:\Users\dhornala.bharadwaj\AppData\Local\Continuum\anaconda3\envs\SKetchModelling\lib\contextlib.py", line 66, in exit
next(self.gen)
File "C:\Users\dhornala.bharadwaj\AppData\Local\Continuum\anaconda3\envs\SKetchModelling\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 466, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.OutOfRangeError: FIFOQueue '_1_batch/fifo_queue' is closed and has insufficient elements (requested 2, current size 0)
[[Node: batch = QueueDequeueManyV2[component_types=[DT_STRING, DT_FLOAT, DT_FLOAT, DT_BOOL, DT_FLOAT], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](batch/fifo_queue, batch/n)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "main.py", line 153, in
tf.app.run()
File "C:\Users\dhornala.bharadwaj\AppData\Local\Continuum\anaconda3\envs\SKetchModelling\lib\site-packages\tensorflow\python\platform\app.py", line 44, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "main.py", line 142, in main
monnet.test(sess, views, num_test_shapes)
File "C:\GenAssi\SketchModeling\Network\code\MonsterNet\monnet.py", line 471, in test
names,results,errors,images = sess.run([self.names, self.results, self.errors, self.pngs])
File "C:\Users\dhornala.bharadwaj\AppData\Local\Continuum\anaconda3\envs\SKetchModelling\lib\site-packages\tensorflow\python\client\session.py", line 767, in run
run_metadata_ptr)
File "C:\Users\dhornala.bharadwaj\AppData\Local\Continuum\anaconda3\envs\SKetchModelling\lib\site-packages\tensorflow\python\client\session.py", line 965, in _run
feed_dict_string, options, run_metadata)
File "C:\Users\dhornala.bharadwaj\AppData\Local\Continuum\anaconda3\envs\SKetchModelling\lib\site-packages\tensorflow\python\client\session.py", line 1015, in _do_run
target_list, options, run_metadata)
File "C:\Users\dhornala.bharadwaj\AppData\Local\Continuum\anaconda3\envs\SKetchModelling\lib\site-packages\tensorflow\python\client\session.py", line 1035, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: FIFOQueue '_1_batch/fifo_queue' is closed and has insufficient elements (requested 2, current size 0)
[[Node: batch = QueueDequeueManyV2[component_types=[DT_STRING, DT_FLOAT, DT_FLOAT, DT_BOOL, DT_FLOAT], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](batch/fifo_queue, batch/n)]]

Run my own sketch

I would like to run the pipeline on my own airplanes sketches, but I couldn't find any documentation about how to do so, specially in how to setup my view.off file.

Could you release your method for creating synthetic line drawings of 3D shapes

HI @happylun

Thank you for your amazing work!
I would like to use more categories of 3D shapes to get through this but I can't find the code to create line drawings.
Looking forward to your reply.

Doubt in DATA_DIR

I have installed all the dependencies and downloaded train and test datasets required for the code execution. Can you provide any sample for this statement.
python main.py --test --data_dir [DATA_DIR] --train_dir [TRAINING_DIR] --test_dir [TESTING_DIR]
which can help my execution easier.

output views is 12 in paper while it turns out to be 14 in your code

Hi @happylun

Nice work!

After reading your code and your paper, I have one confusion:

   In your paper output views is 12, but in your code, it turns out to be 14, why is it not the same?

Could you explain it? Thank you!

Training gets killed

Hi, I have a problem re-training the network, after few iterations, the process get killed. I've tried many fixes found on the net (for example: tensorflow/tensorflow#5289) like decreasing batch_size (which is already at 2) but nothing solves it, I still have the 'killed' problem.

I train the network on a Tesla K40c with 12GB of RAM with python 3.5, NumPy 1.12.1,SciPy 0.19.0
and TensorFlow 1.0.1, the asked requirements.

Does anybody have an idea to solve this?

Thank you