val-iisc/sketch-parse

Code, demos and data for SketchParse (a neural network for sketch segmentation). Paper:

sketch-parse

This repository contains code and additional material for our paper titled "SketchParse: Towards Rich Descriptions For Poorly Drawn Sketches Using Multi-Task Deep Networks", ACM Multimedia (ACMMM) 2017.

This is a neural network for (semantic) sketch segmentation. Use it to associate semantics with your freehand sketches!

The four panels are chosen from the 100th, 75th, 50th, 25th percentile accuracy of segmentation (by IoU) respectively. As you can see, even in (relatively) bad cases, we can provide fairly accurate segmentations!

Model

We have a multi-task deep neural network that can segment freehand sketches as well as predict a global pose:

Contents

• For code and corresponding instructions, navigate to exp-src

• For annotation software tool and corresponding instructions, navigate to valsketch

• For retrieval code, models and a demo, navigate to retrieval-src

• For instructions to get annotated sketch dataset, navigate to exp-src/data/sketch-dataset. Pose dataset is present in exp-src/data/lists/Pose_all_label.txt. Find instruction regarding pose dataset here.

Requirements

This code was developed and tested on an Ubuntu 14.04 machine with python 2.7 and pyTorch (v0.1.12). We used an NVIDIA TITAN X for training and evaluating our model.

If you use this work, please cite the paper:

SketchParse: Towards Rich Descriptions For Poorly Drawn Sketches Using Multi-Task Deep Networks

Questions ?

• For questions regarding the main segmentation network, please contact Isht ([email protected])

• For questions regarding the annotation tool, please contact Sahil ([email protected])

• For questions regarding the pose subnetwork and the sketch-based image retrieval application, please contact Abhijat ([email protected])

• For any other questions, please contact Ravi ([email protected])

sketch-parse's Issues

Pretrained model

The link for the pretrained model given here: https://github.com/val-iisc/sketch-parse/tree/master/exp-src gives a 404 error,
can you pls provide an updated link for the pretrained model?

Requesting for Augmentaion data tool

Hi,
Could please update tool for Augmentation data in this project?
Thanks

the images in test_GT.zip are black in view

Requesting for Caffe folder

Hi, when i run "table1.py" with my caffe folder, it got this error "ImportError: /home/vipgpu/trang/caffe/python/caffe/../../build/lib/libcaffe.so.1.0.0: undefined symbol:_ZN5caffe20hdf5_load_nd_datasetIdEEvlPKciiPNS_4BlobIT_EEb"
Can you upload your your caffe folder which you used in this project?

Thanks

'unexpected key "pose_r0.PoseC1.0.weight" in state_dict'

Hi, I tried to run "Retrieval-demo.ipynb" in 'retrieval-src" folder to get segment for sketch. But i dunno why it got this error "unexpected key "pose_r0.PoseC1.0.weight" in state_dict" when loading model using this command: "model.load_state_dict(saved_state_dict)".
Im sure that i downloaded "model_r5_p50x_D1_17000.pth" file and it is in correct folder.
Do you have any suggestion?
Thanks

Which model is the one that get the best performance?

Hi,

As you have uploaded too much models, I cannot tell which one is the model with the best performance. Can you tell me which one is the best? I think it should corresponds to the one named "BCP-R5" (63.17%) in Table 6 of your paper.

Thank you very much.

Train model with new data

Hi,
I want to train model again with my data. My data is sketch of human object with 14 parts segmentation (hair, face, arm, leg,shoes, dress, skirt...). could you give me some suggestions to get good result for this new data? so far my result is not good.
This is example of my sketch and segmentation of human object:


Thanks