A collection of sketch based applications.
Feel free to create a PR or an issue.
Outline
- 0. Survey
- 1. Sketch Based Image Synthesis
- 2. Sketch Based Image Editing
- 3. Sketch Based Image Retrieval (SBIR)
- 4. Sketch Based 3D Shape Retrieval
- 5. Sketch Based 3D Shape Modeling
- 6. Sketch Based Garment Design
- 7. Sketch Completion
- 8. Sketch Restoration, Retargeting and Inpainting
- 9. Sketch and Shading
- 10. Sketch Enhancement / Beautification
- 11. Sketch-Guided Object Localization
- 12. Sketch-Guided Video Synthesis
- 13. Sketch Recognition
- 14. Sketch Segmentation and Perceptual Grouping
- 15. Sketch Representation Learning
- 16. Sketch and Visual Correspondence
- 17. Sketch Animation/Inbetweening
- 18. Sketch and AR/VR
- 19. Sketch Based Incremental Learning
- 20. Sketch Quality Measurement
- 21. Cloud Augmentation with Sketches
- 22. Sketch and Re-identification
- 23. Sketch-based Salient Object Detection
- 24. Study on Strokes
Paper | Source | Code/Project Link |
---|---|---|
Deep Learning for Free-Hand Sketch: A Survey | TPAMI 2022 | [code] |
- Natural Image or Object
- Human Face / Portrait / Body
- 3D image
Paper | Source | Code/Project Link |
---|---|---|
3D-aware Conditional Image Synthesis | CVPR 2023 | [project] [code] |
- Arbitrary Image
Paper | Source | Code/Project Link |
---|---|---|
Free-Form Image Inpainting with Gated Convolution (DeepFill v2) | ICCV 2019 | [project] [code] |
Fashion Editing with Multi-scale Attention Normalization | 1906.00884 | |
DeFLOCNet: Deep Image Editing via Flexible Low-level Controls | CVPR 2021 | [code] |
DeepSIM: Image Shape Manipulation from a Single Augmented Training Sample | ICCV 2021 | [code] [project] |
SketchEdit: Mask-Free Local Image Manipulation with Partial Sketches | CVPR 2022 | [code] [project] |
Draw2Edit: Mask-Free Sketch-Guided Image Manipulation | ACM MM 2023 | [code] |
Towards Interactive Image Inpainting via Robust Sketch Refinement | TMM 2024 | [code] [project] |
- Human Face / Portrait / Hair
- Anime Editing
Paper | Source | Code/Project Link |
---|---|---|
AniFaceDrawing: Anime Portrait Exploration during Your Sketching | SIGGRAPH 2023 | [project] |
- Video Editing
Paper | Source | Code/Project Link |
---|---|---|
DeepFaceVideoEditing: Sketch-based Deep Editing of Face Videos | SIGGRAPH 2022 | [project] |
- Object-level
- Scene-level
- Video Retrieval
Paper | Source | Code/Project Link |
---|---|---|
Fine-Grained Instance-Level Sketch-Based Video Retrieval | TCSVT 2020 | |
Fine-Grained Video Retrieval with Scene Sketches | TIP 2023 | [project] |
- Free-hand sketch
- Scene Generation
Paper | Source | Code/Project Link |
---|---|---|
GroundUp: Rapid Sketch-Based 3D City Massing | ECCV 2024 | [code] |
Sketch2Scene: Automatic Generation of Interactive 3D Game Scenes from User's Casual Sketches | arxiv 24.08 | [webpage] |
- NeRF
Paper | Source | Code/Project Link |
---|---|---|
Sketch2NeRF: Multi-view Sketch-guided Text-to-3D Generation | arxiv 24.01 | |
SKED: Sketch-guided Text-based 3D Editing | ICCV 2023 | [project] [code] |
SketchDream: Sketch-based Text-to-3D Generation and Editing | SIGGRAPH 2024 | [project] |
- CAD sketch
Paper | Source | Code/Project Link |
---|---|---|
Lifting Freehand Concept Sketches into 3D | SIGGRAPH Asia 2020 | [project] [code] |
Sketch2CAD: Sequential CAD Modeling by Sketching in Context | SIGGRAPH Asia 2020 | [project] [code] |
Free2CAD: Parsing Freehand Drawings into CAD Commands | SIGGRAPH 2022 | [project] [code] |
Symmetry-driven 3D Reconstruction From Concept Sketches | SIGGRAPH 2022 | [project] |
Piecewise-smooth Surface Fitting Onto Unstructured 3D Sketches | SIGGRAPH 2022 | [project] |
Reconstruction of Machine-Made Shapes from Bitmap Sketches | SIGGRAPH Asia 2023 | [project] |
CAD-SIGNet: CAD Language Inference from Point Clouds using Layer-wise Sketch Instance Guided Attention | CVPR 2024 | |
SfmCAD: Unsupervised CAD Reconstruction by Learning Sketch-based Feature Modeling Operations | CVPR 2024 | [code] |
- Sketch completion
- Sketch gap / connectivity detection
Paper | Source | Code/Project Link |
---|---|---|
Joint Gap Detection and Inpainting of Line Drawings | CVPR 2017 | [project] [code] |
Detecting Viewer-Perceived Intended Vector Sketch Connectivity | SIGGRAPH 2022 | [project] |
- Manga
Paper | Source | Code/Project Link |
---|---|---|
Exploiting Aliasing for Manga Restoration | CVPR 2021 | [webpage] [code] |
Seamless Manga Inpainting with Semantics Awareness | SIGGRAPH 2021 | [webpage] [code] |
Screentone-Preserved Manga Retargeting | arxiv 22.03 | |
Manga Rescreening with Interpretable Screentone Representation | arxiv 23.06 |
- Hand-Drawn Drawings
Paper | Source | Code/Project Link |
---|---|---|
Restoration of Hand-Drawn Architectural Drawings using Latent Space Mapping with Degradation Generator | CVPR 2023 |
Paper | Source | Code/Project Link |
---|---|---|
Learning to Shadow Hand-drawn Sketches | CVPR 2020 | [project] [code] |
SmartShadow: Artistic Shadow Drawing Tool for Line Drawings | ICCV 2021 | [project] |
- Sketch Enhancement
Paper | Source | Code/Project Link |
---|---|---|
SketchMan: Learning to Create Professional Sketches | ACM MM 2020 | [code] |
- Sketch Beautification
Paper | Source | Code/Project Link |
---|---|---|
Sketch Beautification: Learning Part Beautification and Structure Refinement for Sketches of Man-made Objects | TVCG 2024 | |
PICASSO: A Feed-Forward Framework for Parametric Inference of CAD Sketches via Rendering Self-Supervision | arxiv 24.07 |
- Super Resolution
Paper | Source | Code/Project Link |
---|---|---|
APISR: Anime Production Inspired Real-World Anime Super-Resolution | CVPR 2024 | [code] |
Paper | Source | Code/Project Link |
---|---|---|
Deep Sketch-guided Cartoon Video Inbetweening | TVCG 2021 | [code] |
ToonCrafter: Generative Cartoon Interpolation | arxiv 24.05 | [code] [webpage] |
- Semantic / Instance Segmentation
- Perceptual Grouping
Type | Paper | Source | Code/Project Link |
Stroke-level | Sketching by perceptual grouping | ICIP 2013 | |
Making Better Use of Edges via Perceptual Grouping | CVPR 2015 | ||
Universal Sketch Perceptual Grouping / Toward Deep Universal Sketch Perceptual Grouper | ECCV 2018 / TIP 2019 | [code] |
- FloorPlan
- Stroke order importance/saliency, sketch abstraction
- Conventional Representation Learning
Paper | Source | Code/Project Link | Data Manner | Related Applications |
---|---|---|---|---|
A Neural Representation of Sketch Drawings (Sketch-RNN) | ICLR 2018 | [code] | Vector/stroke-level | Reconstruction and interpolation |
SketchHealer: A Graph-to-Sequence Network for Recreating Partial Human Sketches | BMVC 2020 | [code] | Vector/stroke-level | Sketch recognition, retrieval, completion and analogy |
Sketchformer: Transformer-based Representation for Sketched Structure | CVPR 2020 | [code] | Vector/stroke-level | Sketch classification, SBIR, reconstruction and interpolation |
CoSE: Compositional Stroke Embeddings | NeurIPS 2020 | [Code] | Vector/stroke-level | Auto-completing diagrams |
SketchLattice: Latticed Representation for Sketch Manipulation | ICCV 2021 | [code] | Lattice graph | Sketch healing and image-to-sketch synthesis |
SketchODE: Learning neural sketch representation in continuous time | ICLR 2022 | [Project] | Vector/Stroke-level | Continuous-time representation, reconstruction & interpolation |
Linking Sketch Patches by Learning Synonymous Proximity for Graphic Sketch Representation | AAAI 2023 | [code] | Vector/Stroke-level | Sketch synthesis and sketch healing |
SketchKnitter: Vectorized Sketch Generation with Diffusion Models | ICLR 2023 | [code] | Vector/Stroke-level | vectorized sketch generation |
ChiroDiff: Modelling chirographic data with Diffusion Models | ICLR 2023 | [Project] | Vector/Stroke-level | vectorization, de-noising/healing, abstraction |
Enhance Sketch Recognition's Explainability via Semantic Component-Level Parsing | AAAI 2024 | [code] | Vector/Stroke-level | recognition and segmentation |
Modelling complex vector drawings with stroke-clouds | ICLR 2024 | [code] | Vector/Stroke-level | reconstruction and generation |
Sketchformer++: A Hierarchical Transformer Architecture for Vector Sketch Representation | CVM 2024 | Vector/Stroke-level | reconstruction, recognition and semantic segmentation |
Paper | Source | Code/Project Link | Data Manner | Related Applications |
---|---|---|---|---|
A Learned Representation for Scalable Vector Graphics | ICCV 2019 | [code] | SVG | Font design |
DeepSVG: A Hierarchical Generative Network for Vector Graphics Animation | NeurIPS 2020 | [Code] [Project] | SVG | Vector Graphics Animation, reconstruction and interpolation |
SVGformer: Representation Learning for Continuous Vector Graphics using Transformers | CVPR 2023 | SVG | Reconstruction, classification, interpolation, retrieval | |
StrokeNUWA: Tokenizing Strokes for Vector Graphic Synthesis | arxiv 24.01 | SVG | generation |
- Self-supervised or few/zero-shot learning
Paper | Source | Code/Project Link | Data Manner | Related Applications |
---|---|---|---|---|
Sketch Fewer to Recognize More by Learning a Co-Regularized Sparse Representation | TCSVT 2019 | few-shot classification | ||
On Learning Semantic Representations for Large-Scale Abstract Sketches | TCSVT 2020 | [Code] | Bitmap and Vector | Retrieval and recognition |
Deep Self-Supervised Representation Learning for Free-Hand Sketch | TCSVT 2020 | [Code] | Vector/stroke-level | Retrieval and recognition |
Sketch-BERT: Learning Sketch Bidirectional Encoder Representation from Transformers by Self-supervised Learning of Sketch Gestalt | CVPR 2020 | [Code] | Vector/stroke-level | Sketch recognition, retrieval, and gestalt |
Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting | CVPR 2021 | [Code] | Both Vector and Raster-level | Recognition |
Self-Supervised Learning for Enhancing Spatial Awareness in Free-Hand Sketches | IJCAI 2024 | [Code] | Vector-level | Synthesis |
- Scene-level
Paper | Source | Code/Project Link |
---|---|---|
SceneTrilogy: On Human Scene-Sketch and its Complementarity with Photo and Text | CVPR 2023 |
- Implicit Neural Representations
Paper | Source | Code/Project Link |
---|---|---|
SketchINR: A First Look into Sketches as Implicit Neural Representations | CVPR 2024 |
- Datasets
Name | Paper | Source | Code/Project Link |
---|---|---|---|
CreativeFlow+ | Creative Flow+ Dataset | CVPR 2019 | [code] |
ATD-12K | Deep animation video interpolation in the wild | CVPR 2021 | [code] |
AnimeRun | AnimeRun: 2D Animation Correspondence from Open Source 3D Movies | NeurIPS 2022 | [code] |
PaintBucket-Character | Learning Inclusion Matching for Animation Paint Bucket Colorization | CVPR 2024 | [code] |
- Methods
Matching Type | Paper | Source | Code/Project Link |
---|---|---|---|
Region | Globally optimal toon tracking | SIGGRAPH 2016 | [project] |
Region | The Animation Transformer: Visual Correspondence via Segment Matching | ICCV 2021 | |
Region | Learning Inclusion Matching for Animation Paint Bucket Colorization | CVPR 2024 | [code] [project] |
Point | SketchDesc: Learning Local Sketch Descriptors for Multi-view Correspondence | TCSVT 2020 | |
Point | SketchZooms: Deep Multi-view Descriptors for Matching Line Drawings | CGF 2021 | [project] |
Stroke | Differential Operators on Sketches via Alpha Contours | SIGGRAPH 2023 | [code] |
- Inbetweening
Paper | Source | Representation | Code/Project Link |
---|---|---|---|
BetweenIT: An Interactive Tool for Tight Inbetweening | CGF 2010 | stroke | |
Context-Aware Computer Aided Inbetweening | TVCG 2017 | stroke | |
FTP-SC: Fuzzy Topology Preserving Stroke Correspondence | SCA 2018 | stroke | [webpage] [video] |
Cacani: 2d animation and inbetween software | / | stroke | [software] |
Stroke-Based Drawing and Inbetweening with Boundary Strokes | CGF 2021 | stroke | |
Joint Stroke Tracing and Correspondence for 2D Animation | TOG 2024 | stroke | [code] [webpage] |
Deep Geometrized Cartoon Line Inbetweening | ICCV 2023 | vertex/point | [code] |
Optical Flow Based Line Drawing Frame Interpolation Using Distance Transform to Support Inbetweenings | ICIP 2019 | raster | |
Bridging the Gap: Sketch-Aware Interpolation Network for High-Quality Animation Sketch Inbetweening | ACM MM 2024 | raster | |
ToonCrafter: Generative Cartoon Interpolation | arxiv 24.05 | raster | [code] [webpage] |
- Animation
Paper | Source | Code/Project Link |
---|---|---|
SweepCanvas: Sketch-based 3D Prototyping on an RGB-D Image | UIST 2017 | [video] |
Model-Guided 3D Sketching | TVCG 2018 | [video] |
Mobi3DSketch: 3D Sketching in Mobile AR | CHI 2019 | [video] |
Interactive Liquid Splash Modeling by User Sketches | SIGGRAPH Asia 2020 | [video] |
Towards 3D VR-Sketch to 3D Shape Retrieval | 3DV 2020 | [code] [project] |
3D Curve Creation on and around Physical Objects with Mobile AR | TVCG 2021 | [video] |
HandPainter - 3D Sketching in VR with Hand-based Physical Proxy | CHI 2021 | [video] |
Fine-Grained VR Sketching: Dataset and Insights | 3DV 2021 | [code] |
Structure-Aware 3D VR Sketch to 3D Shape Retrieval | 3DV 2022 | [code] |
GestureSurface: VR Sketching through Assembling Scaffold Surface with Non-Dominant Hand | TVCG 2023 | |
3D VR Sketch Guided 3D Shape Prototyping and Exploration | ICCV 2023 | [code] |
Paper | Source | Code/Project Link |
---|---|---|
Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches | CVPR 2022 | [code] |
Paper | Source | Code/Project Link |
---|---|---|
Annotation-Free Human Sketch Quality Assessment | IJCV 2024 | [code] |
Finding Badly Drawn Bunnies | CVPR 2022 | [code] |
Paper | Source | Code/Project Link |
---|---|---|
Cloud2Sketch: Augmenting Clouds with Imaginary Sketches | ACM MM 2022 | [project] |
Paper | Source | Code/Project Link |
---|---|---|
Cross-Compatible Embedding and Semantic Consistent Feature Construction for Sketch Re-identification | ACM MM 2022 | [code] |
SketchTrans: Disentangled Prototype Learning with Transformer for Sketch-Photo Recognition | TPAMI 2023 | [code] |
Paper | Source | Code/Project Link |
---|---|---|
Sketch2Saliency: Learning to Detect Salient Objects from Human Drawings | CVPR 2023 |
Paper | Source | Code/Project Link |
---|---|---|
Neural Brushstroke Engine: Learning a Latent Style Space of Interactive Drawing Tools | SIGGRAPH Asia 2022 | [project] |
Ciallo: GPU-Accelerated Rendering of Vector Brush Strokes | SIGGRAPH 2024 | [code] |