mukosame/anime2sketch

A sketch extractor for anime/illustration.

Anime2Sketch

Anime2Sketch: A sketch extractor for illustration, anime art, manga

By Xiaoyu Xiang

Updates

• 2022.1.14: Add Docker environment by kitoria
• 2021.12.25: Update README. Merry Christmas!
• 2021.5.24: Fix an interpolation error and a GPU inference error.
• 2021.5.12: Web Demo by AK391
• 2021.5.2: Upload more example results of anime video.
• 2021.4.30: Upload the test scripts. Now our repo is ready to run!
• 2021.4.11: Upload the pretrained weights, and more test results.
• 2021.4.8: Create the repo.

Introduction

The repository contains the testing codes and pretrained weights for Anime2Sketch.

Anime2Sketch is a sketch extractor that works well on illustration, anime art, and manga. It is an application based on the paper "Adversarial Open Domain Adaption for Sketch-to-Photo Synthesis".

Prerequisites

• Linux, macOS, Docker
• Python 3 (Recommend to use Anaconda)
• CPU or NVIDIA GPU + CUDA CuDNN
• Pillow, PyTorch

Get Started

Installation

Install the required packages: pip install -r requirements.txt

Download Pretrained Weights

Please download the weights from GoogleDrive, and put it into the weights/ folder.

We also have an artifact-free version of the model which works with dark / low contrast images. You can download the weights from GoogleDrive, and put it into weights/ folder.

Test

python3 test.py --dataroot /your_input/dir --load_size 512 --output_dir /your_output/dir

The above command includes three arguments:

• dataroot: your test file or directory
• load_size: due to the memory limit, we need to resize the input image before processing. By default, we resize it to 512x512.
• output_dir: path of the output directory

Run our example:

python3 test.py --dataroot test_samples/madoka.jpg --load_size 512 --output_dir results/

Docker

If you want to run on Docker, you can easily do so by customizing the input/output images directory.
Build docker image

make docker-build

Setting input/output directory
You can customize mount volumes for input/output images by Makefile. Please setting your target directory.

docker run -it --rm --gpus all -v `pwd`:/workspace -v {your_input_dir}:/input -v {your_output_dir}:/output anime2sketch

example:

docker run -it --rm --gpus all -v `pwd`:/workspace -v `pwd`/test_samples:/input -v `pwd`/output:/output anime2sketch

Run

make docker-run

if you want to run cpu only, you will need to fix two things (remove gpu options).

• Dockerfile CMD line to CMD [ "python", "test.py", "--dataroot", "/input", "--load_size", "512", "--output_dir", "/output" ]
• Makefile docker-run line to docker run -it --rm -v `pwd`:/workspace -v `pwd`/images/input:/input -v `pwd`/images/output:/output anime2sketch

Train

This project is a sub-branch of AODA. Please check it for the training instructions.

More Results

Our model works well on illustration arts: Turn handrawn photos to clean linearts: Simplify freehand sketches: And more anime results:

Contact

Xiaoyu Xiang.

You can also leave your questions as issues in the repository. I will be glad to answer them!

License

This project is released under the MIT License.

Citations

@misc{Anime2Sketch,
  author = {Xiaoyu Xiang, Ding Liu, Xiao Yang, Yiheng Zhu, Xiaohui Shen},
  title = {Anime2Sketch: A Sketch Extractor for Anime Arts with Deep Networks},
  year = {2021},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/Mukosame/Anime2Sketch}}
}

@inproceedings{xiang2022adversarial,
  title={Adversarial Open Domain Adaptation for Sketch-to-Photo Synthesis},
  author={Xiang, Xiaoyu and Liu, Ding and Yang, Xiao and Zhu, Yiheng and Shen, Xiaohui and Allebach, Jan P},
  booktitle={Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision},
  year={2022}
}

anime2sketch's Issues

ValueError: Expected more than 1 spatial element when training, got input size torch.Size([1, 512, 1, 1])

raise ValueError("Expected more than 1 spatial element when training, got input size {}".format(size))
ValueError: Expected more than 1 spatial element when training, got input size torch.Size([1, 512, 1, 1])

analytics-python 1.4.0
backoff 1.10.0
bcrypt 3.2.0
certifi 2021.10.8
cffi 1.15.0
charset-normalizer 2.0.8
click 8.0.3
colorama 0.4.4
cryptography 36.0.0
cycler 0.11.0
ffmpy 0.3.0
Flask 2.0.2
Flask-CacheBuster 1.0.0
Flask-Cors 3.0.10
Flask-Login 0.5.0
fonttools 4.28.2
gradio 2.4.6
idna 3.3
itsdangerous 2.0.1
Jinja2 3.0.3
kiwisolver 1.3.2
markdown2 2.4.1
MarkupSafe 2.0.1
matplotlib 3.5.0
monotonic 1.6
numpy 1.21.4
packaging 21.3
pandas 1.3.4
paramiko 2.8.1
pep517 0.12.0
Pillow 8.4.0
pip 21.3.1
pycparser 2.21
pycryptodome 3.11.0
pydub 0.25.1
PyNaCl 1.4.0
pyparsing 3.0.6
python-dateutil 2.8.2
pytz 2021.3
requests 2.26.0
setuptools 41.2.0
setuptools-scm 6.3.2
six 1.16.0
tomli 1.2.2
torch 1.10.0
torchtext 0.11.0
torchvision 0.11.1
tqdm 4.62.3
typing_extensions 4.0.0
urllib3 1.26.7
Werkzeug 2.0.2

ValueError: Unknown resampling filter (InterpolationMode.BICUBIC).

ValueError: Unknown resampling filter (InterpolationMode.BICUBIC). Use Image.Resampling.NEAREST (0), Image.Resampling.LANCZOS (1), Image.Resampling.BILINEAR (2), Image.Resampling.BICUBIC (3), Image.Resampling.BOX (4) or Image.Resampling.HAMMING (5)

Artifacts in result sketch

Hi

Thanks for your great work.

When I test on my own dataset, sometimes it will show some artifacts in darkness place.
Do you have any good suggestions to fix it?

How can we make it work on normal images?

Hi, I would like to convert normal face images to sketch. Would a model trained on ffhq dataset perform well with normal images? Have you tried something like this before?

Why dont you add a simple bat file for windows to go through all files in folder ?

heres what works for me and converts all files from in folder to out folder with 2048 tex quality :

@echo off
for %%F in (in*.*) do test.py --dataroot in --load_size 2048 --output_dir out/

how to train for my own dataset

Hi， thanks for your sharing , I want to train my own dataset using this network

I see in readme for training in AODA, but AODA repo also has nothing about how to train 。。。

Dataset for training

Hello, I love your research and output is so great

I'm just wondering how mach size of dataset are used for pretrained model of this Anime2sketch, through the original code
https://github.com/Mukosame/AODA

waiting for reply

Thank you

can't install

How can access the dataset for this project?

If possible, could you redirect me to the dataset for this project? @Mukosame @AK391

ValueError: Unknown resampling filter error

I'm using Windows 10 and python 3.9

PS C:\Users\Tamal\Desktop\Anime2Sketch> python3 test.py --dataroot image --load_size 512 --output_dir results
Traceback (most recent call last):
File "C:\Users\Tamal\Desktop\Anime2Sketch\test.py", line 46, in
save_image(aus_img, aus_path, aus_resize)
File "C:\Users\Tamal\Desktop\Anime2Sketch\data.py", line 96, in save_image
image_pil = image_pil.resize(output_resize, bic)
File "C:\Users\Tamal\AppData\Local\Programs\Python\Python39\lib\site-packages\PIL\Image.py", line 1957, in resize
raise ValueError(
ValueError: Unknown resampling filter (InterpolationMode.BICUBIC). Use Image.NEAREST (0), Image.LANCZOS (1), Image.BILINEAR (2), Image.BICUBIC (3), Image.BOX (4) or Image.HAMMING (5)
PS C:\Users\Tamal\Desktop\Anime2Sketch>

enhancement : add a dockerfile

could be useful to have a dockerfile that contains all the programs needed to run it. We could just mount a volume for the input/output.

how can launch web interface?

how can launch web interface?

ValueError: Unknown resampling filter

When I run test.py, I get a ValueError: Unknown resampling filter (InterpolationMode.BICUBIC) error in data.py.
To avoid errors
data.py line 96 image_pil = image_pil.resize (output_resize, bic) is
image_pil = image_pil.resize (output_resize, Image.BICUBIC) or
Shouldn't image_pil = image_pil.resize (output_resize, 3)?
This way the error no longer occurs.

大佬，请问一下权重文件在哪获取呀

大佬，请问一下权重文件在哪获取呀

可以使用这个模型用来训练普通图片到草图吗？

你好，可以使用这个模型用来训练普通图片到草图吗？我使用AODA进行了一些尝试，但是效果并不理想，我猜可能是我哪里做错了。
这是我的训练参数，希望能得到你的帮助，谢谢！

python train.py --dataroot ./dataset/sketch/ --name scribble_aoda  --model aoda_gan  --gan_mode vanilla --no_dropout --n_classes 9 --direction BtoA --save_epoch_freq 1 --load_size 260 --continue_train --gpu_ids 0 

Higher load_size leads to thinner stroke line ?

User Warning

I am getting the following error while trying the test code.

UserWarning: Argument interpolation should be of type InterpolationMode instead of int. Please, use InterpolationMode enum.
warnings.warn(

Made simple GUI script for it

Place it in folder where test.py is and run bat file (a2sgui.bat)
a2sgui.zip

pycharm 运行时报错

你好，我尝试用pycharm运行报错，可以麻烦看看嘛？

Where is the test code?

Can you provide the test code?

一个关于使用AODA训练这个模型时遇到的问题

请问一下作者大大，在使用AODA训练Anime2Sketch时是否有使用GC，是将类别数设置为了0还是1呢。

Artifact-free version of model

Overview

Right now, this model has issues in dark / low contrast regions #21. I've made a version of the model which significantly reduces the artifacts of this model without degrading performance on existing images.
Below are a couple of examples model outputs

Normal image

Image

Original

Improved

Dark image

Image

Original

Improved

Of course, the model outputs aren't perfect... but if the author of the repository or anyone is interested. Would be glad to write a PR or share methods, weights and code.

Error when using Gradio

Hello, just like the title said. I can't perform Anime2Sketch using the Gradio demo website and I got an error.

And I can't use the demo image from the website.

I don't know it is a bug or just my internet because I already try to delete the cookies and I still have that trouble.

Thank you

line extraction but with colored lines

Hi, thanks to great works.

tho found this project has been stalled for a year

can it do line extraction but with colored lines (mark) , train model on multi-colour lines pairs ?

In fact, in animation production, called 色トレス, need work that tracing the scanned paper drawing into digital

the tracing easy to coloring (compare to illustration making), tracing following coloring reference to standardize lines colour, like solid line, shadow, highlight mark