LyCORIS - Lora beYond Conventional methods, Other Rank adaptation Implementations for Stable diffusion.

A project that implements different parameter-efficient fine-tuning algorithms for Stable Diffusion.

This project originated from LoCon (see archive branch).

If you are interested in discussing more details, you can join our Discord server

If you want to check more in-depth experiment results and discussions for LyCORIS, you can check our paper

Algorithm Overview

LyCORIS currently contains LoRA (LoCon), LoHa, LoKr, (IA)^3, DyLoRA, Native fine-tuning (aka dreambooth). GLoRA and GLoKr are coming soon. Please check List of Implemented Algorithms and Guidelines for more details.

A simple comparison of some of these methods are provided below (to be taken with a grain of salt)

	Full	LoRA	LoHa	LoKr low factor	LoKr high factor
Fidelity	★	●	▲	◉	▲
Flexibility $^*$	★	●	◉	▲	● $^†$
Diversity	▲	◉	★	●	★
Size	▲	●	●	●	★
Training Speed Linear	★	●	●	★	★
Training Speed Conv	●	★	▲	●	●

★ > ◉ > ● > ▲ [> means better and smaller size is better]

$^*$ Flexibility means anything related to generating images not similar to those in the training set, and combination of multiple concepts, whether they are trained together or not
$^†$ It may become more difficult to switch base model or combine multiple concepts in this situation

Usage

Image Generation

a1111/sd-webui

After sd-webui 1.5.0, LyCORIS models are officially supported by the built-in LoRA system. You can put them in either models/Lora or models/LyCORIS and use the default syntax <lora:filename:multiplier> to trigger it.

When we add new model types, we will always make sure they can be used with the newest version of sd-webui.

As for sd-webui with version < 1.5.0, please check this extension.

Others

As far as we are aware, LyCORIS models are also supported in the following interfaces / online generation services (please help us complete the list!)

However, newer model types may not always be supported. If you encounter this issue, consider requesting the developers of the corresponding interface or website to include support for the new type.

Training

For the time being LyCORIS is mainly trained with kohya-ss/sd-scripts (see a list of compatible graphical interfaces and colabs at the end of the section). Supports for other trainers are coming soon.

A detilaed description of the network arguments is provided here.

kohya script

Activate sd-scripts' venv with

source PATH_TO_SDSCRIPTS_VENV/Scripts/activate

PATH_TO_SDSCRIPTS_VENV\Scripts\Activate.ps1 # or .bat     for cmd

Install this package

through pip
```
pip install lycoris_lora
```

from source

git clone https://github.com/KohakuBlueleaf/LyCORIS
cd LyCORIS
pip install .

Use this package's kohya module to run kohya's training script to train lycoris module for SD models

with command line arguments

python3 sd-scripts/train_network.py \
  --network_module lycoris.kohya \
  --network_dim "DIM_FOR_LINEAR" --network_alpha "ALPHA_FOR_LINEAR"\
  --network_args "conv_dim=DIM_FOR_CONV" "conv_alpha=ALPHA_FOR_CONV" \
  "dropout=DROPOUT_RATE" "algo=locon" \

with toml files
```
python train_network.py --config_file XXX.toml
```
For your convenience, some example toml files for LyCORIS training are provided in example/training_configs.

Tips:
- Use network_dim=0 or conv_dim=0 to disable linear/conv layer
- LoHa/LoKr/(IA)^3 doesn't support dropout yet.

Graphical Interfaces and Colabs

You can also train LyCORIS with the following graphical interfaces

and colabs (please help us complete the list!)

However, they are not guaranteed to be up-to-date. In particular, newer types may not be supported. Consider requesting the developpers for support or simply use the original kohya script in this case.

Utilities

Extract LoCon

You can extract LoCon from a dreambooth model with its base model.

python3 extract_locon.py <settings> <base_model> <db_model> <output>

Use --help to get more info

$ python3 extract_locon.py --help
usage: extract_locon.py [-h] [--is_v2] [--device DEVICE] [--mode MODE] [--safetensors] [--linear_dim LINEAR_DIM] [--conv_dim CONV_DIM]
                        [--linear_threshold LINEAR_THRESHOLD] [--conv_threshold CONV_THRESHOLD] [--linear_ratio LINEAR_RATIO] [--conv_ratio CONV_RATIO]
                        [--linear_percentile LINEAR_PERCENTILE] [--conv_percentile CONV_PERCENTILE]
                        base_model db_model output_name

Merge LyCORIS back to model

You can merge your LyCORIS model back to your checkpoint(base model)

python3 merge.py <settings> <base_model> <lycoris_model> <output>

Use --help to get more info

$ python3 merge.py --help
usage: merge.py [-h] [--is_v2] [--device DEVICE] [--dtype DTYPE] [--weight WEIGHT] base_model lycoris_model output_name

Change Log

For full log, please see Change.md

2023/09/27 update to 1.9.0

Add norm modules (for training LayerNorm and GroupNorm, which should be good for style)
Add full modules (So you can "native fine-tune" with LyCORIS now, should be convenient to try different weight)
Add preset config system
Add custom config system
Merge script support norm and full modules
Fix errors with optional requirements
Fix errors with not necessary import
Fix wrong factorization behaviors

Todo list

Citation

@misc{LyCORIS,
      title={Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluation}, 
      author={Shin-Ying Yeh and Yu-Guan Hsieh and Zhidong Gao and Bernard B W Yang and Giyeong Oh and Yanmin Gong},
      year={2023},
      eprint={2309.14859},
      archivePrefix={arXiv}
}

dqz00116 / lycoris Goto Github PK

lycoris's Introduction