ComfyUI implementation for ELLA.
- [2024.4.30] Add a new node
ELLA Text Encodeto automatically concat ella and clip condition. - [2024.4.24] Upgraded ELLA Apply method. Better compatibility with the comfyui ecosystem. Refer to the method mentioned in ComfyUI_ELLA PR #25
- DEPRECATED:
Apply ELLAwithoutsimgasis deprecated and it will be removed in a future version.
- DEPRECATED:
- [2024.4.22] Fix unstable quality of image while multi-batch. Add CLIP concat (support lora trigger words now).
- [2024.4.19] Documenting nodes.
- [2024.4.19] Initial repo.
- SIGMAS from node
BasicScheduleror TIMESTEPS by nodeSet ELLA Timestepsmust be the same as the KSampler settings. Because Timestep-Aware Semantic Connector (TSC), which dynamically adapts semantics features over sampling time steps, has been introduced. - If you need concat clip
CONDITIONINGto make LoRA trigger words effective, ELLA outputCONDITIONINGalways needs to be linked to theconditioning_toofConditioning (Concat)node.
The examples directory has workflow examples. You can directly load these images as workflow into ComfyUI for use.
All legacy workflows was compatible. But it is deprecated and will be removed in a future version.
🎉 It works with controlnet!
🎉 It works with lora trigger words by concat CLIP CONDITIONING!
ELLA CONDITIONING always needs to be linked to the conditioning_to of Conditioning (Concat) node.
With ELLA Text Encode node, can simplify the workflow.
With the upgrade(2024.4.24), some interesting workflow can be implemented, such as using ELLA only in positive. As shown below:
| positive + negative | positive only |
|---|---|
 |
 |
However, there is no guarantee that positive-only will bring better results.
Workflow with AYS.
AYS got with more visual details and better text-alignment, ref to paper.
| w/ AYS | w/o AYS |
|---|---|
 |
 |
And EMMA is working in progress.
Download or git clone this repository inside ComfyUI/custom_nodes/ directory. ComfyUI-ELLA requires the latest version of ComfyUI. If something doesn't work be sure to upgrade.
cd ComfyUI/custom_nodes
git clone https://github.com/TencentQQGYLab/ComfyUI-ELLANext install dependencies.
cd ComfyUI-ELLA
pip install -r requirements.txtThese models must be placed in the corresponding directories under models.
Remember you can also use any custom location setting an ella & ella_encoder entry in the extra_model_paths.yaml file.
ComfyUI/models/ella, create it if not present.- Place ELLA Models here
ComfyUI/models/ella_encoder, create it if not present.- Place FLAN-T5 XL Text Encoder here, it should be a folder of transfomers structure with config.json
In summary, you should have the following model directory structure:
ComfyUI/models/ella/
└── ella-sd1.5-tsc-t5xl.safetensors
ComfyUI/models/ella_encoder/
└── models--google--flan-t5-xl--text_encoder
├── config.json
├── model.safetensors
├── special_tokens_map.json
├── spiece.model
├── tokenizer_config.json
└── tokenizer.json- XXX not implemented for 'Half'. See issue #12
- AYS + Ella getting dark image generations. See issue #39
- Check if
add_noiseofSamplerCustomnode is enabled. - Lower the
cfgofSamplerCustomnode.
- Check if
- Support prompt weighting
 JettHu |
 budui |
 kijai |
 huagetai |
- ComfyUI: https://github.com/comfyanonymous/ComfyUI
- Diffusers (borrowed timestep modules): https://github.com/huggingface/diffusers
@misc{hu2024ella,
title={ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment},
author={Xiwei Hu and Rui Wang and Yixiao Fang and Bin Fu and Pei Cheng and Gang Yu},
year={2024},
eprint={2403.05135},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent