tjm35/prompt-fusion-extension

auto1111 webui extension for all sorts of prompt interpolations!

Prompt Fusion

Prompt Fusion is an auto1111 webui extension that adds more possibilities to the native prompt syntax. Among other additions, it allows to interpolate between the embeddings of different prompts, continuously:

# linear prompt interpolation
[night light:magical forest: 5, 15]

# catmull-rom curve prompt interpolation
[night light:magical forest:norvegian territory: 5, 15, 25:catmull]

# linear attention interpolation
(fire extinguisher: 1.0, 2.0)

# prompt-editing-aware attention interpolation
[(fire extinguisher: 1.0, 2.0)::5]

# define functions and variables to simplify repeating patterns and use a consistent structure
$prompt($style, $quality, $character, $background) = (
    A detailed picture in the style of $style,
    $quality,
    $character lying back,
    $background in the background
:1)

The prompt interpolation feature is similar to Prompt Travel, which allows to create videos of images generated by navigating the latent space iteratively. Unlike Prompt Travel however, instead of generating multiple images, Prompt Fusion allows you to travel during the sampling process of a single image. Also, instead of interpolating the latent space, it uses the embedding space to determine intermediate embeddings.

Prompt interpolation is also similar to Prompt Blending. The main difference is that this extension calculates a new embedding for every step, as opposed to calculating it once and using that same one embedding for all the steps.

The attention interpolation feature is similar to Shift Attention, which allows to generate multiple images with slight variations in the attention given to certain parts of the prompt. Unlike Shift Attention, instead of generating multiple images, Prompt Fusion allows to shift the attention of certain parts of a prompt during the sampling process of a single image.

Examples

1. Influencing the beginning of the sampling process

Interpolate linearly (by default) from lion (step 0) to bird (step 8) to girl (step 11), and stay at girl for the rest of the sampling steps:

[lion:bird:girl: , 7, 10]

2. Influencing the middle of the sampling process

Interpolate using a bezier curve from fireball monster (step 0) to dragon monster (step 12, because 30 steps * 0.4 = step 12), while using seawater monster as an intermediate control point to steer the curve away during interpolation and to get creative results:

[fireball:seawater:dragon: , .1, .4:bezier] monster

Features

• Prompt interpolation using a curve function
• Attention interpolation aware of contextual prompt editing
• Complete backwards compatibility with prompt syntax from https://github.com/AUTOMATIC1111/stable-diffusion-webui
• Prompt variables and functions

Webui supported releases

The following webui releases are officially supported:

• v1.0.0-pre

Installation

1. Visit the Extensions tab of Automatic's WebUI.
2. Visit the Install from URL subtab.
3. Copy and paste https://github.com/ljleb/prompt-fusion-extension in the URL for extension's git repository textbox.
4. Press the Install button.

Usage

• Check the wiki pages for the extension documentation.

Related Projects

• Prompt Travel: https://github.com/Kahsolt/stable-diffusion-webui-prompt-travel
• Shift Attention: https://github.com/yownas/shift-attention
• Prompt Blending: https://github.com/amotile/stable-diffusion-backend/tree/master/src/process/implementations/automatic1111_scripts