ComfyUI-Prediction

Fully customizable Classifier Free Guidance for ComfyUI.

Installation

Clone this repo into ComfyUI/custom_nodes or use ComfyUI-Manager.

(Optional) If you want beautiful teal PREDICTION edges like the example apply patches/colorPalette.js.patch to ComfyUI/web/extensions/core/colorPalette.js.

Usage

All custom nodes are provided under Add Node > sampling > prediction. An example workflow is in examples/custom_prediction.json.

Follow these steps for fully custom prediction:

You will need to use the sampling > prediction > Sample Predictions node as your sampler.
The sampler input comes from sampling > custom_sampling > samplers. Generally you'll use KSamplerSelect.
The sigmas input comes from sampling > custom_sampling > schedulers. If you don't know what sigmas you are using, try BasicScheduler. (NOTE: These nodes are not in the "sigmas" menu.)
You'll need one or more prompts. Chain conditioning > CLIP Text Encode (Prompt) to sampling > prediction > Conditioned Prediction to get started.
After your prediction chain, connect the result to the noise_prediction input of your Sample Predictions node.

Predictors

Primitive Nodes

All other predictions can be implemented in terms of these nodes. However, it may get a little messy.

Conditioned Prediction - Evaluate your chosen model with a prompt (conditioning). You need to pick a unique conditioning name like "positive", "negative", or "empty". (The names are arbitrary and you can choose any name, but the names may evenutally interact with ControlNet if/when it's implemented.)

Combine Predictions - Operates on two predictions. Supports add (+), subtract (-), multiply (*), divide (/), vector projection (proj), vector rejection (oproj), min, and max.
prediction_A <operation> prediction_B

Scale Prediction - Linearly scales a prediction.
prediction * scale

Convinence Nodes

Scaled Guidance Prediction - Combines a baseline prediction with a scaled guidance prediction, similar to CFG.
baseline + guidance * scale

Avoid and Erase Prediction - Re-aligns a desirable (positive) prediction called guidance away from an undesirable (negative) prediction called avoid_and_erase, and erases some of the negative prediction as well.
guidance - (guidance proj avoid_and_erase) * avoid_scale - avoid_and_erase * erase_scale

Prebuilt Nodes

CFG Prediction - Vanilla Classifer Free Guidance (CFG) with a postive prompt and a negative/empty prompt.
(positive - negative) * cfg_scale + negative

Perp-Neg Prediction - Implements https://arxiv.org/abs/2304.04968. This is also implemented less flexibly in vanilla ComfyUI under _for_testing > Perp-Neg.
pos_ind = positive - empty; neg_ind = negative - empty
(pos_ind - (pos_ind proj neg_ind) * neg_scale) * cfg_scale + empty

Limitations

ControlNet is not supported at this time.

Regional prompting may work but is totally untested.

Any other advanced features affecting conditioning are not likely to work.

License

The license is the same as ComfyUI, GPL 3.0.

feffy380 / comfyui-prediction Goto Github PK