Comfy Textures is an Unreal Engine plugin which integrates the editor with ComfyUI. It allows you to quickly create and refine textures for your scene using generative diffusion models.
Features:
- Single point-of-view texture projection
- Multiple point-of-view texture projection (WIP)
- Perspective camera
- Orthographic camera
- Inpainting
- Image to image
- Remote ComfyUI instance support
Works out of the box with Unreal Engine 5.x. Should work with 4.x with minor code changes.
Recommended at least 16GiB VRAM and 32GiB RAM for the SDXL workflows or run ComfyUI with the --lowvram
flag.
demo_editor.mp4
demo.mp4
Join the Discord for help and support.
If you want to contribute to the project, feel free to open a pull request or an issue.
-
Install ComfyUI by following the official installation instructions for your OS.
-
Download the necessary models and place them in the ComfyUI
models
directory.
The models directory is relative to the ComfyUI root directory i.e. <ComfyUI Root>/ComfyUI/models/
.
models/checkpoints/sd_xl_base_1.0_0.9vae.safetensors
- Downloadmodels/checkpoints/sd_xl_refiner_1.0_0.9vae.safetensors
- Downloadmodels/controlnet/diffusers_xl_canny_full.safetensors
- Downloadmodels/controlnet/diffusers_xl_depth_full.safetensors
- Downloadmodels/loras/lcm_lora_sdxl.safetensors
- Download (Note: Rename the file tolcm_lora_sdxl.safetensors
)models/upscale_models/4x-UltraSharp.pth
- Download
-
Clone this repository.
-
Build the Unreal project by right clicking on
MyProject.uproject
and selectingGenerate Visual Studio project files
.This will generate a
MyProject.sln
file in the project directory. Open the file in Visual Studio and build the project by pressingF7
. If you are encountering errors, make sure Visual Studio is correctly configured for Unreal Engine by following this guide. -
Open the project in the Unreal Engine editor.
-
Configure the plugin by going to
Project Settings -> Plugins -> Comfy Textures
.If you are running ComfyUI on a remote machine, you need to set the
Comfy Url
to the correct address. -
Open the plugin window by clicking on
Tools -> Editor Utility Widgets -> Comfy Textures Widget
.If the menu item is missing you need to open the
ComfyTexturesWidget
from the content browser inPlugins/Comfy Textures Content/
and clickRun Utility Widget
in the blueprint editor.
Make sure ComfyUI is up and running before proceeding.
-
Select the actors you want to texture in the Outliner.
-
Set your desired settings in the Comfy Textures widget.
-
Click
Render
to start the rendering process.
Notes:
- Your meshes must have UVs (autogenerated UVs are fine).
- The plugin will automatically create a new material instance and texture for each selected actor.
This mode uses a fast SDXL LCM model to create a low-resolution texture for each selected actor. Use this to quickly prototype and iterate on your scene.
This mode uses a slower SDXL workflow to refine the low-resolution textures created in the Create
mode. Use this to create the final high-resolution textures for your scene.
This mode allows you to edit the textures created in the Create
or Refine
mode by using an inpainting workflow. Use this mode to fix any artifacts or errors in the textures.
You can select from two edit modes - From Texture
and From Object
. From Texture
allows you to do precise edits by painting undesired areas to magenta (255, 0, 255, 255) using Mesh Paint. From Object
will inpaint all selected actors.
You can find the ComfyUI workflows used by the plugin in the Plugins/ComfyTextures/Content/Workflows/Original
folder. Load the JSONs into ComfyUI to see the full workflow and make changes. After making changes, save the workflow using the Save (API Format)
button in ComfyUI and copy the JSON to the corresponding file in the Plugins/ComfyTextures/Content/Workflows
folder.
You need to have Enable Dev mode Options
enabled in the ComfyUI settings to see the Save (API Format)
button.
Made by me (Alexander Dzhoganov). If you are hiring, hit me up at alexanderdzhoganov [at] gmail [dot] com.
Thank you to the Stability.ai and to the ComfyUI teams for their wonderful work on Stable Diffusion and ComfyUI.