ssube / onnx-web Goto Github PK
View Code? Open in Web Editor NEWweb UI for GPU-accelerated ONNX pipelines like Stable Diffusion, even on Windows and AMD
Home Page: https://discord.gg/7CdQmutGuw
License: MIT License
web UI for GPU-accelerated ONNX pipelines like Stable Diffusion, even on Windows and AMD
Home Page: https://discord.gg/7CdQmutGuw
License: MIT License
Generating an image is relatively expensive. Add a mechanism to lock pipelines or platforms and limit the number of simultaneous requests, usually to 1.
Make this easy to work on by setting up the devcontainer configuration files for Gitpod and Codespaces.
Add an option to blend the edges of the mask when outpainting.
The API containers are very large, even the CPU one that is not based on the CUDA image. Most of that space is coming from the pip packages, and the nvidia and torch directories, specifically. The CPU version of torch shouldn't need to install a bunch of CUDA runtime packages, so that might be installed properly.
Add buttons to the image history cards that allow you to copy the image to the img2img and inpainting source field, making it easier to generate and then refine an image.
The API currently creates the pipeline for each request, which was an easy way of setting the scheduler, but is unnecessary and slow.
Create the pipeline once during setup, then change the scheduler if necessary for the request.
When switching between txt2img and settings, keep the values in the image control inputs.
Even with the thread limit from #15, making multiple requests to generate images even with the same pipeline and model will likely crash the API:
reusing existing pipeline
10.2.2.16 - - [13/Jan/2023 10:34:50] "POST /api/txt2img?cfg=6.0&steps=25&model=stable-diffusion-onnx-v1-5&platform=amd&scheduler=euler-a&seed=-1&prompt=a+stone+magnifying+glass,+a+wooden+desk,
+steampunk,+realistic,+highly+detailed,+oil+painting&negativePrompt=&width=512&height=512 HTTP/1.1" 200 -
10.2.2.16 - - [13/Jan/2023 10:34:50] "GET /api/ready?output=txt2img_1200220748_9ff672bc153bad59ccd8bf173beb011cf170aaa3ef6f579713ab1ceaf5fbf04c.png HTTP/1.1" 200 -
2
023-01-13 10:34:50.0965716 [E:onnxruntime:, sequential_executor.cc:369 onnxruntime::SequentialExecutor::Execute] Non-zero status code returned while running Reshape node. Name:'/up_blocks.3/at
tentions.1/transformer_blocks.0/attn2/Reshape_2' Status Message: D:\a\_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\MLOperatorAuthorImpl.cpp(1971)\onnxruntime_pybind11_stat
e.pyd!00007FFAF7CC11AB: (caller: 00007FFAF82D0F7F) Exception(2) tid(c90) 8007023E {Application Error}
The exception %s (0x
13%|████████████████████▌ | 10/75 [00:04<00:28, 2.25it/s]
(onnx_env) ssube@SSUBE-DESKTOP C:\Users\ssube\stabdiff\onnx-try-2\onnx-web\api>flask --app=onnx_web.serve run --host=0.0.0.0 | 1/25 [00:00<00:08, 2.74it/s]
* Serving Flask app 'onnx_web.serve'
Add a tab called blend/blending that allows you to take multiple images from the image history and combine them, using a combination of sliders and the mask painting canvas from inpainting.
The regular flask
server in a venv
works correctly with AMD hardware acceleration, but to use that in the container, the device needs to be mounted - at least. Figure out what the command should be and document it.
Add an option for facial correction in img2img.
Add a tab or checkbox in img2img for inpainting mode.
This will need a way to draw or upload the mask.
Params:
The current implementation of inpainting always uses the original pixels from the source image.
Provide options to fill those pixels with noise, nothing, and blur them.
Add a tab or option for upscaling images using appropriate models.
Add a way to generate multiple images with the same request, either using the same parameters or unique ones (like different seeds). They should share the same mode, model, and platform, but may use different schedulers.
This will take longer than single images and may necessitate a polling system.
When outpainting is not selected, it almost looks like the outpainting button is in a disabled state, and not something you can click. That is misleading and should be changed.
Build OCI containers for both the API and GUI, based on both Debian and Alpine.
The settings tab is currently empty, but customizing the number of past images shown in the history lists would be a good setting.
For inpainting mode, it can be difficult to paint a mask once you have covered up most of the original details with grayscale.
Show the original source image under the canvas and make unpainted parts transparent, so the image shows through. If possible, put a second canvas over the mask and show a preview of the brush, to make it easier to see what will be masked.
Add at least the following options:
The image generation currently leaves a request open, waiting for the image to be ready. When running a large number of steps on a slow machine, or over the internet with cloudflare balancer in the middle, that may time out.
Since the image filename is now based on the parameters, it should be possible for the server to respond immediately and leave the conversion running. The client can check back later with the output name and request the status, and eventually the image itself.
Use the more efficient pipeline constructors to try and reduce memory usage when instantiating all of the modes. Since the pipelines have to be recreated when the model changes, this should create and cache all modes for the current model/scheduler combination, then use them as needed.
Implement ((emphasis)) and [deemphasis]
following the same syntax from diffusers, https://github.com/huggingface/diffusers/blob/main/examples/community/lpw_stable_diffusion_onnx.py#L78, and as documented in https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Features#attentionemphasis.
(increase:N)
(increase:1.1)
((increase:1.21))
[decrease:N]
[decrease:1.1]
[decrease:1.21]
Optionally support the {}
syntax and 1.05 base weight as well.
The GUI client can validate inputs, but the limits are currently hard-coded into the GUI, and not shared with the server.
Add an endpoint that returns the min/max/default values for the image params:
Figure out all of the images that need to be built. Currently, that includes:
I don't think there needs to be a container for ONNX DirectML, but maybe for WSL?
Possibly add:
Upload the output images to S3-compatible storage, like Ceph and Swift, instead of storing them locally.
The config.json
file is being loaded from ./config.json
on the page origin, not the page path, which is breaking Github pages. Load the config from a path relative to the page's path, which will not change the behavior when the index is not under a subdirectory.
reusing existing pipeline
expanding image for outpainting
[2023-01-15 11:33:31,112] ERROR in app: Exception on /api/inpaint [POST]
Traceback (most recent call last):
File "C:\Users\ssube\stabdiff\onnx-try-2\onnx-web\api\onnx_env\lib\site-packages\flask\app.py", line 2525, in wsgi_app
response = self.full_dispatch_request()
File "C:\Users\ssube\stabdiff\onnx-try-2\onnx-web\api\onnx_env\lib\site-packages\flask\app.py", line 1822, in full_dispatch_request
rv = self.handle_user_exception(e)
File "C:\Users\ssube\stabdiff\onnx-try-2\onnx-web\api\onnx_env\lib\site-packages\flask_cors\extension.py", line 165, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
File "C:\Users\ssube\stabdiff\onnx-try-2\onnx-web\api\onnx_env\lib\site-packages\flask\app.py", line 1820, in full_dispatch_request
rv = self.dispatch_request()
File "C:\Users\ssube\stabdiff\onnx-try-2\onnx-web\api\onnx_env\lib\site-packages\flask\app.py", line 1796, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
File "C:\Users\ssube\stabdiff\onnx-try-2\onnx-web\api\onnx_web\serve.py", line 566, in inpaint
executor.submit_stored(
File "C:\Users\ssube\stabdiff\onnx-try-2\onnx-web\api\onnx_env\lib\site-packages\flask_executor\executor.py", line 209, in submit_stored
self.futures.add(future_key, future)
File "C:\Users\ssube\stabdiff\onnx-try-2\onnx-web\api\onnx_env\lib\site-packages\flask_executor\futures.py", line 67, in add
raise ValueError("future_key {} already exists".format(future_key))
ValueError: future_key inpaint_964648543_e399ee8cf9f642e19c1c589fb4b3d087c3d2af525f0e0720bbd48437c9103255.png already exists
The background jobs are not being removed after they are finished, from the looks of it, and a new job cannot be scheduled with the same key. However, the future for the job has already been created and the job does run - it just doesn't show up in the GUI.
The flask_executor docs are pretty clear the futures should be pop()
'd once they are done: https://flask-executor.readthedocs.io/en/latest/api/flask_executor.html#flask_executor.futures.FutureCollection.pop
Make sure it works there too. Might be tricky since I don't have any of their hardware.
Publish the API as a pip package.
The txt2img seed is always random, at the moment. Add a field to the image controls to provide a fixed seed, generate one on click, or generate one on the server (-1).
When you try to run inpainting with a regular model or txt2img with an inpainting model, it currently fails and the image card shows an empty/error image.
It would be nice to warn the user about incompatible models, or even enable/disable the appropriate tabs.
Add a tab or option in img2img for outpainting.
Params:
On my Win10/AMD test machine, I get the following error while running the im2img pipeline:
[2023-01-07 20:37:22,695] ERROR in app: Exception on /img2img [POST]
Traceback (most recent call last):
File "C:\Users\ssube\stabdiff\onnx-try-2\onnx-web\api\onnx_env\lib\site-packages\flask\app.py", line 2525, in wsgi_app
response = self.full_dispatch_request()
File "C:\Users\ssube\stabdiff\onnx-try-2\onnx-web\api\onnx_env\lib\site-packages\flask\app.py", line 1822, in full_dispatch_request
rv = self.handle_user_exception(e)
File "C:\Users\ssube\stabdiff\onnx-try-2\onnx-web\api\onnx_env\lib\site-packages\flask\app.py", line 1820, in full_dispatch_request
rv = self.dispatch_request()
File "C:\Users\ssube\stabdiff\onnx-try-2\onnx-web\api\onnx_env\lib\site-packages\flask\app.py", line 1796, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
File "C:\Users\ssube\stabdiff\onnx-try-2\onnx-web\api\onnx_web\serve.py", line 234, in img2img
image = pipe(
File "C:\Users\ssube\stabdiff\onnx-try-2\onnx-web\api\onnx_env\lib\site-packages\diffusers\pipelines\stable_diffusion\pipeline_onnx_stable_d
iffusion_img2img.py", line 355, in __call__
init_latents = self.vae_encoder(sample=image)[0]
File "C:\Users\ssube\stabdiff\onnx-try-2\onnx-web\api\onnx_env\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "C:\Users\ssube\stabdiff\onnx-try-2\onnx-web\api\onnx_env\lib\site-packages\diffusers\models\vae.py", line 628, in forward
posterior = self.encode(x).latent_dist
File "C:\Users\ssube\stabdiff\onnx-try-2\onnx-web\api\onnx_env\lib\site-packages\diffusers\models\vae.py", line 566, in encode
h = self.encoder(x)
File "C:\Users\ssube\stabdiff\onnx-try-2\onnx-web\api\onnx_env\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "C:\Users\ssube\stabdiff\onnx-try-2\onnx-web\api\onnx_env\lib\site-packages\diffusers\models\vae.py", line 130, in forward
sample = self.conv_in(sample)
File "C:\Users\ssube\stabdiff\onnx-try-2\onnx-web\api\onnx_env\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "C:\Users\ssube\stabdiff\onnx-try-2\onnx-web\api\onnx_env\lib\site-packages\torch\nn\modules\conv.py", line 464, in forward
return self._conv_forward(input, self.weight, self.bias)
File "C:\Users\ssube\stabdiff\onnx-try-2\onnx-web\api\onnx_env\lib\site-packages\torch\nn\modules\conv.py", line 460, in _conv_forward
return F.conv2d(input, weight, bias, self.stride,
TypeError: conv2d() received an invalid combination of arguments - got (numpy.ndarray, Parameter, Parameter, tuple, tuple, tuple, int), but ex
pected one of:
* (Tensor input, Tensor weight, Tensor bias, tuple of ints stride, tuple of ints padding, tuple of ints dilation, int groups)
didn't match because some of the arguments have invalid types: (numpy.ndarray, Parameter, Parameter, tuple, tuple, tuple, int)
* (Tensor input, Tensor weight, Tensor bias, tuple of ints stride, str padding, tuple of ints dilation, int groups)
didn't match because some of the arguments have invalid types: (numpy.ndarray, Parameter, Parameter, tuple, tuple, tuple, int)
The txt2img pipeline works perfectly on the same machine, with acceleration, so this is somehow related to a difference between them. The input image is a good suspect, since the first parameter is a numpy.ndarray
and not a Tensor
, but it looks like there is some code in diffusers and/or onnx to convert PIL images into Tensors, so that may not be the issue.
This happens using both the DmlExecutionProvider and CPUExecutionProvider, which makes me think it is related to Win10 rather than AMD.
Rather than making people build and serve their own GUI bundle, host the latest main
version on GH pages that defaults to using the API server at http://localhost:5000
.
Potentially add a setting to change the API server, although the API client is currently created before the React tree and is immutable, so that might be tricky.
Many hosting providers have a way to give containers some persistent storage, but that starts out empty. The API currently checks for the existence of the models directory and throws an error if it doesn't exist. That doesn't work with a fresh volume unless you launch some kind of shell and convert the models ahead of time.
Provide a way to specify one or more models and have the container download and convert them when it starts up. If the persistent storage is mounted under /home
, the diffusers
cache should automatically be persisted, and the ONNX_WEB_
variables can be set to directories on the same volume.
Offer at least one version of the API container with the GUI bundle embedded within it, so it can serve both components without needing two containers/commands/ports/processes.
This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.
These updates are currently rate-limited. Click on a checkbox below to force their creation now.
chai-as-promised
, @types/chai-as-promised
)mocha
, @types/mocha
)@emotion/react
, @emotion/styled
)@typescript-eslint/eslint-plugin
, @typescript-eslint/parser
)These updates have all been created already. Click a checkbox below to force a retry/rebase of any.
@types/react
, @types/react-dom
, react
, react-dom
)@typescript-eslint/eslint-plugin
, @typescript-eslint/parser
)@mui/icons-material
, @mui/lab
, @mui/material
)run/cpu/docker-compose.yaml
run/cuda/docker-compose.yaml
run/rocm/docker-compose.yaml
api/Containerfile.cpu.buster
docker.io/python 3.10-buster
api/Containerfile.cuda.ubuntu
docker.io/nvidia/cuda 11.7.1-runtime-ubuntu20.04
api/Containerfile.directml.buster
docker.io/python 3.10-buster
api/Containerfile.rocm.ubuntu
gui/Containerfile.nginx.alpine
docker.io/nginx 1.23-alpine
gui/Containerfile.nginx.bullseye
docker.io/nginx 1.23
gui/Containerfile.node.alpine
docker.io/node 18-alpine
gui/Containerfile.node.buster
docker.io/node 18-buster
.gitlab-ci.yml
docker.io/node 18
docker.io/docker 20.10
docker.io/docker 20.10-dind
docker.io/python 3.10
docker.io/python 3.10
docker.io/python 3.9
docker.io/python 3.8
gui/package.json
@apextoaster/js-utils ^0.5.0
@emotion/react ^11.10.6
@emotion/styled ^11.10.6
@mui/icons-material ^5.11.16
@mui/lab ^5.0.0-alpha.126
@mui/material ^5.12.0
@tanstack/react-query ^4.0.5
@types/lodash ^4.14.192
@types/node ^18.15.11
browser-bunyan ^1.8.0
exifreader ^4.13.0
i18next ^22.4.14
i18next-browser-languagedetector ^7.0.1
lodash ^4.17.21
noicejs ^5.0.0-3
react ^18.2.0
react-dom ^18.2.0
react-i18next ^12.2.0
react-use ^17.4.0
semver ^7.4.0
tslib ^2.5.0
zustand ^4.3.7
@mochajs/multi-reporter ^1.1.0
@types/chai-as-promised ^7.1.5
@types/mocha ^10.0.1
@types/react ^18.0.34
@types/react-dom ^18.0.10
@types/sinon-chai ^3.2.9
@typescript-eslint/eslint-plugin ^5.59.0
@typescript-eslint/parser ^5.59.0
c8 ^7.13.0
chai ^4.3.7
chai-as-promised ^7.1.1
esbuild ^0.17.17
esbuild-plugin-alias ^0.2.1
esbuild-plugin-copy ^2.1.0
eslint ^8.38.0
eslint-plugin-chai ^0.0.1
eslint-plugin-chai-expect ^3.0.0
eslint-plugin-chai-expect-keywords ^2.1.0
eslint-plugin-import ^2.27.5
eslint-plugin-mocha ^10.1.0
eslint-plugin-no-null ^1.0.2
eslint-plugin-sonarjs ^0.19.0
mocha ^10.2.0
mocha-junit-reporter ^2.2.0
sinon ^15.0.4
sinon-chai ^3.7.0
source-map-support ^0.5.21
typescript ^4.9.5
@types/react 18.0.34
api/requirements/amd-linux-nightly.txt
torch ==2.0.1
torchvision ==0.15.2
api/requirements/amd-linux.txt
torch ==2.0.1
torchvision ==0.15.2
api/requirements/amd-windows-nightly.txt
torch ==2.1.1
torchvision ==0.16.1
api/requirements/amd-windows.txt
torch ==2.1.1
torchvision ==0.16.1
onnxruntime-directml ==1.16.3
api/requirements/base.txt
numpy ==1.24.1
protobuf ==3.20.3
accelerate ==0.25.0
coloredlogs ==15.0.1
controlnet_aux ==0.0.7
datasets ==2.15.0
diffusers ==0.24.0
huggingface-hub ==0.19.4
invisible-watermark ==0.2.0
mediapipe ==0.9.2.1
omegaconf ==2.3.0
onnx ==1.15.0
optimum ==1.16.0
safetensors ==0.4.1
timm ==0.9.12
torchsde ==0.2.6
transformers ==4.36.1
basicsr ==1.4.2
codeformer-pip ==0.0.4
facexlib ==0.2.5
gfpgan ==1.3.8
realesrgan ==0.3.0
arpeggio ==2.0.2
boto3 ==1.26.87
flask ==3.0.0
flask-cors ==3.0.10
jsonschema ==4.17.3
piexif ==1.1.3
pyyaml ==6.0
setproctitle ==1.3.2
waitress ==2.1.2
werkzeug ==3.0.1
api/requirements/cpu-nightly.txt
torch ==2.1.1
torchvision ==0.16.1
api/requirements/cpu.txt
torch ==2.1.1
torchvision ==0.16.1
onnxruntime ==1.16.3
api/requirements/dev.txt
api/requirements/nvidia-nightly.txt
torch ==2.1.1
torchvision ==0.16.1
api/requirements/nvidia.txt
torch ==2.1.1
torchvision ==0.16.1
onnxruntime-gpu ==1.16.3
api/requirements/optional-linux.txt
pytorch_lightning ==2.0.0
PyGObject ==3.44.1
api/requirements/optional-windows.txt
pytorch_lightning ==2.0.0
win10toast ==0.9
Jobs are immediately removed from memory if the client comes back to check /ready
, with #55, but abandoned jobs are not removed until the Executor starts to prune them, which defaults to 50.
Look for jobs that finished more than N minutes ago and remove their futures, probably as part of the /ready
endpoint.
Rather than painting hard-edged circles all the time, there should be a slider to adjust brush hardness. Draw a radial gradient using multiply or screen, depending on the shade of the brush, and fade out around the edges.
While onnx
is always needed, users should usually only have one of onnxruntime
, onnxruntime-gpu
, or onnxruntime-directml
installed. The onnxruntime
package is for CPU usage, while onnxruntime-gpu
and onnxruntime-directml
are for the GPU and also GPU. When using the onnxruntime-directml
, it also helps to install the nightly version, which can be faster.
For torch
, it seems like only torch
itself is really needed.
Now that there are two containers and some volume mounts, add files for https://docs.podman.io/en/latest/markdown/podman-pod.1.html and probably https://docs.docker.com/compose/compose-file/.
Now that the GUI client loads the image control parameters from the server, if the server is not available, the client will not load. That means that the Github pages demo looks like a blank white screen for anyone who is not already running the API server.
That's not a great experience, so check if the server is available, and add some friendly messaging if it is not. Some parameters will be missing and queries will not load, so many dropdown menus will be empty, but the tab controls can be replaced by an error message.
There is no way to see an image again once it has been sent to the client, although they are saved to disk for later reference.
Add an endpoint to serve the output of previous requests, using a unique identifier created along with the image.
The current txt2img endpoint will probably need to change to return a JSON structure containing the image ID.
Make the img2img and inpaint source and mask fields persist between tab changes, like the image size/step controls now do (#11).
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.