Error installing in Automatic1111

[nvidia]

Error installing in Automatic1111

[4lt3r3go]

same issue here

[Altrue]

For my part, the python -m pip uninstall -y nvidia-cudnn-cu11 didn't seem to work as the extension was "not installed".

So instead I went to venv\Lib\site-packages and I removed the cudnn.dist-info & the cudnn folder in the nvidia folder. It seems to be working fine. At least, I can start without errors, and I can start generating the engine. It seems to be generating without issues now.

Edit: I can confirm that after doing this fix, everything works for me.

[Jonseed]

ok, figured out the pip freeze.

absl-py==1.4.0
accelerate==0.21.0
addict==2.4.0
aenum==3.1.12
aiofiles==23.2.1
aiohttp==3.8.4
aiosignal==1.3.1
altair==5.0.0
antlr4-python3-runtime==4.9.3
anyio==3.6.2
async-timeout==4.0.2
attrs==23.1.0
basicsr==1.4.2
beautifulsoup4==4.12.2
blendmodes==2022
blis==0.7.9
boltons==23.0.0
Brotli==1.1.0
cachetools==5.3.0
catalogue==2.0.8
certifi==2023.5.7
cffi==1.15.1
chardet==4.0.0
charset-normalizer==3.1.0
clean-fid==0.1.35
click==8.1.7
clip @ git+https://github.com/openai/CLIP.git@d50d76daa670286dd6cacf3bcd80b5e4823fc8e1
colorama==0.4.6
confection==0.0.4
contourpy==1.0.7
cssselect2==0.7.0
cycler==0.11.0
cymem==2.0.7
deprecation==2.1.0
duckduckgo-search==3.9.3
dynamicprompts==0.29.0
einops==0.4.1
facexlib==0.3.0
fastapi==0.94.0
ffmpy==0.3.0
filelock==3.12.0
filterpy==1.4.5
flatbuffers==23.5.9
font-roboto==0.0.1
fonts==0.0.3
fonttools==4.39.4
freetype-py==2.3.0
frozenlist==1.3.3
fsspec==2023.5.0
ftfy==6.1.1
future==0.18.3
fvcore==0.1.5.post20221221
gdown==4.7.1
gfpgan==1.3.8
gitdb==4.0.10
GitPython==3.1.32
google-auth==2.18.1
google-auth-oauthlib==1.0.0
gradio==3.41.2
gradio_client==0.5.0
grpcio==1.54.2
h11==0.12.0
h2==4.1.0
hpack==4.0.0
httpcore==0.15.0
httpx==0.24.1
huggingface-hub==0.14.1
hyperframe==6.0.1
idna==2.10
imageio==2.28.1
importlib-resources==6.0.1
inflection==0.5.1
iopath==0.1.9
Jinja2==3.1.2
jsonmerge==1.8.0
jsonschema==4.17.3
kiwisolver==1.4.4
kornia==0.6.7
langcodes==3.3.0
lark==1.1.2
lazy_loader==0.2
lightning-utilities==0.8.0
linkify-it-py==2.0.2
llvmlite==0.40.0
lmdb==1.4.1
lpips==0.1.4
lxml==4.9.3
Markdown==3.4.3
markdown-it-py==2.2.0
MarkupSafe==2.1.2
matplotlib==3.7.1
mdit-py-plugins==0.3.3
mdurl==0.1.2
mediapipe==0.10.5
mpmath==1.3.0
multidict==6.0.4
murmurhash==1.0.9
networkx==3.1
numba==0.57.0
numpy==1.23.5
nvidia-cublas-cu11==11.11.3.6
nvidia-cuda-nvrtc-cu11==11.8.89
nvidia-cuda-runtime-cu11==11.8.89
nvidia-cudnn-cu11==8.9.4.25
oauthlib==3.2.2
omegaconf==2.2.3
onnx==1.14.1
onnx-graphsurgeon==0.3.27
open-clip-torch==2.20.0
opencv-contrib-python==4.7.0.72
opencv-python==4.8.0.76
orjson==3.8.12
packaging==23.1
pandas==2.0.1
pathy==0.10.1
piexif==1.1.3
Pillow==9.5.0
polygraphy==0.49.0
portalocker==2.7.0
preshed==3.0.8
protobuf==3.20.2
psutil==5.9.5
py-cpuinfo==9.0.0
pyasn1==0.5.0
pyasn1-modules==0.3.0
pycairo==1.23.0
pycparser==2.21
pydantic==1.10.7
pydub==0.25.1
Pygments==2.15.1
pyparsing==3.0.9
pyrsistent==0.19.3
PySocks==1.7.1
python-dateutil==2.8.2
python-multipart==0.0.6
pytorch-lightning==1.9.4
pytz==2023.3
PyWavelets==1.4.1
pywin32==306
PyYAML==6.0
realesrgan==0.3.0
regex==2023.5.5
reportlab==4.0.0
requests==2.31.0
requests-oauthlib==1.3.1
resize-right==0.0.2
rich==13.6.0
rlPyCairo==0.2.0
rsa==4.9
safetensors==0.3.1
scikit-image==0.21.0
scipy==1.10.1
seaborn==0.13.0
semantic-version==2.10.0
Send2Trash==1.8.0
sentencepiece==0.1.99
six==1.16.0
smart-open==6.3.0
smmap==5.0.0
sniffio==1.3.0
socksio==1.0.0
sounddevice==0.4.6
soupsieve==2.4.1
spacy==3.5.3
spacy-legacy==3.0.12
spacy-loggers==1.0.4
srsly==2.4.6
starlette==0.26.1
svglib==1.5.1
sympy==1.12
tabulate==0.9.0
tb-nightly==2.14.0a20230520
tensorboard-data-server==0.7.0
tensorrt==9.0.1.post11.dev4
tensorrt-bindings==9.0.1.post11.dev4
tensorrt-libs==9.0.1.post11.dev4
termcolor==2.3.0
thinc==8.1.10
thop==0.1.1.post2209072238
tifffile==2023.4.12
timm==0.9.2
tinycss2==1.2.1
tokenizers==0.13.3
tomesd==0.1.3
tomli==2.0.1
toolz==0.12.0
torch==2.0.1+cu118
torchdiffeq==0.2.3
torchmetrics==0.11.4
torchsde==0.2.5
torchvision==0.15.2+cu118
tqdm==4.65.0
trampoline==0.1.2
transformers==4.30.2
typer==0.7.0
typing_extensions==4.5.0
tzdata==2023.3
uc-micro-py==1.0.2
ultralytics==8.0.195
urllib3==1.26.15
uvicorn==0.22.0
wasabi==1.1.1
wcwidth==0.2.6
webencodings==0.5.1
websockets==11.0.3
Werkzeug==2.3.4
xformers==0.0.20
yacs==0.1.8
yapf==0.33.0
yarl==1.9.2

[MorkTheOrk]

I can see that the packages are installed that our extension needs, but cudnn is still installed.
The extension won't uninstall it because it only checks if tensorrt is installed, and if it is, it won't uninstall cudnn.... might need a code change here. You can try to run the following in the venv of automatic1111 :

python -m pip uninstall -y nvidia-cudnn-cu11

[andreacostigliolo]

the problem is medvram. Solved!

[Altrue]

What is that about "-vidia-cudnn-cu11" being an invalid distribution?

From my understanding, when pip is installing a package, it removes the first letter of the package name. When the installation completes, this extra file is removed. When the install crashes, the extra file isn't removed and is left as a fake package that is ignored. It's fine to leave it, but you can also navigate to venv\Lib\site-packages to remove it.

(I had a "-rotobuf" warning myself :p)

[Jonseed]

I tried stopping and restarting the webui server. I made sure I am on the SD 1.5 model. If I try to export the default engine I get the same full error that I did above, ending in 'AsyncRequest' object has no attribute '_json_response_data'
I don't think it was the model I was on, as it is giving the same error with SD 1.5.

[racerx2oo3]

i also foudn that if i remove the models i converted then the extention disappear from the UI. restoring those deleted files will make the extention appear again... wtf lol

You removed the generated trt model from the Unet-trt folder? If so you need to also remove the reference to that model in the model.json folder. Or if you only have one model generated, just delete the model file and the .json file both.

[Jonseed]

I shouldn't have to install anything manually... Why are there separate cudnn libraries? They seem to be conflicting... Or did TensorRT fail to remove the package after installing the tensorrt wheel?

[Altrue]

For my part, the python -m pip uninstall -y nvidia-cudnn-cu11 didn't seem to work as the extension was "not installed".
So instead I went to venv\Lib\site-packages and I removed the cudnn.dist-info & the cudnn folder in the nvidia folder. It seems to be working fine. At least, I can start without errors, and I can start generating the engine. It seems to be generating without issues now.

Have you exported the engine and enabled it? See here: https://github.com/NVIDIA/Stable-Diffusion-WebUI-TensorRT#how-to-use

Yes, I was currently following the instructions actually, to see if "seems to be working" could be converted into "is working". I can confirm it works flawlessly! I was able to export an engine and confirm that it increases my generation speed by ~40%!

In summary, for me the issue was that during the installation, somehow it didn't uninstall cudnn, and cudnn was taking priority over the dlls in tensorrt. After removing cudnn manually, my problem was solved.

[Jonseed]

Ok, so I removed medvram, and that did seem to be causing the problem with the tensors being on the same device. But now I get a bunch of other errors when trying to export default engine:

{'sample': [(1, 4, 64, 64), (2, 4, 64, 64), (8, 4, 96, 96)], 'timesteps': [(1,), (2,), (8,)], 'encoder_hidden_states': [(1, 77, 768), (2, 77, 768), (8, 154, 768)]}
Disabling attention optimization
D:\repos\stable-diffusion-webui\venv\lib\site-packages\einops\einops.py:314: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  known = {axis for axis in composite_axis if axis_name2known_length[axis] != _unknown_axis_length}
D:\repos\stable-diffusion-webui\venv\lib\site-packages\einops\einops.py:315: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  unknown = {axis for axis in composite_axis if axis_name2known_length[axis] == _unknown_axis_length}
D:\repos\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\openaimodel.py:158: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert x.shape[1] == self.channels
D:\repos\stable-diffusion-webui\modules\sd_hijack_unet.py:26: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if a.shape[-2:] != b.shape[-2:]:
D:\repos\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\openaimodel.py:109: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert x.shape[1] == self.channels
ERROR:asyncio:Exception in callback H11Protocol.timeout_keep_alive_handler()
handle: <TimerHandle when=7159.015 H11Protocol.timeout_keep_alive_handler()>
Traceback (most recent call last):
  File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\h11\_state.py", line 249, in _fire_event_triggered_transitions
    new_state = EVENT_TRIGGERED_TRANSITIONS[role][state][event_type]
KeyError: <class 'h11._events.ConnectionClosed'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\X\AppData\Local\Programs\Python\Python310\lib\asyncio\events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 383, in timeout_keep_alive_handler
    self.conn.send(event)
  File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\h11\_connection.py", line 468, in send
    data_list = self.send_with_data_passthrough(event)
  File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\h11\_connection.py", line 493, in send_with_data_passthrough
    self._process_event(self.our_role, event)
  File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\h11\_connection.py", line 242, in _process_event
    self._cstate.process_event(role, type(event), server_switch_event)
  File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\h11\_state.py", line 238, in process_event
    self._fire_event_triggered_transitions(role, event_type)
  File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\h11\_state.py", line 251, in _fire_event_triggered_transitions
    raise LocalProtocolError(
h11._util.LocalProtocolError: can't handle event type ConnectionClosed when role=SERVER and state=SEND_RESPONSE

I thought it might be the model I had selected, so I switched to the SD 1.5 model, and then got this error:

Loading model v1-5-pruned-emaonly.safetensors [6ce0161689] (2 out of 3)
Loading weights [6ce0161689] from D:\repos\stable-diffusion-webui\models\Stable-diffusion\v1-5-pruned-emaonly.safetensors
Creating model from config: D:\repos\stable-diffusion-webui\configs\v1-inference.yaml
============= Diagnostic Run torch.onnx.export version 2.0.1+cu118 =============
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================

[W] 'colored' module is not installed, will not use colors when logging. To enable colors, please install the 'colored' module: python3 -m pip install colored
[E] ONNX-Runtime is not installed, so constant folding may be suboptimal or not work at all.
    Consider installing ONNX-Runtime: D:\repos\stable-diffusion-webui\venv\Scripts\python.exe -m pip install onnxruntime
[I] Folding Constants | Pass 1
[!] Module: 'onnxruntime.tools.symbolic_shape_infer' is required but could not be imported.
    Note: Error was: No module named 'onnxruntime'
    You can set POLYGRAPHY_AUTOINSTALL_DEPS=1 in your environment variables to allow Polygraphy to automatically install missing modules.
[W] Falling back to `onnx.shape_inference` because `onnxruntime.tools.symbolic_shape_infer` either could not be loaded or did not run successfully.
    Note that using ONNX-Runtime for shape inference may be faster and require less memory.
    Consider installing ONNX-Runtime or setting POLYGRAPHY_AUTOINSTALL_DEPS=1 in your environment variables to allow Polygraphy to do so automatically.
[W] Attempting to run shape inference on a large model (1641.0 MiB). This may require a large amount of memory.
    If memory consumption becomes too high, the process may be killed. You may want to try disabling shape inference in that case.
[I]     Total Nodes | Original:  8992, After Folding:  6216 |  2776 Nodes Folded
[I] Folding Constants | Pass 2
[W] colored module is not installed, will not use colors when logging. To enable colors, please install the colored module: python3 -m pip install colored
[W] Inference failed. You may want to try enabling partitioning to see better results. Note: Error was:
No module named 'onnxruntime'
[!] Module: 'onnxruntime.tools.symbolic_shape_infer' is required but could not be imported.
    Note: Error was: No module named 'onnxruntime'
    You can set POLYGRAPHY_AUTOINSTALL_DEPS=1 in your environment variables to allow Polygraphy to automatically install missing modules.
[W] Attempting to run shape inference on a large model (1642.0 MiB). This may require a large amount of memory.
    If memory consumption becomes too high, the process may be killed. You may want to try disabling shape inference in that case.
[I]     Total Nodes | Original:  6216, After Folding:  6152 |    64 Nodes Folded
[I] Folding Constants | Pass 3
[W] colored module is not installed, will not use colors when logging. To enable colors, please install the colored module: python3 -m pip install colored
[W] Inference failed. You may want to try enabling partitioning to see better results. Note: Error was:
No module named 'onnxruntime'
[!] Module: 'onnxruntime.tools.symbolic_shape_infer' is required but could not be imported.
    Note: Error was: No module named 'onnxruntime'
    You can set POLYGRAPHY_AUTOINSTALL_DEPS=1 in your environment variables to allow Polygraphy to automatically install missing modules.
[I]     Total Nodes | Original:  6152, After Folding:  6152 |     0 Nodes Folded
*** API error: POST: http://127.0.0.1:7860/api/predict {'error': 'LocalProtocolError', 'detail': '', 'body': '', 'errors': "Can't send data when our state is ERROR"}
    Traceback (most recent call last):
      File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\errors.py", line 162, in __call__
        await self.app(scope, receive, _send)
      File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\base.py", line 109, in __call__
        await response(scope, receive, send)
      File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\starlette\responses.py", line 270, in __call__
        async with anyio.create_task_group() as task_group:
      File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 662, in __aexit__
        raise exceptions[0]
      File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\starlette\responses.py", line 273, in wrap
        await func()
      File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\base.py", line 134, in stream_response
        return await super().stream_response(send)
      File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\starlette\responses.py", line 255, in stream_response
        await send(
      File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\errors.py", line 159, in _send
        await send(message)
      File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 512, in send
        output = self.conn.send(event)
      File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\h11\_connection.py", line 468, in send
        data_list = self.send_with_data_passthrough(event)
      File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\h11\_connection.py", line 483, in send_with_data_passthrough
        raise LocalProtocolError("Can't send data when our state is ERROR")
    h11._util.LocalProtocolError: Can't send data when our state is ERROR

---
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 428, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\uvicorn\middleware\proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
  File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\fastapi\applications.py", line 273, in __call__
    await super().__call__(scope, receive, send)
  File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\starlette\applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\errors.py", line 184, in __call__
    raise exc
  File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\base.py", line 109, in __call__
    await response(scope, receive, send)
  File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\starlette\responses.py", line 270, in __call__
    async with anyio.create_task_group() as task_group:
  File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 662, in __aexit__
    raise exceptions[0]
  File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\starlette\responses.py", line 273, in wrap
    await func()
  File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\base.py", line 134, in stream_response
    return await super().stream_response(send)
  File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\starlette\responses.py", line 255, in stream_response
    await send(
  File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\errors.py", line 159, in _send
    await send(message)
  File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 512, in send
    output = self.conn.send(event)
  File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\h11\_connection.py", line 468, in send
    data_list = self.send_with_data_passthrough(event)
  File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\h11\_connection.py", line 483, in send_with_data_passthrough
    raise LocalProtocolError("Can't send data when our state is ERROR")
h11._util.LocalProtocolError: Can't send data when our state is ERROR
'AsyncRequest' object has no attribute '_json_response_data'
Applying attention optimization: sdp-no-mem... done.
Model loaded in 112.6s (load weights from disk: 15.3s, create model: 0.5s, apply weights to model: 96.5s).

[4lt3r3go]

i also foudn that if i remove the models i converted then the extention disappear from the UI.
restoring those deleted files will make the extention appear again... wtf lol

[Jonseed]

Ok, I'm going to delete everything in the models/Unet-trt and models/Unet-onnx folders, reboot webui server, and try exporting default engine again. Hopefully this time the model.json file is created correctly.

[Jonseed]

Export finished, and properly added to SD Unet dropdown. I can now use the TensorRT model, and it is indeed much faster on my 3060.

So it seems my errors were mostly caused by --medvram (which I still have not re-enabled), possibly conflicting with another extension (Photoshop plugin) using the --api. I had to manually uninstall cudnn via python -m pip uninstall -y nvidia-cudnn-cu11.

After all that, it seems to be working. I will try to reenable --medvram and --api commandline args and see if it still works.

[andreacostigliolo]

If I change the vae before generating image it seems to work.
BTW it's not faster :(

[Jonseed]

And then when I try to export the default engine, I get this error in the console:

{'sample': [(1, 4, 64, 64), (2, 4, 64, 64), (8, 4, 96, 96)], 'timesteps': [(1,), (2,), (8,)], 'encoder_hidden_states': [(1, 77, 768), (2, 77, 768), (8, 154, 768)]}
Disabling attention optimization
============= Diagnostic Run torch.onnx.export version 2.0.1+cu118 =============
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================

ERROR:root:Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)
Traceback (most recent call last):
  File "D:\repos\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\exporter.py", line 84, in export_onnx
    torch.onnx.export(
  File "d:\repos\stable-diffusion-webui\venv\lib\site-packages\torch\onnx\utils.py", line 506, in export
    _export(
  File "d:\repos\stable-diffusion-webui\venv\lib\site-packages\torch\onnx\utils.py", line 1548, in _export
    graph, params_dict, torch_out = _model_to_graph(
  File "d:\repos\stable-diffusion-webui\venv\lib\site-packages\torch\onnx\utils.py", line 1113, in _model_to_graph
    graph, params, torch_out, module = _create_jit_graph(model, args)
  File "d:\repos\stable-diffusion-webui\venv\lib\site-packages\torch\onnx\utils.py", line 989, in _create_jit_graph
    graph, torch_out = _trace_and_get_graph_from_model(model, args)
  File "d:\repos\stable-diffusion-webui\venv\lib\site-packages\torch\onnx\utils.py", line 893, in _trace_and_get_graph_from_model
    trace_graph, torch_out, inputs_states = torch.jit._get_trace_graph(
  File "d:\repos\stable-diffusion-webui\venv\lib\site-packages\torch\jit\_trace.py", line 1268, in _get_trace_graph
    outs = ONNXTracedModule(f, strict, _force_outplace, return_inputs, _return_inputs_states)(*args, **kwargs)
  File "d:\repos\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "d:\repos\stable-diffusion-webui\venv\lib\site-packages\torch\jit\_trace.py", line 127, in forward
    graph, out = torch._C._create_graph_by_tracing(
  File "d:\repos\stable-diffusion-webui\venv\lib\site-packages\torch\jit\_trace.py", line 118, in wrapper
    outs.append(self.inner(*trace_inputs))
  File "d:\repos\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "d:\repos\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1488, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "D:\repos\stable-diffusion-webui\modules\sd_unet.py", line 91, in UNetModel_forward
    return ldm.modules.diffusionmodules.openaimodel.copy_of_UNetModel_forward_for_webui(self, x, timesteps, context, *args, **kwargs)
  File "D:\repos\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\openaimodel.py", line 789, in forward
    emb = self.time_embed(t_emb)
  File "d:\repos\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "d:\repos\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1488, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "d:\repos\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\container.py", line 217, in forward
    input = module(input)
  File "d:\repos\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "d:\repos\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1488, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "D:\repos\stable-diffusion-webui\extensions-builtin\Lora\networks.py", line 429, in network_Linear_forward
    return originals.Linear_forward(self, input)
  File "d:\repos\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\linear.py", line 114, in forward
    return F.linear(input, self.weight, self.bias)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "d:\repos\stable-diffusion-webui\venv\lib\site-packages\gradio\routes.py", line 488, in run_predict
    output = await app.get_blocks().process_api(
  File "d:\repos\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1431, in process_api
    result = await self.call_function(
  File "d:\repos\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1117, in call_function
    prediction = await utils.async_iteration(iterator)
  File "d:\repos\stable-diffusion-webui\venv\lib\site-packages\gradio\utils.py", line 350, in async_iteration
    return await iterator.__anext__()
  File "d:\repos\stable-diffusion-webui\venv\lib\site-packages\gradio\utils.py", line 343, in __anext__
    return await anyio.to_thread.run_sync(
  File "d:\repos\stable-diffusion-webui\venv\lib\site-packages\anyio\to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "d:\repos\stable-diffusion-webui\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "d:\repos\stable-diffusion-webui\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "d:\repos\stable-diffusion-webui\venv\lib\site-packages\gradio\utils.py", line 326, in run_sync_iterator_async
    return next(iterator)
  File "d:\repos\stable-diffusion-webui\venv\lib\site-packages\gradio\utils.py", line 695, in gen_wrapper
    yield from f(*args, **kwargs)
  File "D:\repos\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\ui_trt.py", line 154, in export_unet_to_trt
    export_onnx(
  File "D:\repos\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\exporter.py", line 129, in export_onnx
    exit()
  File "C:\Users\X\AppData\Local\Programs\Python\Python310\lib\_sitebuiltins.py", line 26, in __call__
    raise SystemExit(code)
SystemExit: None

[MorkTheOrk]

Hello!

What packages have you installed? Anything updated from auto1111 default packages?
Your 2nd error might be due to the failed installation.

[Jonseed]

Here are my other installed extensions. Pretty standard stuff. Otherwise, it is all default. (I also already updated my NVIDIA drivers to 537.58 too. I have a RTX 3060.)

[Jonseed]

I should mention I'm on the latest version of Auto1111:
version: v1.6.0
python: 3.10.6
torch: 2.0.1+cu118
xformers: 0.0.20
gradio: 3.41.2
checkpoint: 1240e811e2

[MorkTheOrk]

We also used the lastest version of Auto1111.

Does the first error occur again when you restart the webui?
Have you tried manually installing protobuf?

[Jonseed]

If I restart the webui, I get the popups again, and this in the console:

Requirement already satisfied: protobuf==3.20.2 in .\venv\lib\site-packages (3.20.2)
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: onnx-graphsurgeon in .\venv\lib\site-packages (0.3.27)
Requirement already satisfied: onnx in .\venv\lib\site-packages (from onnx-graphsurgeon) (1.14.1)
Requirement already satisfied: numpy in .\venv\lib\site-packages (from onnx-graphsurgeon) (1.23.5)
Requirement already satisfied: protobuf>=3.20.2 in .\venv\lib\site-packages (from onnx->onnx-graphsurgeon) (3.20.2)
Requirement already satisfied: typing-extensions>=3.6.2.1 in .\venv\lib\site-packages (from onnx->onnx-graphsurgeon) (4.5.0)
GS is not installed! Installing...
Installing protobuf
Installing onnx-graphsurgeon
UI Config not initialized

I have not tried manually installing protobuf.

[MorkTheOrk]

You can see the packages that a required and how to install them in the install.py in the root dir.

The pop up errors are due to PyTorch having their own cudnn libraries, but TensorRT requires them aswell and installs them too. We remove that package after installing the tensorrt wheel. see here:
https://github.com/NVIDIA/Stable-Diffusion-WebUI-TensorRT/blob/d8eee382158cd7373e44fe7ac9265aa985a8fd29/install.py#L5C5-L10C1

[Jonseed]

Should I delete the extension and try to install it again?

[MorkTheOrk]

No you shouldn't, that's why we have the install.py that auto1111 runs. It might have failed to uninstall... Can you also sent me a pip freeze ?

[MorkTheOrk]

Should I delete the extension and try to install it again?

You can, maybe it makes a difference.

[Altrue]

I have the same problem. Same error messages upon restarting, same console. I installed automatic 1.52 since it says 1.5 is supported, not 1.6. RTX 3090.
EDIT: Turns out Automatic 1.5 instead of 1.6 is not relevant.

[Jonseed]

Ok, I deleted the extension folder, and restarted webui. Then I tried reinstalling the extension. I did not get the console error messages this time about protobuf (probably because it is already installed). Just this:

Requirement already satisfied: protobuf==3.20.2 in .\venv\lib\site-packages (3.20.2)
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: onnx-graphsurgeon in .\venv\lib\site-packages (0.3.27)
Requirement already satisfied: numpy in .\venv\lib\site-packages (from onnx-graphsurgeon) (1.23.5)
Requirement already satisfied: onnx in .\venv\lib\site-packages (from onnx-graphsurgeon) (1.14.1)
Requirement already satisfied: typing-extensions>=3.6.2.1 in .\venv\lib\site-packages (from onnx->onnx-graphsurgeon) (4.5.0)
Requirement already satisfied: protobuf>=3.20.2 in .\venv\lib\site-packages (from onnx->onnx-graphsurgeon) (3.20.2)
GS is not installed! Installing...
Installing protobuf
Installing onnx-graphsurgeon
UI Config not initialized

Then when I restarted the webui, I got the popup errors again, the same console messages above, and when trying to export the default engine I get the same console error: RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0!

[Jonseed]

Tried running that, and got this:

WARNING: Ignoring invalid distribution -rotobuf (d:\repos\stable-diffusion-webui\venv\lib\site-packages)
WARNING: Ignoring invalid distribution -rotobuf (d:\repos\stable-diffusion-webui\venv\lib\site-packages)
Found existing installation: nvidia-cudnn-cu11 8.9.4.25
Uninstalling nvidia-cudnn-cu11-8.9.4.25:
  Successfully uninstalled nvidia-cudnn-cu11-8.9.4.25
ERROR: Exception:
Traceback (most recent call last):
  File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\pip\_internal\cli\base_command.py", line 167, in exc_logging_wrapper
    status = run_func(*args)
  File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\pip\_internal\commands\uninstall.py", line 103, in run
    uninstall_pathset.commit()
  File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\pip\_internal\req\req_uninstall.py", line 424, in commit
    self._moved_paths.commit()
  File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\pip\_internal\req\req_uninstall.py", line 277, in commit
    save_dir.cleanup()
  File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\pip\_internal\utils\temp_dir.py", line 173, in cleanup
    rmtree(self._path)
  File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\pip\_vendor\tenacity\__init__.py", line 326, in wrapped_f
    return self(f, *args, **kw)
  File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\pip\_vendor\tenacity\__init__.py", line 406, in __call__
    do = self.iter(retry_state=retry_state)
  File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\pip\_vendor\tenacity\__init__.py", line 362, in iter
    raise retry_exc.reraise()
  File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\pip\_vendor\tenacity\__init__.py", line 195, in reraise
    raise self.last_attempt.result()
  File "C:\Users\X\AppData\Local\Programs\Python\Python310\lib\concurrent\futures\_base.py", line 451, in result
    return self.__get_result()
  File "C:\Users\X\AppData\Local\Programs\Python\Python310\lib\concurrent\futures\_base.py", line 403, in __get_result
    raise self._exception
  File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\pip\_vendor\tenacity\__init__.py", line 409, in __call__
    result = fn(*args, **kwargs)
  File "D:\repos\stable-diffusion-webui\venv\lib\site-packages\pip\_internal\utils\misc.py", line 128, in rmtree
    shutil.rmtree(dir, ignore_errors=ignore_errors, onerror=rmtree_errorhandler)
  File "C:\Users\X\AppData\Local\Programs\Python\Python310\lib\shutil.py", line 749, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Users\X\AppData\Local\Programs\Python\Python310\lib\shutil.py", line 614, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Users\X\AppData\Local\Programs\Python\Python310\lib\shutil.py", line 619, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Users\X\AppData\Local\Programs\Python\Python310\lib\shutil.py", line 617, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 5] Access is denied: 'D:\\repos\\stable-diffusion-webui\\venv\\Lib\\site-packages\\nvidia\\~udnn\\bin\\cudnn64_8.dll'

[Jonseed]

Restarted webui, and no more popup errors, but still get this when trying to export default engine: RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)

[Jonseed]

If I do a pip freeze now, I see this in the console:

WARNING: Ignoring invalid distribution -vidia-cudnn-cu11 (d:\repos\stable-diffusion-webui\venv\lib\site-packages)
WARNING: Ignoring invalid distribution -rotobuf (d:\repos\stable-diffusion-webui\venv\lib\site-packages)

And here is the current state of the pip freeze:

absl-py==1.4.0
accelerate==0.21.0
addict==2.4.0
aenum==3.1.12
aiofiles==23.2.1
aiohttp==3.8.4
aiosignal==1.3.1
altair==5.0.0
antlr4-python3-runtime==4.9.3
anyio==3.6.2
async-timeout==4.0.2
attrs==23.1.0
basicsr==1.4.2
beautifulsoup4==4.12.2
blendmodes==2022
blis==0.7.9
boltons==23.0.0
Brotli==1.1.0
cachetools==5.3.0
catalogue==2.0.8
certifi==2023.5.7
cffi==1.15.1
chardet==4.0.0
charset-normalizer==3.1.0
clean-fid==0.1.35
click==8.1.7
clip @ git+https://github.com/openai/CLIP.git@d50d76daa670286dd6cacf3bcd80b5e4823fc8e1
colorama==0.4.6
confection==0.0.4
contourpy==1.0.7
cssselect2==0.7.0
cycler==0.11.0
cymem==2.0.7
deprecation==2.1.0
duckduckgo-search==3.9.3
dynamicprompts==0.29.0
einops==0.4.1
facexlib==0.3.0
fastapi==0.94.0
ffmpy==0.3.0
filelock==3.12.0
filterpy==1.4.5
flatbuffers==23.5.9
font-roboto==0.0.1
fonts==0.0.3
fonttools==4.39.4
freetype-py==2.3.0
frozenlist==1.3.3
fsspec==2023.5.0
ftfy==6.1.1
future==0.18.3
fvcore==0.1.5.post20221221
gdown==4.7.1
gfpgan==1.3.8
gitdb==4.0.10
GitPython==3.1.32
google-auth==2.18.1
google-auth-oauthlib==1.0.0
gradio==3.41.2
gradio_client==0.5.0
grpcio==1.54.2
h11==0.12.0
h2==4.1.0
hpack==4.0.0
httpcore==0.15.0
httpx==0.24.1
huggingface-hub==0.14.1
hyperframe==6.0.1
idna==2.10
imageio==2.28.1
importlib-resources==6.0.1
inflection==0.5.1
iopath==0.1.9
Jinja2==3.1.2
jsonmerge==1.8.0
jsonschema==4.17.3
kiwisolver==1.4.4
kornia==0.6.7
langcodes==3.3.0
lark==1.1.2
lazy_loader==0.2
lightning-utilities==0.8.0
linkify-it-py==2.0.2
llvmlite==0.40.0
lmdb==1.4.1
lpips==0.1.4
lxml==4.9.3
Markdown==3.4.3
markdown-it-py==2.2.0
MarkupSafe==2.1.2
matplotlib==3.7.1
mdit-py-plugins==0.3.3
mdurl==0.1.2
mediapipe==0.10.5
mpmath==1.3.0
multidict==6.0.4
murmurhash==1.0.9
networkx==3.1
numba==0.57.0
numpy==1.23.5
nvidia-cublas-cu11==11.11.3.6
nvidia-cuda-nvrtc-cu11==11.8.89
nvidia-cuda-runtime-cu11==11.8.89
oauthlib==3.2.2
omegaconf==2.2.3
onnx==1.14.1
onnx-graphsurgeon==0.3.27
open-clip-torch==2.20.0
opencv-contrib-python==4.7.0.72
opencv-python==4.8.0.76
orjson==3.8.12
packaging==23.1
pandas==2.0.1
pathy==0.10.1
piexif==1.1.3
Pillow==9.5.0
polygraphy==0.49.0
portalocker==2.7.0
preshed==3.0.8
protobuf==3.20.2
psutil==5.9.5
py-cpuinfo==9.0.0
pyasn1==0.5.0
pyasn1-modules==0.3.0
pycairo==1.23.0
pycparser==2.21
pydantic==1.10.7
pydub==0.25.1
Pygments==2.15.1
pyparsing==3.0.9
pyrsistent==0.19.3
PySocks==1.7.1
python-dateutil==2.8.2
python-multipart==0.0.6
pytorch-lightning==1.9.4
pytz==2023.3
PyWavelets==1.4.1
pywin32==306
PyYAML==6.0
realesrgan==0.3.0
regex==2023.5.5
reportlab==4.0.0
requests==2.31.0
requests-oauthlib==1.3.1
resize-right==0.0.2
rich==13.6.0
rlPyCairo==0.2.0
rsa==4.9
safetensors==0.3.1
scikit-image==0.21.0
scipy==1.10.1
seaborn==0.13.0
semantic-version==2.10.0
Send2Trash==1.8.0
sentencepiece==0.1.99
six==1.16.0
smart-open==6.3.0
smmap==5.0.0
sniffio==1.3.0
socksio==1.0.0
sounddevice==0.4.6
soupsieve==2.4.1
spacy==3.5.3
spacy-legacy==3.0.12
spacy-loggers==1.0.4
srsly==2.4.6
starlette==0.26.1
svglib==1.5.1
sympy==1.12
tabulate==0.9.0
tb-nightly==2.14.0a20230520
tensorboard-data-server==0.7.0
tensorrt==9.0.1.post11.dev4
tensorrt-bindings==9.0.1.post11.dev4
tensorrt-libs==9.0.1.post11.dev4
termcolor==2.3.0
thinc==8.1.10
thop==0.1.1.post2209072238
tifffile==2023.4.12
timm==0.9.2
tinycss2==1.2.1
tokenizers==0.13.3
tomesd==0.1.3
tomli==2.0.1
toolz==0.12.0
torch==2.0.1+cu118
torchdiffeq==0.2.3
torchmetrics==0.11.4
torchsde==0.2.5
torchvision==0.15.2+cu118
tqdm==4.65.0
trampoline==0.1.2
transformers==4.30.2
typer==0.7.0
typing_extensions==4.5.0
tzdata==2023.3
uc-micro-py==1.0.2
ultralytics==8.0.195
urllib3==1.26.15
uvicorn==0.22.0
wasabi==1.1.1
wcwidth==0.2.6
webencodings==0.5.1
websockets==11.0.3
Werkzeug==2.3.4
xformers==0.0.20
yacs==0.1.8
yapf==0.33.0
yarl==1.9.2

What is that about "-vidia-cudnn-cu11" being an invalid distribution?

[andreacostigliolo]

with python -m pip uninstall -y nvidia-cudnn-cu11 I also don't get more the error on startup. But I still get
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0!

[MorkTheOrk]

That looks like a different error. I will figure something out so cudnn is being uninstalled even if tensorrt is installed.

[andreacostigliolo]

it's the same error of Jonseed

[MorkTheOrk]

I mean this error seems like a different one than the cudnn pop ups:

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0!

[Jonseed]

So it appears like I no longer have nvidia-cudnn-cu11 installed, per the pip freeze... Is that what we want?

[niffelheim87]

Popups errors solved with python -m pip uninstall -y nvidia-cudnn-cu11 but still get
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)
when trying to generate images

[DennLie]

since i struggle with this error myself and tried various things. this is what i found out by now:
delete and rebuild venv - no change
deactivate all extensions and reinstall tensorrt extension - no change
deactivate all COMMANDLINE_ARGS - Error is gone and it builded the TensorRT engine
This were my COMMANDLINE_ARGS before = --api --xformers --medvram-sdxl --opt-sdp-attention --medvram --disable-safe-unpickle
so i guess that one of them was causing the problem, i will get them now one by one back and check which one caused the error.

[Jonseed]

I can still generate images without errors. It is only when I try to export the default engine that I get the erorr about expecting all tensors to be on the same device...

[MorkTheOrk]

So it appears like I no longer have nvidia-cudnn-cu11 installed, per the pip freeze... Is that what we want?

Yes

[Jonseed]

my commandline args are: --autolaunch --update-check --xformers --no-half-vae --medvram --api

[MorkTheOrk]

Can you try without the xformers arg?

[ByblosHex]

Same errors as OP.

[MorkTheOrk]

It might be that VRAM limitations fall back to CPU tensors and therefore are not working with the Torch ONNX export.

[Jonseed]

I removed xformers arg and when exporting default engine still get error: RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)

Will try removing medvram arg...

[ByblosHex]

Nothing under SD Unet after generating engine completed?

[MorkTheOrk]

For my part, the python -m pip uninstall -y nvidia-cudnn-cu11 didn't seem to work as the extension was "not installed".

So instead I went to venv\Lib\site-packages and I removed the cudnn.dist-info & the cudnn folder in the nvidia folder. It seems to be working fine. At least, I can start without errors, and I can start generating the engine. It seems to be generating without issues now.

Have you exported the engine and enabled it? See here: https://github.com/NVIDIA/Stable-Diffusion-WebUI-TensorRT#how-to-use

[andreacostigliolo]

but after exporting should I use medvram again?

[niffelheim87]

I'm the only one when generate images with already selected engine gets this error?

[Aeo2000]

I'm the only one when generate images with already selected engine gets this error?

I'm getting the same errors. Tried deleting venv folder, extensions folder, getting rid of any arguements in the web batch file. now the tab for tensorrt doesnt even appear

[ByblosHex]

Following the steps in the discussion able to build an engine but it errors out at the saving step

[ByblosHex]

For my part, the python -m pip uninstall -y nvidia-cudnn-cu11 didn't seem to work as the extension was "not installed".
So instead I went to venv\Lib\site-packages and I removed the cudnn.dist-info & the cudnn folder in the nvidia folder. It seems to be working fine. At least, I can start without errors, and I can start generating the engine. It seems to be generating without issues now.

Have you exported the engine and enabled it? See here: https://github.com/NVIDIA/Stable-Diffusion-WebUI-TensorRT#how-to-use

Yes, I was currently following the instructions actually, to see if "seems to be working" could be converted into "is working". I can confirm it works flawlessly! I was able to export an engine and confirm that it increases my generation speed by ~40%!

In summary, for me the issue was that during the installation, somehow it didn't uninstall cudnn, and cudnn was taking priority over the dlls in tensorrt. After removing cudnn manually, my problem was solved.

What did you remove?

[andreacostigliolo]

So now I can export model (with some out of memory errors on 12GB), but when I try to use I get again this error :
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm).
Thi error I got exporting with medvram but now I don't have it on startup

[Jonseed]

I thought I might try uninstalling cudnn manually as @Altrue did, but when I go to venv\Lib\site-packages I don't see folders for cudnn.dist-info & cudnn folders in the nvidia folder. Here is what I have in the nvidia folder:

[Jonseed]

It seems my error now might be this: No module named 'onnxruntime'

[Jonseed]

Should I try manually installing onnxruntime in the venv per python.exe -m pip install onnxruntime?

[aria1th]

onnxruntime-gpu afaik, but it should be added in install.py if its the case

[Jonseed]

I already have onnx and onnx-graphsurgeon installed. Do I need onnxruntime or onnxruntime-gpu?

[Jonseed]

After all that, I just restarted the webui, and tried exporting the default engine again, and now it seems to be exporting... I did not install onnxruntime or onnxruntime-gpu...

It still output this error to the console, even though it continued with exporting this time.

ERROR:asyncio:Exception in callback H11Protocol.timeout_keep_alive_handler()
handle: <TimerHandle when=8572.921 H11Protocol.timeout_keep_alive_handler()>
Traceback (most recent call last):
  File "d:\repos\stable-diffusion-webui\venv\lib\site-packages\h11\_state.py", line 249, in _fire_event_triggered_transitions
    new_state = EVENT_TRIGGERED_TRANSITIONS[role][state][event_type]
KeyError: <class 'h11._events.ConnectionClosed'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\X\AppData\Local\Programs\Python\Python310\lib\asyncio\events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "d:\repos\stable-diffusion-webui\venv\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 383, in timeout_keep_alive_handler
    self.conn.send(event)
  File "d:\repos\stable-diffusion-webui\venv\lib\site-packages\h11\_connection.py", line 468, in send
    data_list = self.send_with_data_passthrough(event)
  File "d:\repos\stable-diffusion-webui\venv\lib\site-packages\h11\_connection.py", line 493, in send_with_data_passthrough
    self._process_event(self.our_role, event)
  File "d:\repos\stable-diffusion-webui\venv\lib\site-packages\h11\_connection.py", line 242, in _process_event
    self._cstate.process_event(role, type(event), server_switch_event)
  File "d:\repos\stable-diffusion-webui\venv\lib\site-packages\h11\_state.py", line 238, in process_event
    self._fire_event_triggered_transitions(role, event_type)
  File "d:\repos\stable-diffusion-webui\venv\lib\site-packages\h11\_state.py", line 251, in _fire_event_triggered_transitions
    raise LocalProtocolError(
h11._util.LocalProtocolError: can't handle event type ConnectionClosed when role=SERVER and state=SEND_RESPONSE

Maybe this error is from another extension I have active that is using the API (the Photoshop plugin)?

[4lt3r3go]

I'm the only one when generate images with already selected engine gets this error?

I'm getting the same errors. Tried deleting venv folder, extensions folder, getting rid of any arguements in the web batch file. now the tab for tensorrt doesnt even appear

SAME. tab disappeared, no matter what i do reinstall, upgrade requirement. nothing

[Terroosh]

I was having the "procedure entry point" error popups as well but deleting the cudnn folder in venv\Lib\site-packages\nvidia fixed it. I'm able to generate images now.

[Tonic0]

I'm the only one when generate images with already selected engine gets this error?

I'm getting the same errors. Tried deleting venv folder, extensions folder, getting rid of any arguements in the web batch file. now the tab for tensorrt doesnt even appear

Same issue here. Removed TensorRT extension, deleted venv folder. Started WebUI, venv reinstalled, reinstalled TensorRT extension, now it doesn't appear in UI anymore. What's the procedure to fully uninstall/reinstall TensortRT without deleting the A1111 WebUI completely?

Error:

Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: onnx-graphsurgeon in d:\scratch\stable-diffusion-webui\venv\lib\site-packages (0.3.27)
Requirement already satisfied: numpy in d:\scratch\stable-diffusion-webui\venv\lib\site-packages (from onnx-graphsurgeon) (1.23.5)
Requirement already satisfied: onnx in d:\scratch\stable-diffusion-webui\venv\lib\site-packages (from onnx-graphsurgeon) (1.14.1)
Requirement already satisfied: protobuf>=3.20.2 in d:\scratch\stable-diffusion-webui\venv\lib\site-packages (from onnx->onnx-graphsurgeon) (3.20.2)
Requirement already satisfied: typing-extensions>=3.6.2.1 in d:\scratch\stable-diffusion-webui\venv\lib\site-packages (from onnx->onnx-graphsurgeon) (4.8.0)
GS is not installed! Installing...
Installing protobuf
Installing onnx-graphsurgeon
UI Config not initialized
Launching Web UI with arguments:
No module 'xformers'. Proceeding without it.
*** Error loading script: trt.py
    Traceback (most recent call last):
      File "D:\Scratch\stable-diffusion-webui\modules\scripts.py", line 382, in load_scripts
        script_module = script_loading.load_module(scriptfile.path)
      File "D:\Scratch\stable-diffusion-webui\modules\script_loading.py", line 10, in load_module
        module_spec.loader.exec_module(module)
      File "<frozen importlib._bootstrap_external>", line 883, in exec_module
      File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
      File "D:\Scratch\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\scripts\trt.py", line 10, in <module>
        import ui_trt
      File "D:\Scratch\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\ui_trt.py", line 10, in <module>
        from exporter import export_onnx, export_trt
      File "D:\Scratch\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\exporter.py", line 10, in <module>
        from utilities import Engine
      File "D:\Scratch\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\utilities.py", line 32, in <module>
        import tensorrt as trt
      File "D:\Scratch\stable-diffusion-webui\venv\lib\site-packages\tensorrt\__init__.py", line 18, in <module>
        from tensorrt_bindings import *
    ModuleNotFoundError: No module named 'tensorrt_bindings'

[Aeo2000]

I'm the only one when generate images with already selected engine gets this error?

I'm getting the same errors. Tried deleting venv folder, extensions folder, getting rid of any arguements in the web batch file. now the tab for tensorrt doesnt even appear

SAME. tab disappeared, no matter what i do reinstall, upgrade requirement. nothing

same i been trying to fix this the past hour. anyone find any fixes yet?

[racerx2oo3]

I'm the only one when generate images with already selected engine gets this error?

I'm getting the same errors. Tried deleting venv folder, extensions folder, getting rid of any arguements in the web batch file. now the tab for tensorrt doesnt even appear

Same issue here. Removed TensorRT extension, deleted venv folder. Started WebUI, venv reinstalled, reinstalled TensorRT extension, now it doesn't appear in UI anymore. What's the procedure to fully uninstall/reinstall TensortRT without deleting the A1111 WebUI completely?

Error:

Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: onnx-graphsurgeon in d:\scratch\stable-diffusion-webui\venv\lib\site-packages (0.3.27)
Requirement already satisfied: numpy in d:\scratch\stable-diffusion-webui\venv\lib\site-packages (from onnx-graphsurgeon) (1.23.5)
Requirement already satisfied: onnx in d:\scratch\stable-diffusion-webui\venv\lib\site-packages (from onnx-graphsurgeon) (1.14.1)
Requirement already satisfied: protobuf>=3.20.2 in d:\scratch\stable-diffusion-webui\venv\lib\site-packages (from onnx->onnx-graphsurgeon) (3.20.2)
Requirement already satisfied: typing-extensions>=3.6.2.1 in d:\scratch\stable-diffusion-webui\venv\lib\site-packages (from onnx->onnx-graphsurgeon) (4.8.0)
GS is not installed! Installing...
Installing protobuf
Installing onnx-graphsurgeon
UI Config not initialized
Launching Web UI with arguments:
No module 'xformers'. Proceeding without it.
*** Error loading script: trt.py
    Traceback (most recent call last):
      File "D:\Scratch\stable-diffusion-webui\modules\scripts.py", line 382, in load_scripts
        script_module = script_loading.load_module(scriptfile.path)
      File "D:\Scratch\stable-diffusion-webui\modules\script_loading.py", line 10, in load_module
        module_spec.loader.exec_module(module)
      File "<frozen importlib._bootstrap_external>", line 883, in exec_module
      File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
      File "D:\Scratch\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\scripts\trt.py", line 10, in <module>
        import ui_trt
      File "D:\Scratch\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\ui_trt.py", line 10, in <module>
        from exporter import export_onnx, export_trt
      File "D:\Scratch\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\exporter.py", line 10, in <module>
        from utilities import Engine
      File "D:\Scratch\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\utilities.py", line 32, in <module>
        import tensorrt as trt
      File "D:\Scratch\stable-diffusion-webui\venv\lib\site-packages\tensorrt\__init__.py", line 18, in <module>
        from tensorrt_bindings import *
    ModuleNotFoundError: No module named 'tensorrt_bindings'

Deleting the stable-diffusion-webui-tensorrt extension in the extensions folder should be all you need to do.

[Jonseed]

So it finally exported the default engine for SD 1.5, and I see it in models\Unet-trt, but now SD Unet doesn't show it in the dropdown menu. It only shows automatic or none... Refreshing that list does nothing. I have tried restarting the UI. I will now restart webui server to see if that helps.

[racerx2oo3]

So it finally exported the default engine for SD 1.5, and I see it in models\Unet-trt, but now SD Unet doesn't show it in the dropdown menu. It only shows automatic or none... Refreshing that list does nothing. I have tried restarting the UI. I will now restart webui server to see if that helps.

Do you have a model file in the models/Unet-trt folder? If so, it might be that the model.json file didn't get properly written. I had that happen at one point.

If you open the model.json file and it is empty than that is the issue.

[Jonseed]

Yes, I have a model file in the models/Unet-trt folder, but the model.json file is empty, just {}. How do I regenerate the model.json file? Do I have to export the default engine again?

[racerx2oo3]

Yes, I have a model file in the models/Unet-trt folder, but the model.json file is empty, just {}. How do I regenerate the model.json file? Do I have to export the default engine again?

Normally you'd need to rebuild the profile, I don't think there's a clear way to regenerate the model.json file.

You can try deleting everything in the model.json file and pasting in this
{ "cc89": { "v1-5-pruned-emaonly": [ { "filepath": "v1-5-pruned-emaonly_d7049739_cc89_sample=1x4x64x64+2x4x64x64+8x4x96x96-timesteps=1+2+8-encoder_hidden_states=1x77x768+2x77x768+8x154x768.trt", "config": { "profile": { "sample": [ [ 1, 4, 64, 64 ], [ 2, 4, 64, 64 ], [ 8, 4, 96, 96 ] ], "timesteps": [ [ 1 ], [ 2 ], [ 8 ] ], "encoder_hidden_states": [ [ 1, 77, 768 ], [ 2, 77, 768 ], [ 8, 154, 768 ] ] }, "static_shapes": false, "fp32": false, "inpaint": false, "refit": true, "lora": false, "vram": 0, "unet_hidden_dim": 4 } } ] } }

[Jonseed]

Hmm, I deleted everything in the model.json file, and pasted in that, and still nothing in the dropdown. I restarted UI, still nothing. Will now restart webui server. After restarting, and refreshing the dropdown menu, still nothing. It looks like I may have to export the default engine again? Should I delete everything in the models/Unet-trt folder? What about the files in models/Unet-onnx folder?

[4lt3r3go]

what is this now

[racerx2oo3]

what is this now

Looks like you specified settings in your image generation that aren't supported by an engine you have built. What are you specifying for your image generation parameters?

[ByblosHex]

I have two models in the folder, but the .json is empty as well.

[4lt3r3go]

what is this now

Looks like you specified settings in your image generation that aren't supported by an engine you have built. What are you specifying for your image generation parameters?

this sounds more like that the model i converted, wich was a custom size (512x768 batch 1) wasnt working as expected. So now i'm redoing it using default settings. i already used TensorRT a lot in the past, and i remeber that there were some trial and error in converting models.. maybe this is the case here.

[Jonseed]

Restarting the webui server, and got this error:

*** Error loading script: trt.py
    Traceback (most recent call last):
      File "D:\repos\stable-diffusion-webui\modules\scripts.py", line 382, in load_scripts
        script_module = script_loading.load_module(scriptfile.path)
      File "D:\repos\stable-diffusion-webui\modules\script_loading.py", line 10, in load_module
        module_spec.loader.exec_module(module)
      File "<frozen importlib._bootstrap_external>", line 883, in exec_module
      File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
      File "D:\repos\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\scripts\trt.py", line 10, in <module>
        import ui_trt
      File "D:\repos\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\ui_trt.py", line 16, in <module>
        from model_manager import modelmanager, cc_major, TRT_MODEL_DIR
      File "D:\repos\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\model_manager.py", line 219, in <module>
        modelmanager = ModelManager()
      File "D:\repos\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\model_manager.py", line 35, in __init__
        self.update()
      File "D:\repos\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\model_manager.py", line 67, in update
        for base_model, models in base_models.items():
    RuntimeError: dictionary changed size during iteration

Could be because we tried pasting in the values to model.json? Looks like it deleted everything in the model.json file when I restarted the webui server...

[ByblosHex]

In my Nvidia folder there is a blank _init.py file

[racerx2oo3]

Restarting the webui server, and got this error:

*** Error loading script: trt.py
    Traceback (most recent call last):
      File "D:\repos\stable-diffusion-webui\modules\scripts.py", line 382, in load_scripts
        script_module = script_loading.load_module(scriptfile.path)
      File "D:\repos\stable-diffusion-webui\modules\script_loading.py", line 10, in load_module
        module_spec.loader.exec_module(module)
      File "<frozen importlib._bootstrap_external>", line 883, in exec_module
      File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
      File "D:\repos\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\scripts\trt.py", line 10, in <module>
        import ui_trt
      File "D:\repos\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\ui_trt.py", line 16, in <module>
        from model_manager import modelmanager, cc_major, TRT_MODEL_DIR
      File "D:\repos\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\model_manager.py", line 219, in <module>
        modelmanager = ModelManager()
      File "D:\repos\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\model_manager.py", line 35, in __init__
        self.update()
      File "D:\repos\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\model_manager.py", line 67, in update
        for base_model, models in base_models.items():
    RuntimeError: dictionary changed size during iteration

Could be because we tried pasting in the values to model.json? Looks like it deleted everything in the model.json file when I restarted the webui server...

Could be, I wasn't really sure if that would work or not.

[Jonseed]

Exporting now... It is still reporting that ONNX-Runtime is not installed, and it looks like Polygraphy uses it, and it is giving a bunch of errors because of that, but the export continues. Do I want to manually install that module?

[racerx2oo3]

Exporting now... It is still reporting that ONNX-Runtime is not installed, and it looks like Polygraphy uses it, and it is giving a bunch of errors because of that, but the export continues. Do I want to manually install that module?

No it's fine. That error is "normal".

[andreacostigliolo]

with 12 gb I get W] Requested amount of GPU memory (11922309120 bytes) could not be allocated. There may not be enough free memory for allocation to succeed.
[W] UNSUPPORTED_STATESkipping tactic 0 due to insufficient memory on requested size of 11922309120 detected for tactic 0x0000000000000000. during exporting.

[Jonseed]

with 12 gb I get W] Requested amount of GPU memory (11922309120 bytes) could not be allocated. There may not be enough free memory for allocation to succeed. [W] UNSUPPORTED_STATESkipping tactic 0 due to insufficient memory on requested size of 11922309120 detected for tactic 0x0000000000000000. during exporting.

I also got that error on my 3060 with 12gb vram, but the export continued, and seems to work.

[racerx2oo3]

with 12 gb I get W] Requested amount of GPU memory (11922309120 bytes) could not be allocated. There may not be enough free memory for allocation to succeed. [W] UNSUPPORTED_STATESkipping tactic 0 due to insufficient memory on requested size of 11922309120 detected for tactic 0x0000000000000000. during exporting.

If the engine generation continues after getting that message, its fine. The engine build process will try different "tactics" to build the TensorRT engine and some use more memory than others, if a given tactic can't be used the build should continue with a different tactic. If the build fails to build after getting a memory error, then the specified engine is probably running out of VRAM.

[niffelheim87]

Tried this time with sd 1.5 instead of sdXL and now its working. sd 1.5 from 3-4it/s to 10/11it/s batch size 1

[Jonseed]

Ok, I re-enabled --medvram and --api, and the TensorRT model I built before seems to still work. I do get this message when I use it:

[W] CUDA lazy loading is not enabled. Enabling it can significantly reduce device memory usage and speed up TensorRT initialization. See "Lazy Loading" section of CUDA documentation https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#lazy-loading

Should I enable CUDA lazy loading? How would I do that?

With --medvram re-enabled, I cannot export any new engines though. I get this error as I did before:

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)

So something with the --medvram argument is conflicting here. I'll have to remove that commandline arg whenever I want to export new engines until fixed.

[racerx2oo3]

CUDA Lazy loading can help with memory usage, it's enabled using an environment variable: SET CUDA_MODULE_LOADING=LAZY

[racerx2oo3]

CUDA Lazy loading can help with memory usage, it's enabled using an environment variable: SET CUDA_MODULE_LOADING=LAZY

I'm not sure exactly how much it helps in this case, but you can give it a shot.

[andreacostigliolo]

Generating images with unet gives me this error : RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)

Any idea?

[racerx2oo3]

Generating images with unet gives me this error : RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)

Any idea?

Are you trying to use SDXL?

[Jonseed]

As far as performance goes on my 3060, using SD 1.5 at 512x512 resolution, for a batch size of 1, I'm getting speed increases from about 3.9 it/s to 9 it/s, an increase of 2.3x! That's impressive.

But if I increase the batch size, something strange happens. With TRT set to 'none', the generation takes way longer now. A batch size of 4, without TRT (SD Unet set to 'none'), takes a minute and 40 seconds (1:40), whereas with a batch size of 1, generating 4 images takes only 25 seconds when TRT is off. What is going on here? Something seems wrong. Is the TRT UNET not deactivated correctly when TRT is set to 'none' for batch sizes greater than 1? With TRT active, a batch size of 4 is very fast, generating 4 images in only 8 seconds.

[racerx2oo3]

As far as performance goes on my 3060, using SD 1.5 at 512x512 resolution, for a batch size of 1, I'm getting speed increases from about 3.9 it/s to 9 it/s, an increase of 2.3x! That's impressive.

But if I increase the batch size, something strange happens. With TRT set to 'none', the generation takes way longer now. A batch size of 4, without TRT (SD Unet set to 'none'), takes a minute and 40 seconds (1:40), whereas with a batch size of 1, generating 4 images takes only 25 seconds when TRT is off. What is going on here? Something seems wrong. Is the TRT UNET not deactivated correctly when TRT is set to 'none' for batch sizes greater than 1? With TRT active, a batch size of 4 is very fast, generating 4 images in only 8 seconds.

You're likely hitting the memory limits of your GPU. Your --medvram and --xformers command line arguments previously were directed at allowing you to use less VRAM for image generation. So without those active, you're probably hitting VRAM limitations that are slowing you down at larger batch sizes.

TensorRT is more memory efficient than vanilla PyTorch so it can generally handle larger batch sizes. If you want to run larger batch sizes without using TensorRT you'll likely need to re-enable your previous memory optimizations.

[andreacostigliolo]

Generating images with unet gives me this error : RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)
Any idea?

Are you trying to use SDXL?

yes

[Akyariss]

Generating images with unet gives me this error : RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)
Any idea?

Are you trying to use SDXL?

yes

I have the same issue with SDXL and TRT, works fine without

[Jonseed]

These performance numbers were with --medvram and --xformers enabled (although I'm using sdp-no-mem at the moment, not xformers).

If I disable the TensorRT extension, then my speeds are about 5 it/s with batch size 1 (4 images in 19 seconds), and 2.2 it/s with batch size of 4 (11 seconds). So even with a batch size of 1, the speeds are slower with the TensorRT extension enabled (5 it/s vs 3.9 it/s). So it seems there is a problem here...

[racerx2oo3]

Generating images with unet gives me this error : RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)
Any idea?

Are you trying to use SDXL?

yes

The extension supports SDXL, but it relies on functionality that hasn't been implemented in the release branch. If you need to work with SDXL you'll need to use a Automatic1111 build from the Dev branch at the moment. Note that the Dev branch is not intended for production work and may break other things that you are currently using.

[andreacostigliolo]

Generating images with unet gives me this error : RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)
Any idea?

Are you trying to use SDXL?

yes

I have the same issue with SDXL and TRT, works fine without

If i change the vae few times (enabled on the quick menù) it seems starting. Can't understand....

[alexbofa]

This is an error because it broke when downloading.
Copy from here Stable-Diffusion\venv\Lib\site-packages\torch\lib to here Stable-Diffusion\venv\Lib\site-packages\nvidia\cudnn\bin

[andreacostigliolo]

cudnn

I don't have cudnn folder.

[Aeo2000]

the TensorRT tab is still isn't appearing for me. After I uninstalled it the first time it no longer appears. Anyone facing the same issue?

i am seeing this in the command terminal, could that have something to do with tensorrt not appearing?

[Zw3pter]

Can anyone tell me whats wrong here? i tried to reinstall it multiple times and even tried to reinstall automatic1111 multiple times. I got it working for some odd reason on my old version, but i accidently deleted that.

Launching Web UI with arguments: --xformers
*** Error loading script: trt.py
Traceback (most recent call last):
File "E:\AIsoftware\stable-diffusion-webui\modules\scripts.py", line 382, in load_scripts
script_module = script_loading.load_module(scriptfile.path)
File "E:\AIsoftware\stable-diffusion-webui\modules\script_loading.py", line 10, in load_module
module_spec.loader.exec_module(module)
File "", line 883, in exec_module
File "", line 241, in call_with_frames_removed
File "E:\AIsoftware\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\scripts\trt.py", line 10, in
import ui_trt
File "E:\AIsoftware\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\ui_trt.py", line 10, in
from exporter import export_onnx, export_trt
File "E:\AIsoftware\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\exporter.py", line 10, in
from utilities import Engine
File "E:\AIsoftware\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\utilities.py", line 32, in
import tensorrt as trt
File "E:\AIsoftware\stable-diffusion-webui\venv\lib\site-packages\tensorrt_init.py", line 18, in
from tensorrt_bindings import *
ModuleNotFoundError: No module named 'tensorrt_bindings'

[verifiedsyckozis]

I got mine to export a model thing, but I get this every time I try to use it...
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)
I saw up there someone else had the same problem.

[racerx2oo3]

Can anyone tell me whats wrong here? i tried to reinstall it multiple times and even tried to reinstall automatic1111 multiple times. I got it working for some odd reason on my old version, but i accidently deleted that.

Launching Web UI with arguments: --xformers *** Error loading script: trt.py Traceback (most recent call last): File "E:\AIsoftware\stable-diffusion-webui\modules\scripts.py", line 382, in load_scripts script_module = script_loading.load_module(scriptfile.path) File "E:\AIsoftware\stable-diffusion-webui\modules\script_loading.py", line 10, in load_module module_spec.loader.exec_module(module) File "", line 883, in exec_module File "", line 241, in call_with_frames_removed File "E:\AIsoftware\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\scripts\trt.py", line 10, in import ui_trt File "E:\AIsoftware\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\ui_trt.py", line 10, in from exporter import export_onnx, export_trt File "E:\AIsoftware\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\exporter.py", line 10, in from utilities import Engine File "E:\AIsoftware\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\utilities.py", line 32, in import tensorrt as trt File "E:\AIsoftware\stable-diffusion-webui\venv\lib\site-packages\tensorrt__init_.py", line 18, in from tensorrt_bindings import * ModuleNotFoundError: No module named 'tensorrt_bindings'

Are you installing using the "Available" tab within extensions? If so that hasn't yet been updated to point to this current version. You'll want to delete the stable-diffusion-webui-tesnsorrt folder found in the extensions folder. And the install using the Install from URL option.

[racerx2oo3]

I got mine to export a model thing, but I get this every time I try to use it... RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm) I saw up there someone else had the same problem.

If you are trying to use SDXL then this is currently expected. The extension supports SDXL, but it relies on functionality that hasn't been implemented in the release branch of Automatic1111. If you need to work with SDXL you'll need to use a Automatic1111 build from the Dev branch at the moment. Note that the Dev branch is not intended for production work and may break other things that you are currently using.