Error loading script: trt.py, ModuleNotFoundError: No module named 'tensorrt_bindings'

[nvidia]

Error loading script: trt.py, ModuleNotFoundError: No module named 'tensorrt_bindings'

[racerx2oo3]

What appears to have worked for others.

From your base SD webui folder: (E:\Stable diffusion\SD\webui\ in your case).

• In the extensions folder delete: stable-diffusion-webui-tensorrt folder if it exists
• Delete the venv folder

Open a command prompt and navigate to the base SD webui folder

• Run webui.bat - this should rebuild the virtual environment venv
• When the WebUI appears close it and close the command prompt

Open a command prompt and navigate to the base SD webui folder

• enter: venv\Scripts\activate.bat
• the command line should now have (venv) shown at the beginning.
• enter the following commands:
• • python.exe -m pip install --upgrade pip
• • python -m pip install nvidia-cudnn-cu11==8.9.4.25 --no-cache-dir
• • python -m pip install --pre --extra-index-url https://pypi.nvidia.com/ tensorrt==9.0.1.post11.dev4 --no-cache-dir
• • python -m pip uninstall -y nvidia-cudnn-cu11
• • venv\Scripts\deactivate.bat
• webui.bat
• Install the TensorRT extension using the Install from URL option
• Once installed, go to the Extensions >> Installed tab and Apply and Restart

[rlenci99]

What appears to have worked for others.

From your base SD webui folder: (E:\Stable diffusion\SD\webui\ in your case).

• In the extensions folder delete: stable-diffusion-webui-tensorrt folder if it exists
• Delete the venv folder

Open a command prompt and navigate to the base SD webui folder

• Run webui.bat - this should rebuild the virtual environment venv
• When the WebUI appears close it and close the command prompt

Open a command prompt and navigate to the base SD webui folder

• enter: venv\Scripts\activate.bat
• the command line should now have (venv) shown at the beginning.
• enter the following commands:
• • python.exe -m pip install --upgrade pip
• • python -m pip install nvidia-cudnn-cu11==8.9.4.25 --no-cache-dir
• • python -m pip install --pre --extra-index-url https://pypi.nvidia.com/ tensorrt==9.0.1.post11.dev4 --no-cache-dir
• • python -m pip uninstall -y nvidia-cudnn-cu11
• • venv\Scripts\deactivate.bat
• webui.bat
• Install the TensorRT extension using the Install from URL option
• Once installed, go to the Extensions >> Installed tab and Apply and Restart

Thanks for the detailed steps, much appreciated. It worked!!

[Stas-ai]

What appears to have worked for others.
From your base SD webui folder: (E:\Stable diffusion\SD\webui\ in your case).

• In the extensions folder delete: stable-diffusion-webui-tensorrt folder if it exists
• Delete the venv folder

Open a command prompt and navigate to the base SD webui folder

• Run webui.bat - this should rebuild the virtual environment venv
• When the WebUI appears close it and close the command prompt

Open a command prompt and navigate to the base SD webui folder

• enter: venv\Scripts\activate.bat
• the command line should now have (venv) shown at the beginning.
• enter the following commands:
• • python.exe -m pip install --upgrade pip
• • python -m pip install nvidia-cudnn-cu11==8.9.4.25 --no-cache-dir
• • python -m pip install --pre --extra-index-url https://pypi.nvidia.com/ tensorrt==9.0.1.post11.dev4 --no-cache-dir
• • python -m pip uninstall -y nvidia-cudnn-cu11
• • venv\Scripts\deactivate.bat
• webui.bat
• Install the TensorRT extension using the Install from URL option
• Once installed, go to the Extensions >> Installed tab and Apply and Restart

Thanks for the detailed steps, much appreciated. It worked!!

I can't install tensorrt.It give me a error.

note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for tensorrt

My pip is the newest.I don't know what's happen.

Just run cmd in Administrator mode.

[Wontfallo]

If Nvidia is going to advertise this as a main feature of using their GPU, then they really need to take care of this. I'm reading 4 months worth of threads here all with the same issues everyone else seems to be hitting.
AT LEAST fix the documentation. It's lazy at best. Don't they have enough resources to hire someone to fix and maintain this?

Normally this type of troubleshooting crap wouldn't bother me, but this is from the company that I spent a lot stupid money on for their graphics card. This isn't just some Joe Blow repo, this is advertised on their snobby website, and at all over CSE 2024 That's what makes this so frustrating to me.

Fix the doc, if you think it's good enough, you're an elitist snob and should be ashamed of yourself.

┻━┻ ︵ ＼( °□° )／ ︵ ┻━┻

"How to use
Click on the “Generate Default Engines” button." oh is that? Should I just click the button? You know, the button. Ya dummies. just click it. Done.

[Yomisana]

For me msi rtx 2080 ti gaming x trio

issue

the procedure entry point 
?destroyTensorDescriptorEx@...
.....
...venv\Lib\Site-packages\nvidia\cudnn\bin\cudnn_adv_infer64_8.dll

how to fixed

close Stable Diffusion Webui
$> cd stable_diffusion_2080ti(This need change for your sd folder name)\venv\Scripts
$> cmd
$> activate.bat
$> python.exe -m pip install --upgrade pip
$> python -m pip uninstall -y nvidia-cudnn-cu11
open sd not gonna pop the error window again.🚀

[Danny-707]

This appears during installation:

*** Error running install.py for extension S:\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT.
*** Command: "S:\stable-diffusion-webui\venv\Scripts\python.exe" "S:\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\install.py"
*** Error code: 1
*** stderr: Traceback (most recent call last):
***  File "S:\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\install.py", line 3, in <module>
***   from importlib_metadata import version
*** ModuleNotFoundError: No module named 'importlib_metadata'

This happens when starting the web UI:

venv "S:\stable-diffusion-webui\venv\Scripts\Python.exe"
Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug  1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
Version: v1.8.0
Commit hash: bef51aed032c0aaa5cfd80445bc4cf0d85b408b5
*** Error running install.py for extension S:\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT.
*** Command: "S:\stable-diffusion-webui\venv\Scripts\python.exe" "S:\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\install.py"
*** Error code: 1
*** stderr: Traceback (most recent call last):
***   File "S:\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\install.py", line 3, in <module>
***     from importlib_metadata import version
*** ModuleNotFoundError: No module named 'importlib_metadata'
Launching Web UI with arguments:
no module 'xformers'. Processing without...
no module 'xformers'. Processing without...
No module 'xformers'. Proceeding without it.
*** Error loading script: lora.py
    Traceback (most recent call last):
      File "S:\stable-diffusion-webui\modules\scripts.py", line 527, in load_scripts
        script_module = script_loading.load_module(scriptfile.path)
      File "S:\stable-diffusion-webui\modules\script_loading.py", line 10, in load_module
        module_spec.loader.exec_module(module)
      File "<frozen importlib._bootstrap_external>", line 883, in exec_module
      File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
      File "S:\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\scripts\lora.py", line 7, in <module>
        import onnx
    ModuleNotFoundError: No module named 'onnx'

---
*** Error loading script: trt.py
    Traceback (most recent call last):
      File "S:\stable-diffusion-webui\modules\scripts.py", line 527, in load_scripts
        script_module = script_loading.load_module(scriptfile.path)
      File "S:\stable-diffusion-webui\modules\script_loading.py", line 10, in load_module
        module_spec.loader.exec_module(module)
      File "<frozen importlib._bootstrap_external>", line 883, in exec_module
      File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
      File "S:\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\scripts\trt.py", line 8, in <module>
        from polygraphy.logger import G_LOGGER
    ModuleNotFoundError: No module named 'polygraphy'

[wangfeng35]

polygraphy

see here: #286

[NicoBerlo]

Ok, si I simply switched to the dev branch and it worked right away !

[Dacrikka]

*** Error loading script: trt.py
Traceback (most recent call last):
File "G:\Edo\Programmi_Installati\Stable\stable-diffusion-webui\modules\scripts.py", line 382, in load_scripts
script_module = script_loading.load_module(scriptfile.path)
File "G:\Edo\Programmi_Installati\Stable\stable-diffusion-webui\modules\script_loading.py", line 10, in load_module
module_spec.loader.exec_module(module)
File "", line 883, in exec_module
File "", line 241, in _call_with_frames_removed
File "G:\Edo\Programmi_Installati\Stable\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\scripts\trt.py", line 10, in
import ui_trt
File "G:\Edo\Programmi_Installati\Stable\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\ui_trt.py", line 10, in
from exporter import export_onnx, export_trt
File "G:\Edo\Programmi_Installati\Stable\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\exporter.py", line 10, in
from utilities import Engine
File "G:\Edo\Programmi_Installati\Stable\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\utilities.py", line 22, in
import onnx_graphsurgeon as gs
ModuleNotFoundError: No module named 'onnx_graphsurgeon'

---

What is "onnx_graphsurgeon"? I installe CUDA 11.8... on a 4060 16gb

[pixelpainter]

Is there a fix for the portable version, this won't work!

[HeavyTony2]

Oh wait, I may have an explaination

  File "C:\Users\user\Documents\TestSD\stable-diffusion-webui\extensions\Hypernetwork-MonkeyPatch-Extension\patches\external_pr\sd_hijack_checkpoint.py", line 11, in ResBlock_forward
    return checkpoint(self._forward, x, emb
)

EDIT: removing Hypernetwork-MonkeyPatch-Extension solved my problem

[Adzido]

For some reason I randomly started getting this error, probably after accutlating to the latest SD version.

[ajowens001]

What appears to have worked for others.

From your base SD webui folder: (E:\Stable diffusion\SD\webui\ in your case).

* In the extensions folder delete: stable-diffusion-webui-tensorrt folder if it exists

* Delete the venv folder

Open a command prompt and navigate to the base SD webui folder

* Run webui.bat - this should rebuild the virtual environment venv

* When the WebUI appears close it and close the command prompt

Open a command prompt and navigate to the base SD webui folder

* enter:  venv\Scripts\activate.bat

* the command line should now have (venv) shown at the beginning.

* enter the following commands:

* * python.exe -m pip install --upgrade pip

* * python -m pip install nvidia-cudnn-cu11==8.9.4.25 --no-cache-dir

* * python -m pip install --pre --extra-index-url https://pypi.nvidia.com/ tensorrt==9.0.1.post11.dev4 --no-cache-dir

* * python -m pip uninstall -y nvidia-cudnn-cu11

* * venv\Scripts\deactivate.bat

* webui.bat

* Install the TensorRT extension using the Install from URL option

* Once installed, go to the Extensions >> Installed tab and Apply and Restart

Greetings,

I found this because I've tried multiple times to install this TRT extension while on a Linux OS (Ubuntu-based distro) with zero success. Each time I try to install via URL or from the extensions master list, the extension installation seems to hang, with an endless timer. I navigated to the ...webui/extensions/ folder and see that there is a 39.5 MB folder with the TRT files in there. Though, when I try to run ./webui.sh, the UI never loads - it just remains in the terminal, also looking as though it hangs.

My questions are: is this the same general issue as what is described in this issue thread? If so, can you provide the same guidance for Linux users? I don't believe that it's a problem with the venv; it really just seems to be an incomplete install. The reason I don't think this is a problem with the venv folder, is because I can delete the TRT extension folder and run the UI with no problem right after that deletion.

I am supposing that if I go into the .../venv/lib/python3.10/site-packages/ folder and ran the

    **python -m pip install nvidia-cudnn-cu11==8.9.4.25 --no-cache-dir
    **python -m pip install --pre --extra-index-url https://pypi.nvidia.com/ tensorrt==9.0.1.post11.dev4 --no-cache-dir
    **python -m pip uninstall -y nvidia-cudnn-cu11

commands, it could be fixed, but I am not sure at all. I also don't know the Linux equivalent to the Windows commands you mentioned before, i.e. enter: venv\Scripts\activate.bat & venv\Scripts\deactivate.bat.

Can you please help?

[sinand99]

If Nvidia is going to advertise this as a main feature of using their GPU, then they really need to take care of this. I'm reading 4 months worth of threads here all with the same issues everyone else seems to be hitting. AT LEAST fix the documentation. It's lazy at best. Don't they have enough resources to hire someone to fix and maintain this?

Normally this type of troubleshooting crap wouldn't bother me, but this is from the company that I spent a lot stupid money on for their graphics card. This isn't just some Joe Blow repo, this is advertised on their snobby website, and at all over CSE 2024 That's what makes this so frustrating to me.

Fix the doc, if you think it's good enough, you're an elitist snob and should be ashamed of yourself.

┻━┻ ︵ ＼( °□° )／ ︵ ┻━┻

"How to use Click on the “Generate Default Engines” button." oh is that? Should I just click the button? You know, the button. Ya dummies. just click it. Done.

+1

[NicoBerlo]

What appears to have worked for others.
(...)
* Once installed, go to the Extensions >> Installed tab and Apply and Restart

Hi !

I've managed to get it to work with v1-5-pruned-emaonly.safetensors [6ce0161689], but when I try with sd_xl_base_1.0.safetensors [31e35c80fc], I get the following error (I created a profile for the sd_xl_base_1.0 tensors which I selected in the UI, and I'm using a shape that I generated in the profile) :
*** Error completing request *** Arguments: ('task(l8v3an5pj39id83)', 'one robot on a bike', '', [], 80, 'Euler a', 1, 1, 7, 768, 1024, False, 0.7, 2, 'Latent', 0, 0, 0, 'Use same checkpoint', 'Use same sampler', '', '', [], <gradio.routes.Request object at 0x000001EBA30067D0>, 0, False, '', 0.8, 194968604, False, -1, 0, 0, 0, False, False, 'positive', 'comma', 0, False, False, '', 1, '', [], 0, '', [], 0, '', [], True, False, False, False, 0, False) {}

` (...)
File "C:\Users\nicolas\ai\sd.webui\webui\extensions-builtin\Lora\networks.py", line 486, in network_MultiheadAttention_forward
return originals.MultiheadAttention_forward(self, *args, **kwargs)

  File "C:\Users\nicolas\ai\sd.webui\webui\venv\lib\site-packages\torch\nn\modules\activation.py", line 1205, in forward
    attn_output, attn_output_weights = F.multi_head_attention_forward(

  File "C:\Users\nicolas\ai\sd.webui\webui\venv\lib\site-packages\torch\nn\functional.py", line 5373, in multi_head_attention_forward
    attn_output = scaled_dot_product_attention(q, k, v, attn_mask, dropout_p, is_causal)

NameError: name 'scaled_dot_product_attention' is not defined`

actually, when I select the sdxl tensor in the UI, I get this error in the logs :

changing setting sd_model_checkpoint to sd_xl_base_1.0.safetensors [31e35c80fc]: NameError
(...)
File "C:\Users\nicolas\ai\sd.webui\webui\venv\lib\site-packages\torch\nn\functional.py", line 5373, in multi_head_attention_forward
    attn_output = scaled_dot_product_attention(q, k, v, attn_mask, dropout_p, is_causal)
NameError: name 'scaled_dot_product_attention' is not defined

and the tensor isn't shown as selected. I have to select it twice for it to stick

I relaunched the webui.bat and now I'm getting this error with sdxl base :
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)

[BBVamp]

What appears to have worked for others.

From your base SD webui folder: (E:\Stable diffusion\SD\webui\ in your case).

• In the extensions folder delete: stable-diffusion-webui-tensorrt folder if it exists
• Delete the venv folder

Open a command prompt and navigate to the base SD webui folder

• Run webui.bat - this should rebuild the virtual environment venv
• When the WebUI appears close it and close the command prompt

Open a command prompt and navigate to the base SD webui folder

• enter: venv\Scripts\activate.bat
• the command line should now have (venv) shown at the beginning.
• enter the following commands:
• • python.exe -m pip install --upgrade pip
• • python -m pip install nvidia-cudnn-cu11==8.9.4.25 --no-cache-dir
• • python -m pip install --pre --extra-index-url https://pypi.nvidia.com/ tensorrt==9.0.1.post11.dev4 --no-cache-dir
• • python -m pip uninstall -y nvidia-cudnn-cu11
• • venv\Scripts\deactivate.bat
• webui.bat
• Install the TensorRT extension using the Install from URL option
• Once installed, go to the Extensions >> Installed tab and Apply and Restart

Worked like a charm.

[hww44]

What appears to have worked for others.
From your base SD webui folder: (E:\Stable diffusion\SD\webui\ in your case).

• In the extensions folder delete: stable-diffusion-webui-tensorrt folder if it exists
• Delete the venv folder

Open a command prompt and navigate to the base SD webui folder

• Run webui.bat - this should rebuild the virtual environment venv
• When the WebUI appears close it and close the command prompt

Open a command prompt and navigate to the base SD webui folder

• enter: venv\Scripts\activate.bat
• the command line should now have (venv) shown at the beginning.
• enter the following commands:
• • python.exe -m pip install --upgrade pip
• • python -m pip install nvidia-cudnn-cu11==8.9.4.25 --no-cache-dir
• • python -m pip install --pre --extra-index-url https://pypi.nvidia.com/ tensorrt==9.0.1.post11.dev4 --no-cache-dir
• • python -m pip uninstall -y nvidia-cudnn-cu11
• • venv\Scripts\deactivate.bat
• webui.bat
• Install the TensorRT extension using the Install from URL option
• Once installed, go to the Extensions >> Installed tab and Apply and Restart

Thanks for the detailed steps, much appreciated. It worked!!

I can't install tensorrt.It give me a error.

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for tensorrt

My pip is the newest.I don't know what's happen.

[Dacrikka]

Hi, my error is "different"

[AugmentedRealityCat]

Installing the latest nvidia drivers was the solution for me.

[Zafflower]

HI racerx2oo3,
Thanks so much for this answer. I had a different "not found" problem: Every time I launched webAI-user.bat after installing TensorRt, I got a popup error message that begin: "The procedure entry point ?destroyTensorDescriptorEx..." This message would popup just when the UI was launching and I would have to click multiple times to get rid of it. I followed your directions, and the popup went away. Thanks!
Zaffer

[Ca0-10]

[sinand99]

What appears to have worked for others.

From your base SD webui folder: (E:\Stable diffusion\SD\webui\ in your case).

* In the extensions folder delete: stable-diffusion-webui-tensorrt folder if it exists

* Delete the venv folder

Open a command prompt and navigate to the base SD webui folder

* Run webui.bat - this should rebuild the virtual environment venv

* When the WebUI appears close it and close the command prompt

Open a command prompt and navigate to the base SD webui folder

* enter:  venv\Scripts\activate.bat

* the command line should now have (venv) shown at the beginning.

* enter the following commands:

* * python.exe -m pip install --upgrade pip

* * python -m pip install nvidia-cudnn-cu11==8.9.4.25 --no-cache-dir

* * python -m pip install --pre --extra-index-url https://pypi.nvidia.com/ tensorrt==9.0.1.post11.dev4 --no-cache-dir

* * python -m pip uninstall -y nvidia-cudnn-cu11

* * venv\Scripts\deactivate.bat

* webui.bat

* Install the TensorRT extension using the Install from URL option

* Once installed, go to the Extensions >> Installed tab and Apply and Restart

there is no venv folder in portable version. I followed all steps but it does not install. Gets stuck in "onnx_graphsurgeon" step. Nvidia should fix it, this is a mess.

[racerx2oo3]

Sinan, Try this for the portable version. in file explorer open your sd.webui folder >> open the webui folder * In the extensions folder delete: stable-diffusion-webui-tensorrt folder if it exists Open a command prompt and navigate to our base SD webui folder: For the portable version this would be: sd.webui * From the command line run: environment.bat * cd webui * enter the following commands: * * python.exe -m pip install --upgrade pip * * python -m pip install nvidia-cudnn-cu11==8.9.4.25 --no-cache-dir * * python -m pip install --pre --extra-index-url https://pypi.nvidia.com/ tensorrt==9.0.1.post11.dev4 --no-cache-dir * * python -m pip uninstall -y nvidia-cudnn-cu11 * webui.bat * Install the TensorRT extension using the Install from URL option * Once installed, go to the Extensions >> Installed tab and Apply and Restart

…

On Wed, Oct 25, 2023 at 4:33 PM Sinan Dinç ***@***.***> wrote: What appears to have worked for others. From your base SD webui folder: (E:\Stable diffusion\SD\webui\ in your case). * In the extensions folder delete: stable-diffusion-webui-tensorrt folder if it exists * Delete the venv folder Open a command prompt and navigate to the base SD webui folder * Run webui.bat - this should rebuild the virtual environment venv * When the WebUI appears close it and close the command prompt Open a command prompt and navigate to the base SD webui folder * enter: venv\Scripts\activate.bat * the command line should now have (venv) shown at the beginning. * enter the following commands: * * python.exe -m pip install --upgrade pip * * python -m pip install nvidia-cudnn-cu11==8.9.4.25 --no-cache-dir * * python -m pip install --pre --extra-index-url https://pypi.nvidia.com/ tensorrt==9.0.1.post11.dev4 --no-cache-dir * * python -m pip uninstall -y nvidia-cudnn-cu11 * * venv\Scripts\deactivate.bat * webui.bat * Install the TensorRT extension using the Install from URL option * Once installed, go to the Extensions >> Installed tab and Apply and Restart there is no venv folder in portable version. I followed all steps but it does not install. Gets stuck in "onnx_graphsurgeon" step. Nvidia should fix it, this is a mess. — Reply to this email directly, view it on GitHub <#27 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANROKD3S7UNC2N5JKGHF2EDYBGOVRAVCNFSM6AAAAAA6EWWYWKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBQGE4TOMZYGQ> . You are receiving this because you commented.Message ID: ***@***.***>

[contentis]

Closing duplicates + Seems to be resolved.

[HeavyTony2]

I'm facing the same issue. I deleted the VENV dir and followed the proposed procedure, but the output remains the same

HTTP Request: POST http://127.0.0.1:7860/reset "HTTP/1.1 200 OK"
Exporting v1-5-pruned-emaonly to TensorRT
{'sample': [(1, 4, 64, 64), (2, 4, 64, 64), (8, 4, 96, 96)], 'timesteps': [(1,), (2,), (8,)], 'encoder_hidden_states': [(1, 77, 768), (2, 77, 768), (8, 154, 768)]}
No ONNX file found. Exporting ONNX...
Disabling attention optimization
INFO:root:Exporting to ONNX...
Exporting to ONNX...
============= Diagnostic Run torch.onnx.export version 2.0.1+cu118 =============
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================

ERROR:root:invalid unordered_map<K, T> key
invalid unordered_map<K, T> key
Traceback (most recent call last):
  File "C:\Users\user\Documents\TestSD\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\exporter.py", line 84, in export_onnx
    torch.onnx.export(
  File "C:\Users\user\Documents\TestSD\stable-diffusion-webui\venv\lib\site-packages\torch\onnx\utils.py", line 506, in export
    _export(
  File "C:\Users\user\Documents\TestSD\stable-diffusion-webui\venv\lib\site-packages\torch\onnx\utils.py", line 1548, in _export
    graph, params_dict, torch_out = _model_to_graph(
  File "C:\Users\user\Documents\TestSD\stable-diffusion-webui\venv\lib\site-packages\torch\onnx\utils.py", line 1113, in _model_to_graph
    graph, params, torch_out, module = _create_jit_graph(model, args)
  File "C:\Users\user\Documents\TestSD\stable-diffusion-webui\venv\lib\site-packages\torch\onnx\utils.py", line 989, in _create_jit_graph
    graph, torch_out = _trace_and_get_graph_from_model(model, args)
  File "C:\Users\user\Documents\TestSD\stable-diffusion-webui\venv\lib\site-packages\torch\onnx\utils.py", line 893, in _trace_and_get_graph_from_model
    trace_graph, torch_out, inputs_states = torch.jit._get_trace_graph(
  File "C:\Users\user\Documents\TestSD\stable-diffusion-webui\venv\lib\site-packages\torch\jit\_trace.py", line 1268, in _get_trace_graph
    outs = ONNXTracedModule(f, strict, _force_outplace, return_inputs, _return_inputs_states)(*args, **kwargs)
  File "C:\Users\user\Documents\TestSD\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\user\Documents\TestSD\stable-diffusion-webui\venv\lib\site-packages\torch\jit\_trace.py", line 127, in forward
    graph, out = torch._C._create_graph_by_tracing(
  File "C:\Users\user\Documents\TestSD\stable-diffusion-webui\venv\lib\site-packages\torch\jit\_trace.py", line 118, in wrapper
    outs.append(self.inner(*trace_inputs))
  File "C:\Users\user\Documents\TestSD\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\user\Documents\TestSD\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1488, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "C:\Users\user\Documents\TestSD\stable-diffusion-webui\modules\sd_unet.py", line 91, in UNetModel_forward
    return ldm.modules.diffusionmodules.openaimodel.copy_of_UNetModel_forward_for_webui(self, x, timesteps, context, *args, **kwargs)
  File "C:\Users\user\Documents\TestSD\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\openaimodel.py", line 797, in forward
    h = module(h, emb, context)
  File "C:\Users\user\Documents\TestSD\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\user\Documents\TestSD\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1488, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "C:\Users\user\Documents\TestSD\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\openaimodel.py", line 82, in forward
    x = layer(x, emb)
  File "C:\Users\user\Documents\TestSD\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\user\Documents\TestSD\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1488, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "C:\Users\user\Documents\TestSD\stable-diffusion-webui\extensions\Hypernetwork-MonkeyPatch-Extension\patches\external_pr\sd_hijack_checkpoint.py", line 11, in ResBlock_forward
    return checkpoint(self._forward, x, emb)
  File "C:\Users\user\Documents\TestSD\stable-diffusion-webui\venv\lib\site-packages\torch\utils\checkpoint.py", line 249, in checkpoint
    return CheckpointFunction.apply(function, preserve, *args)
  File "C:\Users\user\Documents\TestSD\stable-diffusion-webui\venv\lib\site-packages\torch\autograd\function.py", line 506, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
RuntimeError: invalid unordered_map<K, T> key

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\user\Documents\TestSD\stable-diffusion-webui\venv\lib\site-packages\gradio\routes.py", line 488, in run_predict
    output = await app.get_blocks().process_api(
  File "C:\Users\user\Documents\TestSD\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1431, in process_api
    result = await self.call_function(
  File "C:\Users\user\Documents\TestSD\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1103, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "C:\Users\user\Documents\TestSD\stable-diffusion-webui\venv\lib\site-packages\anyio\to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "C:\Users\user\Documents\TestSD\stable-diffusion-webui\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "C:\Users\user\Documents\TestSD\stable-diffusion-webui\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "C:\Users\user\Documents\TestSD\stable-diffusion-webui\venv\lib\site-packages\gradio\utils.py", line 707, in wrapper
    response = f(*args, **kwargs)
  File "C:\Users\user\Documents\TestSD\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\ui_trt.py", line 135, in export_unet_to_trt
    export_onnx(
  File "C:\Users\user\Documents\TestSD\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\exporter.py", line 129, in export_onnx
    exit()
  File "C:\ProgramData\anaconda3\lib\_sitebuiltins.py", line 26, in __call__
    raise SystemExit(code)
SystemExit: None
INFO:httpx:HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 500 Internal Server Error"
HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 500 Internal Server Error"
INFO:httpx:HTTP Request: POST http://127.0.0.1:7860/reset "HTTP/1.1 200 OK"
HTTP Request: POST http://127.0.0.1:7860/reset "HTTP/1.1 200 OK"

[liuyu-f]

Looks like some people need to add the line "pip install importlib_metadata" (#157 (comment))
These people may just install "importlib_metadata" and they may be able to use TensorRT normally.

The portable version can also be used (I am), the method is: #27 (comment)

(From Google Translate)

[mojiezuo-888]

This is my problem

[banshek7]

What appears to have worked for others.

From your base SD webui folder: (E:\Stable diffusion\SD\webui\ in your case).

• In the extensions folder delete: stable-diffusion-webui-tensorrt folder if it exists
• Delete the venv folder

Open a command prompt and navigate to the base SD webui folder

• Run webui.bat - this should rebuild the virtual environment venv
• When the WebUI appears close it and close the command prompt

Open a command prompt and navigate to the base SD webui folder

• enter: venv\Scripts\activate.bat
• the command line should now have (venv) shown at the beginning.
• enter the following commands:
• • python.exe -m pip install --upgrade pip
• • python -m pip install nvidia-cudnn-cu11==8.9.4.25 --no-cache-dir
• • python -m pip install --pre --extra-index-url https://pypi.nvidia.com/ tensorrt==9.0.1.post11.dev4 --no-cache-dir
• • python -m pip uninstall -y nvidia-cudnn-cu11
• • venv\Scripts\deactivate.bat
• webui.bat
• Install the TensorRT extension using the Install from URL option
• Once installed, go to the Extensions >> Installed tab and Apply and Restart

I successfully fixed my tensorRT shit without deleting vent and my existing vent, I just entered like you said, ran the commands in order, launched webui cringing super hard waiting for the windows error prompt sound to trigger me, but nope, webui loaded, tensorRT tab visible, hypothetically ready? FYI I had no clue this WASNT using tensorRT, im so glad I decided to give the extension tab a look over like wtf. They should really warn you or auto detect if you have an RTX and prompt you that its possible.

[banshek7]

called it vent like 9 times, I meant venv, lol, Anyways thanks a trillion!!

[banshek7]

Responded to the wrong github issue so i had to delete my comment lol

[FurkanGozukara]

For those who can't apply this fix I have fully automated installer that includes fix shared here : https://www.patreon.com/posts/86307255

And here a full no paywall tutorial

Double Your Stable Diffusion Inference Speed with RTX Acceleration TensorRT: A Comprehensive Guide

[CaptainDoubleYou]

I followed all the steps and I still get this error message:
AssertionError: Extension directory already exists

The extension folder is empty. Not sure how it stays the directory exists. I even started from scratch and this still happened.

[andrewtvuong]

What appears to have worked for others.
From your base SD webui folder: (E:\Stable diffusion\SD\webui\ in your case).

* In the extensions folder delete: stable-diffusion-webui-tensorrt folder if it exists

* Delete the venv folder

Open a command prompt and navigate to the base SD webui folder

* Run webui.bat - this should rebuild the virtual environment venv

* When the WebUI appears close it and close the command prompt

Open a command prompt and navigate to the base SD webui folder

* enter:  venv\Scripts\activate.bat

* the command line should now have (venv) shown at the beginning.

* enter the following commands:

* * python.exe -m pip install --upgrade pip

* * python -m pip install nvidia-cudnn-cu11==8.9.4.25 --no-cache-dir

* * python -m pip install --pre --extra-index-url https://pypi.nvidia.com/ tensorrt==9.0.1.post11.dev4 --no-cache-dir

* * python -m pip uninstall -y nvidia-cudnn-cu11

* * venv\Scripts\deactivate.bat

* webui.bat

* Install the TensorRT extension using the Install from URL option

* Once installed, go to the Extensions >> Installed tab and Apply and Restart

Greetings,

I found this because I've tried multiple times to install this TRT extension while on a Linux OS (Ubuntu-based distro) with zero success. Each time I try to install via URL or from the extensions master list, the extension installation seems to hang, with an endless timer. I navigated to the ...webui/extensions/ folder and see that there is a 39.5 MB folder with the TRT files in there. Though, when I try to run ./webui.sh, the UI never loads - it just remains in the terminal, also looking as though it hangs.

My questions are: is this the same general issue as what is described in this issue thread? If so, can you provide the same guidance for Linux users? I don't believe that it's a problem with the venv; it really just seems to be an incomplete install. The reason I don't think this is a problem with the venv folder, is because I can delete the TRT extension folder and run the UI with no problem right after that deletion.

I am supposing that if I go into the .../venv/lib/python3.10/site-packages/ folder and ran the

    **python -m pip install nvidia-cudnn-cu11==8.9.4.25 --no-cache-dir
    **python -m pip install --pre --extra-index-url https://pypi.nvidia.com/ tensorrt==9.0.1.post11.dev4 --no-cache-dir
    **python -m pip uninstall -y nvidia-cudnn-cu11

commands, it could be fixed, but I am not sure at all. I also don't know the Linux equivalent to the Windows commands you mentioned before, i.e. enter: venv\Scripts\activate.bat & venv\Scripts\deactivate.bat.

Can you please help?

I'm also on Ubuntu, found the issue and created a fix #225

Basically if you're stuck:

1. Delete stable-diffusion-webui/venv dir
2. Delete existing (if existing) extensions/Stable-Diffusion-WebUI-TensorRT
3. Rerun webui.sh to rebuild venv
4. source venv/bin/activate in the stable-diffusion-webui dir MAKE ABSOLUTELY SURE THIS IS ENABLED

---

pip uninstall -y tensorrt
pip install -y nvidia-cudnn-cu11==8.9.4.25 --no-cache-dir
pip install --pre --extra-index-url https://pypi.nvidia.com tensorrt==9.0.1.post11.dev4 --no-cache-dir
pip uninstall -y nvidia-cudnn-cu11

6. deactivate
7. Proceed with normal installation of Nvidia TensorRT

--- alternative, not recommend unless you know what you're doing

Then you can either apply my fix 7c93523 to https://github.com/NVIDIA/Stable-Diffusion-WebUI-TensorRT/blob/main/install.py and run python install.py

or using the A1111 SD UI install my repo instead https://github.com/andrewtvuong/Stable-Diffusion-WebUI-TensorRT as a temporary work around. Once the main package is fixed, I would uninstall and install the main again.

[FurkanGozukara]

well for those who can't still make I made a full tutorial and also auto installers

Double Your Stable Diffusion Inference Speed with RTX Acceleration TensorRT: A Comprehensive Guide

[fenyadim]

What appears to have worked for others.
From your base SD webui folder: (E:\Stable diffusion\SD\webui\ in your case).

* In the extensions folder delete: stable-diffusion-webui-tensorrt folder if it exists

* Delete the venv folder

Open a command prompt and navigate to the base SD webui folder

* Run webui.bat - this should rebuild the virtual environment venv

* When the WebUI appears close it and close the command prompt

Open a command prompt and navigate to the base SD webui folder

* enter:  venv\Scripts\activate.bat

* the command line should now have (venv) shown at the beginning.

* enter the following commands:

* * python.exe -m pip install --upgrade pip

* * python -m pip install nvidia-cudnn-cu11==8.9.4.25 --no-cache-dir

* * python -m pip install --pre --extra-index-url https://pypi.nvidia.com/ tensorrt==9.0.1.post11.dev4 --no-cache-dir

* * python -m pip uninstall -y nvidia-cudnn-cu11

* * venv\Scripts\deactivate.bat

* webui.bat

* Install the TensorRT extension using the Install from URL option

* Once installed, go to the Extensions >> Installed tab and Apply and Restart

Greetings,
I found this because I've tried multiple times to install this TRT extension while on a Linux OS (Ubuntu-based distro) with zero success. Each time I try to install via URL or from the extensions master list, the extension installation seems to hang, with an endless timer. I navigated to the ...webui/extensions/ folder and see that there is a 39.5 MB folder with the TRT files in there. Though, when I try to run ./webui.sh, the UI never loads - it just remains in the terminal, also looking as though it hangs.
My questions are: is this the same general issue as what is described in this issue thread? If so, can you provide the same guidance for Linux users? I don't believe that it's a problem with the venv; it really just seems to be an incomplete install. The reason I don't think this is a problem with the venv folder, is because I can delete the TRT extension folder and run the UI with no problem right after that deletion.
I am supposing that if I go into the .../venv/lib/python3.10/site-packages/ folder and ran the

    **python -m pip install nvidia-cudnn-cu11==8.9.4.25 --no-cache-dir
    **python -m pip install --pre --extra-index-url https://pypi.nvidia.com/ tensorrt==9.0.1.post11.dev4 --no-cache-dir
    **python -m pip uninstall -y nvidia-cudnn-cu11

commands, it could be fixed, but I am not sure at all. I also don't know the Linux equivalent to the Windows commands you mentioned before, i.e. enter: venv\Scripts\activate.bat & venv\Scripts\deactivate.bat.
Can you please help?

I'm also on Ubuntu, found the issue and created a fix #225

Basically if you're stuck:

1. Delete stable-diffusion-webui/venv dir
2. Delete existing (if existing) extensions/Stable-Diffusion-WebUI-TensorRT
3. Rerun webui.sh to rebuild venv
4. source venv/bin/activate in the stable-diffusion-webui dir MAKE ABSOLUTELY SURE THIS IS ENABLED

pip uninstall -y tensorrt
pip install -y nvidia-cudnn-cu11==8.9.4.25 --no-cache-dir
pip install --pre --extra-index-url https://pypi.nvidia.com tensorrt==9.0.1.post11.dev4 --no-cache-dir
pip uninstall -y nvidia-cudnn-cu11

6. deactivate
7. Proceed with normal installation of Nvidia TensorRT

--- alternative, not recommend unless you know what you're doing

Then you can either apply my fix 7c93523 to https://github.com/NVIDIA/Stable-Diffusion-WebUI-TensorRT/blob/main/install.py and run python install.py

or using the A1111 SD UI install my repo instead https://github.com/andrewtvuong/Stable-Diffusion-WebUI-TensorRT as a temporary work around. Once the main package is fixed, I would uninstall and install the main again.

Thank you!

[ErixStrong]

What appears to have worked for others.

From your base SD webui folder: (E:\Stable diffusion\SD\webui\ in your case).

• In the extensions folder delete: stable-diffusion-webui-tensorrt folder if it exists
• Delete the venv folder

Open a command prompt and navigate to the base SD webui folder

• Run webui.bat - this should rebuild the virtual environment venv
• When the WebUI appears close it and close the command prompt

Open a command prompt and navigate to the base SD webui folder

• enter: venv\Scripts\activate.bat
• the command line should now have (venv) shown at the beginning.
• enter the following commands:
• • python.exe -m pip install --upgrade pip
• • python -m pip install nvidia-cudnn-cu11==8.9.4.25 --no-cache-dir
• • python -m pip install --pre --extra-index-url https://pypi.nvidia.com/ tensorrt==9.0.1.post11.dev4 --no-cache-dir
• • python -m pip uninstall -y nvidia-cudnn-cu11
• • venv\Scripts\deactivate.bat
• webui.bat
• Install the TensorRT extension using the Install from URL option
• Once installed, go to the Extensions >> Installed tab and Apply and Restart

I tried to run this procedure several times, but it doesn't work fro me, I'm still getting this error:

*** Error running install.py for extension F:\Stable-Diffusion\webui\extensions\Stable-Diffusion-WebUI-TensorRT.
*** Command: "F:\Stable-Diffusion\webui\venv\Scripts\python.exe" "F:\Stable-Diffusion\webui\extensions\Stable-Diffusion-WebUI-TensorRT\install.py"
*** Error code: 1
*** stderr: Traceback (most recent call last):
*** File "F:\Stable-Diffusion\webui\extensions\Stable-Diffusion-WebUI-TensorRT\install.py", line 3, in
*** from importlib_metadata import version
*** ModuleNotFoundError: No module named 'importlib_metadata'

[CGGermany]

I followed the instructions and it worked. 100%. In my case, the fifth point of the instructions (venv\Scripts\deactivate.bat) did not work and/or was not found. I then skipped it (omitted it). Thanks for the precise instructions. Now I have the TensorRT tab in the UI.

I previously set the arguments to default (deleted and kept backup). I have no idea if it has any meaning or use, but I also backed up the venv folder before deleting it.

[reziozhou]

What appears to have worked for others.

From your base SD webui folder: (E:\Stable diffusion\SD\webui\ in your case).

• In the extensions folder delete: stable-diffusion-webui-tensorrt folder if it exists
• Delete the venv folder

Open a command prompt and navigate to the base SD webui folder

• Run webui.bat - this should rebuild the virtual environment venv
• When the WebUI appears close it and close the command prompt

Open a command prompt and navigate to the base SD webui folder

• enter: venv\Scripts\activate.bat
• the command line should now have (venv) shown at the beginning.
• enter the following commands:
• • python.exe -m pip install --upgrade pip
• • python -m pip install nvidia-cudnn-cu11==8.9.4.25 --no-cache-dir
• • python -m pip install --pre --extra-index-url https://pypi.nvidia.com/ tensorrt==9.0.1.post11.dev4 --no-cache-dir
• • python -m pip uninstall -y nvidia-cudnn-cu11
• • venv\Scripts\deactivate.bat
• webui.bat
• Install the TensorRT extension using the Install from URL option
• Once installed, go to the Extensions >> Installed tab and Apply and Restart

I use the [Stable-Diffusion-WebUI-TensorRT] branch master but get errors caused by dll error,but changed to controlnet_v2 branch get fix it

[Danny-707]

I tried to run this procedure several times, but it doesn't work fro me, I'm still getting this error:

*** Error running install.py for extension F:\Stable-Diffusion\webui\extensions\Stable-Diffusion-WebUI-TensorRT. *** Command: "F:\Stable-Diffusion\webui\venv\Scripts\python.exe" "F:\Stable-Diffusion\webui\extensions\Stable-Diffusion-WebUI-TensorRT\install.py" *** Error code: 1 *** stderr: Traceback (most recent call last): *** File "F:\Stable-Diffusion\webui\extensions\Stable-Diffusion-WebUI-TensorRT\install.py", line 3, in *** from importlib_metadata import version *** ModuleNotFoundError: No module named 'importlib_metadata'

I have the same problem(
I've gone through the procedure several times, but with no success. Even on a freshly installed Automatic1111...

[riperbot]

My rude fix: #143 (comment)

[peteer01]

What appears to have worked for others.

From your base SD webui folder: (E:\Stable diffusion\SD\webui\ in your case).

* In the extensions folder delete: stable-diffusion-webui-tensorrt folder if it exists

* Delete the venv folder

Open a command prompt and navigate to the base SD webui folder

* Run webui.bat - this should rebuild the virtual environment venv

* When the WebUI appears close it and close the command prompt

Open a command prompt and navigate to the base SD webui folder

* enter:  venv\Scripts\activate.bat

* the command line should now have (venv) shown at the beginning.

* enter the following commands:

* * python.exe -m pip install --upgrade pip

* * python -m pip install nvidia-cudnn-cu11==8.9.4.25 --no-cache-dir

* * python -m pip install --pre --extra-index-url https://pypi.nvidia.com/ tensorrt==9.0.1.post11.dev4 --no-cache-dir

* * python -m pip uninstall -y nvidia-cudnn-cu11

* * venv\Scripts\deactivate.bat

* webui.bat

* Install the TensorRT extension using the Install from URL option

* Once installed, go to the Extensions >> Installed tab and Apply and Restart

I stumbled upon this in 2024 and used the following commands in case the newer packages help today. Here are the commands I used:

python.exe -m pip install --upgrade pip
python -m pip install nvidia-cudnn-cu12==9.0.0.312 --no-cache-dir
python -m pip install --pre --extra-index-url https://pypi.nvidia.com/ tensorrt==9.3.0.post12.dev1 --no-cache-dir
python -m pip uninstall -y nvidia-cudnn-cu12
venv\Scripts\deactivate.bat

If you use version 10 of nvidia-cudnn-cu12, you will get an error when trying to use TensorRT later, so this uses the latest version of 9.

You can also use the commands in the 2023 post above with the older packages. The older models worked, but I was getting the below errors after using the older packages, although TensorRT seemed to work:

[E] 1: [myelinCache.cpp::nvinfer1::builder::MyelinAutotunerCache::deserialize::62] Error Code 1: Myelin (Myelin error from unknown graph)
[E] 4: The timing cache will not be used!

I do receive "[E] 4: The timing cache will not be used!" regardless of what packages I use.