Stuck at "removing nvidia-cudnn-cu11" forever

[timetheonce]

Stuck at "removing nvidia-cudnn-cu11" forever

[timetheonce]

Output from container is buffered, so you cannot see what's currently . It is probably doing something in the background (like downloading a very large model file for some extension). It is building properly but stuck during running in container. You can do following things:

1. Observe your CPU usage and Internet transfer for container with docker stats command. It will show whether something is being downloaded in the background. If waiting a very long time will not change anything then
2. Start with extensions disabled. You can move all directories from ./data/config/auto/extensions/ to some other location and then run. By trial and error (adding half of extensions and rechecking) you can find out which extension is source of problem then.
3. If both previous steps are not giving anything useful then you can try removing ./data/.cache directory contents to make sure all things are being downloaded fresh and up to date instead of something weird / old cached.

Thanks for your comments. Problem solved by your 2nd suggestion, the problematic extension is called "Stable-Diffusion-WebUI-TensorRT". Up to now still no idea what does this extension do, but removing it seems everything back to normal.

I think it is worthy to add this question and the 3 solutions to FAQ.

Appreciate much for your help.

[DevilaN]

Could you provide full build log instead of screenshot of problematic part?

[timetheonce]

Could you provide full build log instead of screenshot of problematic part?

Sure, please see the logs below:

---

root@Tower:/mnt/user/Software/Stable_Diffusion/stable-diffusion-webui-docker-master# docker compose --profile auto up --build
[+] Building 2.4s (25/25) FINISHED docker:default
=> [auto internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [auto internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 2.99kB 0.0s
=> [auto internal] load metadata for docker.io/pytorch/pytorch:2.1.2-cuda12.1-cudnn8-runtime 2.2s
=> [auto internal] load metadata for docker.io/alpine/git:2.36.2 2.3s
=> [auto internal] load build context 0.0s
=> => transferring context: 298B 0.0s
=> [auto download 1/8] FROM docker.io/alpine/git:2.36.2@sha256:ec491c893597b68c92b88023827faa771772cfd5e106b76c713fa5e1c75dea84 0.0s
=> [auto stage-1 1/12] FROM docker.io/pytorch/pytorch:2.1.2-cuda12.1-cudnn8-runtime@sha256:3387e598cb94fc248d82e712a65b10931a990cea3a2e76 0.0s
=> CACHED [auto stage-1 2/12] RUN --mount=type=cache,target=/var/cache/apt apt-get update && apt-get install -y fonts-dejavu-core rsy 0.0s
=> CACHED [auto stage-1 3/12] RUN --mount=type=cache,target=/root/.cache/pip git clone https://github.com/AUTOMATIC1111/stable-diffusio 0.0s
=> CACHED [auto download 2/8] COPY clone.sh /clone.sh 0.0s
=> CACHED [auto download 3/8] RUN . /clone.sh stable-diffusion-stability-ai https://github.com/Stability-AI/stablediffusion.git cf1d67a6fd 0.0s
=> CACHED [auto download 4/8] RUN . /clone.sh CodeFormer https://github.com/sczhou/CodeFormer.git c5b4593074ba6214284d6acd5f1719b6c5d739af 0.0s
=> CACHED [auto download 5/8] RUN . /clone.sh BLIP https://github.com/salesforce/BLIP.git 48211a1594f1321b00f14c9f7a5b4813144b2fb9 0.0s
=> CACHED [auto download 6/8] RUN . /clone.sh k-diffusion https://github.com/crowsonkb/k-diffusion.git ab527a9a6d347f364e3d185ba6d714e22d8 0.0s
=> CACHED [auto download 7/8] RUN . /clone.sh clip-interrogator https://github.com/pharmapsychotic/clip-interrogator 2cf03aaf6e704197fd0da 0.0s
=> CACHED [auto download 8/8] RUN . /clone.sh generative-models https://github.com/Stability-AI/generative-models 45c443b316737a4ab6e40413 0.0s
=> CACHED [auto stage-1 4/12] COPY --from=download /repositories/ /stable-diffusion-webui/repositories/ 0.0s
=> CACHED [auto stage-1 5/12] RUN mkdir /stable-diffusion-webui/interrogate && cp /stable-diffusion-webui/repositories/clip-interrogator/ 0.0s
=> CACHED [auto stage-1 6/12] RUN --mount=type=cache,target=/root/.cache/pip pip install -r /stable-diffusion-webui/repositories/CodeFo 0.0s
=> CACHED [auto stage-1 7/12] RUN --mount=type=cache,target=/root/.cache/pip pip install pyngrok xformers==0.0.23.post1 git+https://g 0.0s
=> CACHED [auto stage-1 8/12] RUN apt-get -y install libgoogle-perftools-dev && apt-get clean 0.0s
=> CACHED [auto stage-1 9/12] COPY . /docker 0.0s
=> CACHED [auto stage-1 10/12] RUN sed -i 's/in_app_dir = .*/in_app_dir = True/g' /opt/conda/lib/python3.10/site-packages/gradio/routes. 0.0s
=> CACHED [auto stage-1 11/12] WORKDIR /stable-diffusion-webui 0.0s
=> [auto] exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:3488b565a3d89c38d8bd3f96da8a134e4b82ddda74e33c883de24eaa40f36d5a 0.0s
=> => naming to docker.io/library/sd-auto:72 0.0s
[+] Running 1/0
✔ Container webui-docker-auto-1 Created 0.0s
Attaching to webui-docker-auto-1
webui-docker-auto-1 | /stable-diffusion-webui
webui-docker-auto-1 | total 660K
webui-docker-auto-1 | drwxr-xr-x 1 root root 1.1K Apr 24 12:25 .
webui-docker-auto-1 | drwxr-xr-x 1 root root 268 Apr 24 12:25 ..
webui-docker-auto-1 | -rw-r--r-- 1 root root 48 Apr 22 09:02 .eslintignore
webui-docker-auto-1 | -rw-r--r-- 1 root root 3.4K Apr 22 09:02 .eslintrc.js
webui-docker-auto-1 | drwxr-xr-x 1 root root 156 Apr 22 09:02 .git
webui-docker-auto-1 | -rw-r--r-- 1 root root 55 Apr 22 09:02 .git-blame-ignore-revs
webui-docker-auto-1 | drwxr-xr-x 1 root root 94 Apr 22 09:02 .github
webui-docker-auto-1 | -rw-r--r-- 1 root root 495 Apr 22 09:02 .gitignore
webui-docker-auto-1 | -rw-r--r-- 1 root root 119 Apr 22 09:02 .pylintrc
webui-docker-auto-1 | -rw-r--r-- 1 root root 51K Apr 22 09:02 CHANGELOG.md
webui-docker-auto-1 | -rw-r--r-- 1 root root 243 Apr 22 09:02 CITATION.cff
webui-docker-auto-1 | -rw-r--r-- 1 root root 657 Apr 22 09:02 CODEOWNERS
webui-docker-auto-1 | -rw-r--r-- 1 root root 35K Apr 22 09:02 LICENSE.txt
webui-docker-auto-1 | -rw-r--r-- 1 root root 12K Apr 22 09:02 README.md
webui-docker-auto-1 | drwxr-xr-x 1 root root 44 Apr 24 12:25 pycache
webui-docker-auto-1 | lrwxrwxrwx 1 root root 29 Apr 24 12:25 config.json -> /data/config/auto/config.json
webui-docker-auto-1 | lrwxrwxrwx 1 root root 31 Apr 24 12:25 config_states -> /data/config/auto/config_states
webui-docker-auto-1 | drwxr-xr-x 1 root root 252 Apr 22 09:02 configs
webui-docker-auto-1 | lrwxrwxrwx 1 root root 16 Apr 24 12:25 embeddings -> /data/embeddings
webui-docker-auto-1 | -rw-r--r-- 1 root root 167 Apr 22 09:02 environment-wsl2.yaml
webui-docker-auto-1 | lrwxrwxrwx 1 root root 28 Apr 24 12:25 extensions -> /data/config/auto/extensions
webui-docker-auto-1 | drwxr-xr-x 1 root root 194 Apr 22 09:02 extensions-builtin
webui-docker-auto-1 | drwxr-xr-x 1 root root 190 Apr 22 09:02 html
webui-docker-auto-1 | drwxr-xr-x 1 root root 116 Apr 22 09:05 interrogate
webui-docker-auto-1 | drwxr-xr-x 1 root root 764 Apr 22 09:02 javascript
webui-docker-auto-1 | -rw-r--r-- 1 root root 1.3K Apr 22 09:02 launch.py
webui-docker-auto-1 | drwxr-xr-x 1 root root 62 Apr 22 09:02 localizations
webui-docker-auto-1 | lrwxrwxrwx 1 root root 12 Apr 24 12:25 models -> /data/models
webui-docker-auto-1 | drwxr-xr-x 1 root root 4.0K Apr 24 12:25 modules
webui-docker-auto-1 | -rw-r--r-- 1 root root 185 Apr 22 09:02 package.json
webui-docker-auto-1 | -rw-r--r-- 1 root root 813 Apr 22 09:02 pyproject.toml
webui-docker-auto-1 | drwxr-xr-x 1 root root 176 Apr 22 08:23 repositories
webui-docker-auto-1 | -rw-r--r-- 1 root root 49 Apr 22 09:02 requirements-test.txt
webui-docker-auto-1 | -rw-r--r-- 1 root root 360 Apr 22 09:02 requirements.txt
webui-docker-auto-1 | -rw-r--r-- 1 root root 556 Apr 22 09:02 requirements_versions.txt
webui-docker-auto-1 | -rw-r--r-- 1 root root 411K Apr 22 09:02 screenshot.png
webui-docker-auto-1 | -rw-r--r-- 1 root root 5.3K Apr 22 09:02 script.js
webui-docker-auto-1 | drwxr-xr-x 1 root root 742 Apr 22 09:02 scripts
webui-docker-auto-1 | -rw-r--r-- 1 root root 23K Apr 22 09:02 style.css
webui-docker-auto-1 | lrwxrwxrwx 1 root root 28 Apr 24 12:25 styles.csv -> /data/config/auto/styles.csv
webui-docker-auto-1 | drwxr-xr-x 1 root root 178 Apr 22 09:02 test
webui-docker-auto-1 | drwxr-xr-x 1 root root 168 Apr 22 09:02 textual_inversion_templates
webui-docker-auto-1 | lrwxrwxrwx 1 root root 32 Apr 24 12:25 ui-config.json -> /data/config/auto/ui-config.json
webui-docker-auto-1 | -rw-r--r-- 1 root root 670 Apr 22 09:02 webui-macos-env.sh
webui-docker-auto-1 | -rw-r--r-- 1 root root 84 Apr 22 09:02 webui-user.bat
webui-docker-auto-1 | -rw-r--r-- 1 root root 1.4K Apr 22 09:02 webui-user.sh
webui-docker-auto-1 | -rw-r--r-- 1 root root 2.3K Apr 22 09:02 webui.bat
webui-docker-auto-1 | -rw-r--r-- 1 root root 5.3K Apr 22 09:02 webui.py
webui-docker-auto-1 | -rwxr-xr-x 1 root root 7.9K Apr 22 09:02 webui.sh
webui-docker-auto-1 | Mounted embeddings
webui-docker-auto-1 | Mounted .cache
webui-docker-auto-1 | Mounted styles.csv
webui-docker-auto-1 | Mounted ui-config.json
webui-docker-auto-1 | Mounted models
webui-docker-auto-1 | Mounted .cache
webui-docker-auto-1 | Mounted config_states
webui-docker-auto-1 | Mounted config.json
webui-docker-auto-1 | Mounted extensions
webui-docker-auto-1 | Installing extension dependencies (if any)
webui-docker-auto-1 | TensorRT is not installed! Installing...
webui-docker-auto-1 | Installing nvidia-cudnn-cu11
webui-docker-auto-1 | Installing tensorrt
webui-docker-auto-1 | Looking in indexes: https://pypi.org/simple, https://pypi.nvidia.com
webui-docker-auto-1 | Collecting tensorrt==9.0.1.post11.dev4
webui-docker-auto-1 | Downloading https://pypi.nvidia.com/tensorrt/tensorrt-9.0.1.post11.dev4.tar.gz (18 kB)
webui-docker-auto-1 | Preparing metadata (setup.py) ... done
webui-docker-auto-1 | Collecting tensorrt_libs==9.0.1.post11.dev4 (from tensorrt==9.0.1.post11.dev4)
webui-docker-auto-1 | Downloading https://pypi.nvidia.com/tensorrt-libs/tensorrt_libs-9.0.1.post11.dev4-py2.py3-none-manylinux_2_17_x86_64.whl (1060.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 GB 18.8 MB/s eta 0:00:00
webui-docker-auto-1 | Collecting tensorrt_bindings==9.0.1.post11.dev4 (from tensorrt==9.0.1.post11.dev4)
webui-docker-auto-1 | Downloading https://pypi.nvidia.com/tensorrt-bindings/tensorrt_bindings-9.0.1.post11.dev4-cp310-none-manylinux_2_17_x86_64.whl (956 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 956.2/956.2 kB 20.9 MB/s eta 0:00:00
webui-docker-auto-1 | Collecting nvidia-cuda-runtime-cu11 (from tensorrt_libs==9.0.1.post11.dev4->tensorrt==9.0.1.post11.dev4)
webui-docker-auto-1 | Downloading nvidia_cuda_runtime_cu11-11.8.89-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
webui-docker-auto-1 | Requirement already satisfied: nvidia-cudnn-cu11 in /opt/conda/lib/python3.10/site-packages (from tensorrt_libs==9.0.1.post11.dev4->tensorrt==9.0.1.post11.dev4) (8.9.4.25)
webui-docker-auto-1 | Requirement already satisfied: nvidia-cublas-cu11 in /opt/conda/lib/python3.10/site-packages (from tensorrt_libs==9.0.1.post11.dev4->tensorrt==9.0.1.post11.dev4) (11.11.3.6)
webui-docker-auto-1 | Requirement already satisfied: nvidia-cuda-nvrtc-cu11 in /opt/conda/lib/python3.10/site-packages (from nvidia-cudnn-cu11->tensorrt_libs==9.0.1.post11.dev4->tensorrt==9.0.1.post11.dev4) (11.8.89)
webui-docker-auto-1 | Downloading nvidia_cuda_runtime_cu11-11.8.89-py3-none-manylinux1_x86_64.whl (875 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 875.6/875.6 kB 2.1 MB/s eta 0:00:00
webui-docker-auto-1 | Building wheels for collected packages: tensorrt
webui-docker-auto-1 | Building wheel for tensorrt (setup.py) ... done
webui-docker-auto-1 | Created wheel for tensorrt: filename=tensorrt-9.0.1.post11.dev4-py2.py3-none-any.whl size=17623 sha256=a8d32a118da6568a41c9522971cf04d8a093ced46e5bfdbea726deaa28936158
webui-docker-auto-1 | Stored in directory: /tmp/pip-ephem-wheel-cache-_ix5l582/wheels/d1/6d/71/f679d0d23a60523f9a05445e269bfd0bcd1c5272097fa931df
webui-docker-auto-1 | Successfully built tensorrt
webui-docker-auto-1 | Installing collected packages: tensorrt_bindings, nvidia-cuda-runtime-cu11, tensorrt_libs, tensorrt
webui-docker-auto-1 | Successfully installed nvidia-cuda-runtime-cu11-11.8.89 tensorrt-9.0.1.post11.dev4 tensorrt_bindings-9.0.1.post11.dev4 tensorrt_libs-9.0.1.post11.dev4
webui-docker-auto-1 | WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
webui-docker-auto-1 | removing nvidia-cudnn-cu11

[DevilaN]

Output from container is buffered, so you cannot see what's currently . It is probably doing something in the background (like downloading a very large model file for some extension). It is building properly but stuck during running in container.
You can do following things:

1. Observe your CPU usage and Internet transfer for container with docker stats command. It will show whether something is being downloaded in the background. If waiting a very long time will not change anything then
2. Start with extensions disabled. You can move all directories from ./data/config/auto/extensions/ to some other location and then run. By trial and error (adding half of extensions and rechecking) you can find out which extension is source of problem then.
3. If both previous steps are not giving anything useful then you can try removing ./data/.cache directory contents to make sure all things are being downloaded fresh and up to date instead of something weird / old cached.