ivangabriele/docker-llm

Pre-loaded LLMs served as an OpenAI-Compatible API via Docker images.

Docker Image ―
OpenAI API-Compatible Pre-loaded LLM Server

Docker images are based on Nvidia CUDA images. LLMs are pre-loaded and served via vLLM.

Environment Variables

• TENSOR_PARALLEL_SIZE: Number of GPUs to use. Default: 1.

Port

The OpenAI API is exposed on port 8000.

Tags & Deployment Links

Note

The VRAM column is the minimum required amount of VRAM used by the model on a single GPU.

Tag	Model	RunPod	Vast.ai	VRAM
ivangabriele/llm:lmsys__vicuna-13b-v1.5-16k				26GB
ivangabriele/llm:open-orca__llongorca-13b-16k				26GB

Roadmap

• Add more popular models.
• Start the server in background to allow for SSH access.

---

docker-llm's Issues

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Awaiting Schedule

These updates are awaiting their schedule. Click on a checkbox to get an update now.

• fix(deps): update all non-major dependencies (accelerate, nvidia-cublas-cu11, nvidia-cuda-cupti-cu11, nvidia-cuda-nvrtc-cu11, nvidia-cuda-runtime-cu11, nvidia-cudnn-cu11, nvidia-curand-cu11, nvidia-cusolver-cu11, nvidia-cusparse-cu11, nvidia-nccl-cu11, nvidia-nvtx-cu11, torch, vllm)
• chore(deps): update nvidia/cuda docker tag to v12

Detected dependencies

docker-compose

docker-compose.yml

dockerfile

Dockerfile

• nvidia/cuda 11.8.0-cudnn8-devel-ubuntu22.04

poetry

pyproject.toml

• nvidia-cublas-cu11 11.10.3.66
• nvidia-cuda-cupti-cu11 11.7.101
• nvidia-cuda-nvrtc-cu11 11.7.99
• nvidia-cuda-runtime-cu11 11.7.99
• nvidia-cudnn-cu11 8.5.0.96
• nvidia-cufft-cu11 10.9.0.58
• nvidia-curand-cu11 10.2.10.91
• nvidia-cusolver-cu11 11.4.0.1
• nvidia-cusparse-cu11 11.7.4.91
• nvidia-nccl-cu11 2.14.3
• nvidia-nvtx-cu11 11.7.91
• python ^3.11.0
• python-dotenv ^1.0.0
• torch 2.0.1
• vllm 0.2.0
• accelerate 0.24.0
• fschat 0.2.31
• openai 0.28.1

---

• Check this box to trigger a request for Renovate to run again on this repository