Docker & Containerization for Local AI: GPUs, vLLM High-Throughput & Ollama

Q: How do I keep downloaded models between container restarts?

Mount a persistent volume for the Hugging Face cache: ~/.cache/huggingface:/root/.cache/huggingface. The models then live on the host and survive container recreation, so a redeploy does not re-download them.

Quick answer: running local AI with Docker Compose. Install the NVIDIA Container Toolkit, confirm the GPU is visible with docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi, then define a Compose service with a GPU reservation (driver: nvidia, count: all), ipc: host for vLLM, a persistent volume for the model cache (~/.cache/huggingface:/root/.cache/huggingface), and port 8000 for the OpenAI-compatible API. See the Docker production deployments guide for multi-stage builds and the DevOps track for the rest of the series.

By Sarah Lin • Advanced • 21 min read • Updated 2026-11-01

What You Will Master in This Tutorial

Configure NVIDIA Container Toolkit to pass physical GPU acceleration into Docker containers.
Deploy vLLM in Docker for high-throughput serving with PagedAttention and continuous batching.
Orchestrate a complete local AI developer stack using Docker Compose.
Optimize shared memory (shm-size) and persistent model cache storage across deployments.

1. Configuring GPU Passthrough with NVIDIA Container Toolkit

Running local LLMs inside Docker requires exposing the host GPU and CUDA drivers to the container runtime. The modern standard is the NVIDIA Container Toolkit (nvidia-ctk).

BASH

# Configure NVIDIA repository and install toolkit
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# Test GPU access inside container
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

Note: Crucial Configuration: Always verify that nvidia-smi runs successfully inside the test container before deploying AI stacks.

2. Production Docker Compose: vLLM + WebUI + OpenClaw

Below is a battle-tested Docker Compose manifest orchestrating high-throughput vLLM serving, persistent Hugging Face cache volumes, and an OpenAI-compatible API endpoint.

YAML

services:
  vllm-inference:
    image: vllm/vllm-openai:latest
    container_name: vllm-deepseek
    runtime: nvidia
    restart: unless-stopped
    environment:
      - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN}
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
    ports:
      - "8000:8000"
    ipc: host
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    command: >
      --model deepseek-ai/DeepSeek-R1-Distill-Qwen-14B
      --tensor-parallel-size 1
      --max-model-len 8192
      --gpu-memory-utilization 0.90
      --trust-remote-code

Note: Performance Note: ipc: host is mandatory for vLLM and PyTorch to prevent Shared Memory (shm) allocation panics during high-concurrency inference.

Knowledge Check: Test Your Understanding

1. Why is 'ipc: host' required when running vLLM inside Docker?

Frequently asked questions about Docker for local AI

How do I check that Docker can see my GPU before deploying a local AI stack?

Install the NVIDIA Container Toolkit, configure the Docker runtime with sudo nvidia-ctk runtime configure --runtime=docker, restart Docker, then run docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi. Always verify that nvidia-smi runs successfully inside the test container before deploying AI stacks.

Why does the vLLM service in Docker Compose need ipc: host?

Because vLLM and PyTorch need host shared memory for inter-process tensor communication. Without it, high-concurrency inference can hit shared-memory allocation panics that look like out-of-memory crashes.

How do I keep downloaded models between container restarts?

Mount a persistent volume for the Hugging Face cache, as in this guide: ~/.cache/huggingface:/root/.cache/huggingface. The models then live on the host and survive container recreation, so a redeploy does not re-download them. The DeepSeek R1 and Ollama local LLM guide covers running models locally in more depth.