Containerized AI Workloads on Cloud GPUs: Docker, Kubernetes, and Slurm

AI workloads on cloud GPUs typically start with a single developer SSH'ing into a GPU node and running python train.py. That works for experimentation. It does not work when you need reproducible environments, multi-node training, job queuing, or production model serving. This guide covers the three dominant orchestration patterns — Docker, Kubernetes, and Slurm — and when to use each on cloud GPU infrastructure.

Why Containerize GPU Workloads

GPU cloud nodes are ephemeral. You provision a node, you run work, you release it. Without containerization, every new node requires re-installing CUDA, PyTorch, and dependencies — a 15–30 minute tax on every provisioning cycle.

Containers solve this. A Docker image with your exact CUDA version, framework, and dependencies means a new GPU node is productive within seconds of boot. Beyond reproducibility, containers provide:

• Dependency isolation. Training script requires PyTorch 2.4, inference server needs PyTorch 2.5? Containers eliminate version conflicts.
• Consistent environments across GPU generations. An RTX 3090 node and an A100 node run the same container — only GPU count and memory differ.
• CI/CD integration. Build, test, and deploy GPU workloads through the same pipeline as your application code.

Docker on Cloud GPUs: The Foundation

The NVIDIA Container Toolkit (nvidia-container-toolkit) enables Docker containers to access the host GPU. Installation on a fresh GPU node:

# Ubuntu/Debian
distribution=$(. /etc/os-release; echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

After installation, run with --gpus all:

docker run --gpus all --rm -v /data:/data pytorch/pytorch:2.4.0-cuda12.4-cudnn9-runtime \
    python -c "import torch; print(torch.cuda.device_count())"

Multi-GPU Docker

Docker supports GPU selection by index or UUID:

# Use GPU 0 and 1 only
docker run --gpus '"device=0,1"' --rm pytorch/pytorch:latest nvidia-smi

# Use specific GPU by UUID
docker run --gpus '"device=GPU-8e0a2bc7-...-abc123"' --rm pytorch/pytorch:latest nvidia-smi

This is critical for multi-tenant GPU nodes where different containers need different GPUs.

Docker Limitations for GPU Workloads

Docker alone is sufficient for single-node, single-user GPU workloads. It falls short when: - Multiple team members need to share GPU nodes (no scheduling) - Workloads span multiple GPU nodes (no orchestration) - Jobs need queuing with priorities and resource limits (no scheduler)

For these scenarios, Kubernetes or Slurm is required.

Kubernetes on Cloud GPUs

Kubernetes with GPU support provides automated scheduling, resource management, and self-healing for containerized AI workloads. The architecture:

┌──────────────────────────────────────────┐
│              Kubernetes Cluster            │
│                                            │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐ │
│  │ GPU Node │  │ GPU Node │  │ GPU Node │ │
│  │ RTX 3090 │  │ A100 x4  │  │ RTX 3090 │ │
│  │ 24GB     │  │ 320GB    │  │ 24GB     │ │
│  └──────────┘  └──────────┘  └──────────┘ │
│                                            │
│  Scheduler → Pod → GPU Resource Claim      │
└──────────────────────────────────────────┘

GPU Node Setup for Kubernetes

Each GPU node needs the NVIDIA device plugin:

kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.15.0/deployments/static/nvidia-device-plugin.yml

GPU resources are then requestable in pod specs:

apiVersion: v1
kind: Pod
metadata:
  name: training-job
spec:
  containers:
  - name: trainer
    image: pytorch/pytorch:2.4.0-cuda12.4-cudnn9-devel
    resources:
      limits:
        nvidia.com/gpu: 2
    command: ["python", "train.py"]
    volumeMounts:
    - name: data
      mountPath: /data
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: training-data-pvc
  restartPolicy: Never

Time-Slicing and MIG for GPU Sharing

Kubernetes can share GPUs across pods using:

Time-slicing (all GPUs). The device plugin can be configured to expose a GPU as multiple fractional resources (e.g., one RTX 3090 as 4 "gpu-slices" of 6GB each). Time-slicing uses CUDA time-slicing — it does not enforce memory isolation. A pod requesting 6GB can still allocate more.

MIG (A100, H100 only). Multi-Instance GPU partitions one physical GPU into multiple isolated instances with hardware-enforced memory and compute isolation. Each MIG instance appears as a separate CUDA device.

# Request a specific MIG profile
resources:
  limits:
    nvidia.com/mig-1g.10gb: 1

When Kubernetes Makes Sense for GPU Cloud

• Multi-user teams. 5+ data scientists or ML engineers sharing a GPU cluster
• Production model serving. Inference APIs with autoscaling, rolling updates, and health checks
• Hybrid workloads. Mix of training jobs and inference serving on the same GPU pool
• Existing Kubernetes investment. Your organization already runs application workloads on Kubernetes

When Kubernetes Is Overkill

• Single user or single project. Docker or a simple job scheduler is sufficient
• Fewer than 3 GPU nodes. The Kubernetes control plane overhead isn't justified
• Batch training only (no serving). Slurm is purpose-built for HPC-style job scheduling

Slurm on Cloud GPUs: HPC-Style Job Scheduling

Slurm is the dominant workload manager in academic and national-lab HPC environments. It has gained traction in cloud GPU deployments for teams that need fine-grained job scheduling, priority queuing, and resource accounting.

Slurm Architecture on Cloud GPU Nodes

┌─────────────────────────────────────────┐
│            Slurm Controller              │
│  (slurmctld) - Job queue, scheduling    │
└──────────────┬──────────────────────────┘
               │
    ┌──────────┼──────────┐
    ▼          ▼          ▼
┌────────┐ ┌────────┐ ┌────────┐
│GPU Node│ │GPU Node│ │GPU Node│
│slurmd  │ │slurmd  │ │slurmd  │
│4x A100 │ │2x 3090 │ │8x A100 │
└────────┘ └────────┘ └────────┘

Submitting GPU Jobs with Slurm

#!/bin/bash
#SBATCH --job-name=llama-finetune
#SBATCH --nodes=2
#SBATCH --gpus-per-node=4
#SBATCH --time=24:00:00
#SBATCH --partition=gpu
#SBATCH --output=/logs/%j.out

srun --container-image=nvcr.io#nvidia/pytorch:24.04-py3 \
    torchrun --nproc_per_node=4 --nnodes=$SLURM_NNODES \
    train.py --batch-size 128

Slurm's advantages for GPU cloud:

• Job arrays. Submit 100 hyperparameter tuning runs with one command
• Fair-share scheduling. Ensure GPU time is distributed across teams or projects
• Preemption. Low-priority jobs can be preempted for high-priority ones — analogous to spot instances
• Resource tracking. GPU-hours per user, per project, per billing period

Slurm vs Kubernetes: Decision Framework

For teams primarily doing batch training with some interactive development, Slurm on cloud GPUs is often the faster path to productivity than Kubernetes. For teams serving models in production, Kubernetes wins.

Hybrid Patterns: Slurm for Training, Kubernetes for Serving

Many AI teams adopt a hybrid architecture:

┌──────────────────────┐     ┌──────────────────────┐
│   Training Cluster    │     │   Serving Cluster     │
│   (Slurm-managed)     │────▶│   (Kubernetes-managed)│
│                       │     │                       │
│   4×A100 nodes        │     │   2×A10 nodes         │
│   Batch jobs          │     │   Inference APIs      │
│   Experiment tracking │     │   Autoscaling         │
└──────────────────────┘     └──────────────────────┘
         │                            │
         └────────────┬───────────────┘
                      │
              ┌───────▼────────┐
              │  S3-Compatible  │
              │  Object Storage │
              │  (models, data) │
              └────────────────┘

Training jobs consume GPU capacity in bursts — 24–72 hour runs, then idle. Serving workloads need continuous GPU availability but at lower utilization. Running both on the same orchestration platform forces compromises — Kubernetes is weak at batch scheduling, Slurm is weak at service management. A hybrid approach uses each platform for what it does best.

Containerized AI on BHK Cloud

BHK Cloud GPU nodes come with the NVIDIA Container Toolkit pre-installed. Root access allows you to install and configure Docker, Kubernetes (kubeadm/k3s), or Slurm as needed. No managed Kubernetes — you control the full stack. S3-compatible object storage for container images, model weights, datasets, and checkpoints. Private VLANs for multi-node cluster networking. Standard Ubuntu 22.04 images with CUDA 12.4 and NVIDIA driver 550.