How to Deploy AI Models on Cloud GPU: A Practical Guide

You Trained a Model. Now What?

Training a model is half the battle. Deploying it so real users can send requests and get predictions back -- reliably, fast, and without burning through your cloud budget -- is the half most tutorials skip. This guide walks through the practical steps of deploying AI models on cloud GPUs in 2026, from container setup to production monitoring.

Whether you're serving a fine-tuned Llama 3, a Stable Diffusion pipeline, or a custom Whisper transcription model, the deployment patterns are the same. We'll cover them all.

Step 1: Containerize Your Model

Before you touch a GPU instance, your model needs to live in a container. Docker is the standard. The container packages your model weights, inference code, and dependencies into a single portable artifact that runs identically on any GPU cloud instance.

Here's a minimal Dockerfile for a HuggingFace transformer model served with FastAPI:

FROM nvidia/cuda:12.4-runtime-ubuntu22.04

RUN apt-get update && apt-get install -y python3.11 python3-pip
WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY model/ /app/model/
COPY serve.py .

EXPOSE 8000
CMD ["python3", "serve.py"]

Your serve.py loads the model once at startup and keeps it warm in GPU memory:

from fastapi import FastAPI
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

app = FastAPI()
model = AutoModelForCausalLM.from_pretrained(
    "/app/model", torch_dtype=torch.float16, device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("/app/model")

@app.post("/generate")
async def generate(prompt: str, max_tokens: int = 256):
    inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
    outputs = model.generate(**inputs, max_new_tokens=max_tokens)
    return {"text": tokenizer.decode(outputs[0], skip_special_tokens=True)}

Step 2: Choose the Right GPU Instance

Not every model needs an H100. Match your GPU to your model size and latency requirements:

Rule of thumb: Your model weights in GPU memory (FP16) are roughly 2 bytes per parameter. Add 20-30% overhead for KV cache and activations. If your model + overhead fits in a single GPU's VRAM, deploy on one GPU. If not, you need tensor parallelism across multiple GPUs or quantization (INT8/INT4).

Step 3: Deploy to the Cloud GPU Instance

Once your container is built and pushed to a registry (Docker Hub, GitHub Container Registry, or your cloud provider's registry), deploying is a few commands. Here's the workflow for a typical GPU cloud instance running Ubuntu 22.04:

# 1. SSH into your GPU instance
ssh ubuntu@your-gpu-instance-ip

# 2. Install NVIDIA Container Toolkit (one-time setup)
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo systemctl restart docker

# 3. Pull and run your model container
docker run -d \
  --name model-server \
  --gpus all \
  -p 8000:8000 \
  -v /data/models:/app/model \
  your-registry/model-server:latest

# 4. Verify it's running
curl -X POST http://localhost:8000/generate \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Hello, world!", "max_tokens": 10}'

That's it. In under 60 seconds, your model is serving predictions on a cloud GPU. Compare this to the traditional approach: order hardware, wait days for provisioning, rack it, install drivers, configure networking, and then finally deploy. GPU cloud collapses that timeline from days to seconds.

Step 4: Production Hardening

Serving a model on a single GPU instance is step one. Production means handling concurrent requests, managing GPU memory efficiently, and staying up when things break. Here's what to add:

Request Batching

GPUs are throughput-optimized, not latency-optimized. Sending one token at a time wastes GPU cycles. Use a batching engine like vLLM or TensorRT-LLM to combine multiple requests into a single forward pass:

# vLLM replaces your custom FastAPI server
pip install vllm
python -m vllm.entrypoints.openai.api_server \
  --model /app/model \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.90

vLLM handles continuous batching, PagedAttention for KV cache management, and an OpenAI-compatible API endpoint. For most LLM deployments in 2026, vLLM is the default choice.

Load Balancing and Auto-Scaling

One GPU instance handles a few hundred concurrent requests. When traffic spikes, you need more instances. Use a reverse proxy (NGINX, HAProxy) or a cloud-native load balancer to distribute traffic across multiple GPU instances. Auto-scale based on request queue depth or GPU utilization.

Health Checks and Graceful Shutdown

Your model server should expose a /health endpoint that returns 200 when the model is loaded and ready. The load balancer uses this to route traffic. On shutdown, the server should finish in-flight requests before terminating (SIGTERM handling).

GPU Memory Monitoring

Track GPU memory usage in production. If your model server leaks memory (common with dynamic batching), you'll see VRAM climb until OOM. Set up Prometheus metrics with nvidia-smi exporters and alert when usage exceeds 90%.

Step 5: Optimize for Cost

GPU cloud is cheap per hour, but costs add up when instances run 24/7. Here's how to optimize:

• Use spot/preemptible instances for batch inference and non-critical workloads. Spot GPUs are 50-70% cheaper than on-demand but can be reclaimed at any time. Build your deployment to handle interruptions gracefully.
• Right-size your GPU. If your 7B model fits on an RTX 3090 ($0.15/hr), don't pay for an A100 ($1.80/hr). Benchmark your actual latency and throughput needs before choosing a GPU tier.
• Model quantization. INT8 quantization reduces model size by 50% with minimal accuracy loss. INT4 reduces it by 75%. A quantized 7B model can run on a cheaper GPU or serve more concurrent requests on the same hardware.
• Auto-shutdown idle instances. If no requests arrive for 15 minutes, shut down the instance. GPU cloud instances start in under 60 seconds, so cold starts are acceptable for most workloads.

Real-World Deployment: A Production LLM API

Let's walk through a complete deployment of a fine-tuned Llama 3 8B model on BHK Cloud:

1. Build the container with vLLM and your fine-tuned weights.
2. Push to a registry (Docker Hub or GitHub Container Registry).
3. Spin up an RTX 3090 instance on BHK Cloud ($0.15/hr).
4. SSH in, pull the container, run it with --gpus all.
5. Expose port 8000 and point your application at the instance IP.
6. Add a second instance behind a load balancer for redundancy.
7. Set up auto-shutdown after 30 minutes of inactivity for the dev environment.

Total monthly cost for a dev + staging deployment (2 x RTX 3090, 100 hours/month each, 5 TB storage): under $50/month. The same deployment on AWS SageMaker would cost $300-500/month.

Monitoring Your Deployment

Once your model is live, you need visibility into performance. At minimum, track these metrics:

For LLM-specific monitoring, also track tokens per second (generation speed) and time-to-first-token (TTFT), which is the latency users actually feel.

Frequently Asked Questions