LLM Inference in the Cloud: Throughput, Latency, and Cost Optimization

Deploying a large language model for inference — serving completions, chat, or embeddings to end users — has a different performance profile than training. Training is throughput-bound: more FLOPs per second means faster completion. Inference is latency-bound: users expect responses in under 500 milliseconds. This guide covers the key optimization dimensions for LLM inference on cloud GPUs: batch sizing, quantization, KV-cache management, and cost-aware autoscaling.

The Inference Performance Triangle

Three metrics define inference quality, and they trade off against each other:

• Latency. Time to first token (TTFT) plus time per output token (TPOT). Users perceive anything above 500 ms total as slow.
• Throughput. Tokens per second across all concurrent requests. Determines how many users a single GPU can serve.
• Cost. Dollars per million tokens served. Determines unit economics.

Improving one metric often degrades another. Increasing batch size raises throughput but adds latency. Reducing precision (FP16 → INT8) improves throughput and cost but may degrade output quality. The optimization target depends on the application.

GPU Selection for Inference

Memory Capacity Is the Binding Constraint

Unlike training, where compute (TFLOPS) dominates, inference is memory-bound. The model weights must fit in GPU VRAM. Memory requirements at FP16 precision:

Add 2–4 GB overhead for the KV-cache (key-value cache for attention) per concurrent request. A 7B model on a 24 GB RTX 3090 leaves roughly 8 GB for KV-cache — enough to handle 8–16 concurrent users depending on sequence length.

GPU Architecture and Inference Performance

For models under 13B parameters, an RTX 3090 or 4090 provides the best cost-per-token ratio. The A100 and H100 are only necessary for larger models or high-concurrency serving.

Quantization: Less Precision, More Throughput

Quantization reduces model precision from FP16 to INT8 or INT4, shrinking memory footprint and increasing throughput at a small quality cost.

Quantization Levels Compared

INT8 quantization is the sweet spot for most production deployments. It halves VRAM requirements, nearly doubles throughput, and introduces no measurable quality degradation for well-calibrated models. Tools: llama.cpp (GGUF format), vLLM (AWQ quantization), TensorRT-LLM (FP8/INT8/INT4).

Quantization Example: 7B Model on RTX 3090

INT8 doubles throughput and nearly triples concurrent capacity on the same GPU. For a service charging $0.01 per 1K tokens, INT8 halves the GPU cost per token from $0.00024 to $0.00012.

Batching Strategies

Static Batching

Accumulate requests for a fixed time window (e.g., 100 ms) and process them as a single batch. Simple to implement but adds latency proportional to the window. A 100 ms batching window means worst-case latency increases by 100 ms.

Continuous (Dynamic) Batching

Process tokens as they arrive rather than waiting for full requests to complete. A request that finishes early exits the batch immediately; a new request can join mid-generation. This is the standard approach in vLLM, TensorRT-LLM, and most production inference servers.

Continuous batching improves throughput 2–3× over static batching for workloads with variable-length outputs (chat, summarization) and 5–10× for mixed-length workloads.

Speculative Decoding

Use a small "draft" model (1–3B parameters) to predict likely next tokens, verify them against the full model, and accept or reject in parallel. Produces 2–3× faster generation with identical output quality. Cost: runs two models simultaneously, increasing VRAM usage by about 10–20%.

KV-Cache Management

The key-value cache stores the attention computation for each previous token in a sequence. Without caching, each new token would recompute attention for the entire history — an O(n²) operation. The KV-cache makes it O(n).

KV-Cache Memory

KV-cache memory per token depends on model architecture. For a typical 7B model:

KV-cache per token = 2 (K + V) × layers × hidden_dim × precision_bytes
                   = 2 × 32 × 4096 × 2 (FP16)
                   ≈ 0.5 MB per token

A conversation with 4,096 tokens of context requires about 2 GB of KV-cache. Ten concurrent conversations at 4K context each require 20 GB — consuming most of a 24 GB GPU.

KV-Cache Optimizations

PagedAttention (vLLM). Allocates KV-cache in non-contiguous blocks ("pages"), similar to virtual memory. Reduces fragmentation and allows memory sharing across requests with identical prefixes (system prompts). Typical memory savings: 30–50%.

Multi-Query Attention (MQA) and Grouped-Query Attention (GQA). Architectural choices that reduce KV-cache size. GQA (used in Llama 3) uses fewer key-value heads than query heads, reducing cache size 4–8× compared to full multi-head attention.

Prefix caching. When multiple requests share a long system prompt, the KV-cache for the shared prefix is computed once and reused. Reduces prefill latency by 50–80% for system-prompt-heavy applications.

Cost Optimization: Dollars per Million Tokens

Calculating Unit Cost

Cost per 1M tokens = (GPU hourly rate × seconds per request) / (tokens per request) × 1,000,000

Example: RTX 3090 at $0.15/hr serving a 7B INT8 model generating 68 tokens/sec:

Cost per 1M tokens = ($0.15 / 3600) / 68 × 1,000,000
                   = $0.613 per 1M output tokens

With continuous batching handling 8 concurrent users (effective throughput ~500 tokens/sec):

Cost per 1M tokens = ($0.15 / 3600) / 500 × 1,000,000
                   = $0.083 per 1M output tokens

Cost Comparison: Self-Hosted vs API

Self-hosting a 7B-class model on a $0.15/hr GPU is 3–10× cheaper than API services for output tokens. The trade-off: you manage the infrastructure, handle scaling, and are limited to models that fit on your GPU.

When Self-Hosting Wins

• High volume. More than 50M tokens/month — the GPU cost amortizes.
• Data privacy. Model inputs never leave your infrastructure.
• Fine-tuned models. API services charge a premium for fine-tuned model hosting or do not support it.
• Latency control. No API queue; you control the entire serving stack.

When API Services Win

• Low or spiky volume. Fewer than 10M tokens/month. The GPU idle time between requests erases the cost advantage.
• Large models. Hosting Llama 3 405B requires 8+ H100 GPUs ($20,000+/month). API access costs $3–$5 per 1M tokens.
• Zero DevOps. No infrastructure to manage, monitor, or scale.

Autoscaling Inference

Load-Based Scaling

Monitor concurrent request count. Spin up additional GPU instances when the queue depth exceeds a threshold. Spin down when idle. Key metrics:

• Scale-up trigger. Average queue wait time > 200 ms for 30+ seconds.
• Scale-down trigger. GPU utilization < 20% for 5+ minutes.
• Cold start penalty. New GPU provisioning takes 1–3 minutes. Maintain a warm spare during business hours.

Time-Based Scaling

If traffic follows predictable daily patterns, pre-schedule GPU instances. Example: 2 GPUs during business hours (8 AM–8 PM), 1 GPU overnight, 0 GPUs on weekends. Reduces idle GPU cost by 60% compared to 24/7 full capacity.

Multi-Model Serving

Serve multiple fine-tuned LoRA adapters from a single base model on one GPU. Each request specifies which adapter to use. vLLM and TensorRT-LLM support this natively. A single RTX 3090 can serve 5–10 LoRA-tuned 7B models simultaneously, amortizing the GPU cost across multiple applications.

Frequently Asked Questions