Multi-GPU Cloud Setup: Scaling from 1 to 8 GPUs

Moving from a single GPU to a multi-GPU cloud setup unlocks larger models, faster training, and production-scale inference. But distributed GPU computing introduces networking, orchestration, and cost complexity. This guide covers the practical path from one GPU to eight — configuration, tooling, common pitfalls, and cost management.

When to Scale Beyond a Single GPU

A single GPU hits its ceiling in three scenarios:

1. VRAM exhaustion. Your model, optimizer states, and batch size no longer fit in one GPU's memory. A 70B-parameter model in FP16 needs roughly 140 GB just for weights — well beyond any single GPU.
2. Training time. Even if the model fits, training on 1 TB of data with one GPU might take weeks. Adding GPUs with data parallelism reduces wall-clock time proportionally.
3. Inference throughput. Production serving at hundreds of requests per second exceeds a single GPU's capacity. Multiple GPUs running parallel model replicas handle the load.

GPU Topology Options

Single-Node Multi-GPU (2–8 GPUs)

All GPUs live in one physical server, connected via NVLink or PCIe. This is the simplest multi-GPU setup and the most common in cloud environments.

NVLink-connected GPUs (e.g., 4×A100 or 8×H100 in a single node) provide 600–900 GB/s GPU-to-GPU bandwidth. Tensor parallelism and pipeline parallelism work efficiently because communication costs are low.

PCIe-only GPUs (e.g., multiple RTX 3090s or 4090s in a single server) are limited to ~32 GB/s per GPU via PCIe 4.0 x16. Data parallelism is still effective here, but tensor parallelism suffers. For cloud GPU rental, RTX 3090 instances at $0.15/hr are cost-effective for data-parallel training of smaller models.

Multi-Node Multi-GPU (16+ GPUs)

GPUs spread across multiple physical servers, connected by InfiniBand or high-speed Ethernet. Required for models too large for any single node's aggregate memory.

Most teams never need multi-node training. A single 8×A100 node with 640 GB aggregate VRAM handles fine-tuning of models up to 70B parameters with data parallelism alone. Multi-node training adds complexity: NCCL configuration, cross-node communication debugging, and higher susceptibility to straggler effects.

Software Stack for Multi-GPU Training

PyTorch Distributed

PyTorch's torch.distributed package is the standard for multi-GPU training. The most common strategies:

Distributed Data Parallel (DDP). Each GPU holds a full model replica and processes a different data batch. Gradients are synchronized with all-reduce after each backward pass. DDP scales near-linearly up to 8 GPUs on a single node. This is the default choice for most workloads.

Fully Sharded Data Parallel (FSDP). Shards model parameters, gradients, and optimizer states across GPUs instead of replicating them. Reduces per-GPU memory usage, enabling larger models or bigger batch sizes. FSDP is effective for models that barely exceed single-GPU memory.

Tensor Parallelism + Pipeline Parallelism. For the largest models (70B+), combine tensor parallelism (splitting individual layers across GPUs) with pipeline parallelism (splitting layer groups across GPUs). Frameworks like Megatron-LM and DeepSpeed provide optimized implementations.

Hugging Face Accelerate

For teams that don't want to write distributed training boilerplate, Hugging Face Accelerate abstracts away torch.distributed:

from accelerate import Accelerator

accelerator = Accelerator()
model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)

for batch in dataloader:
    outputs = model(batch)
    loss = compute_loss(outputs)
    accelerator.backward(loss)
    optimizer.step()

Launch with accelerate launch --num_processes=8 train.py for 8 GPUs. Accelerate handles DDP, FSDP, and DeepSpeed integration automatically.

DeepSpeed

Microsoft's DeepSpeed library provides ZeRO (Zero Redundancy Optimizer) stages for memory-efficient distributed training:

• ZeRO Stage 1: Shards optimizer states across GPUs. ~4× memory reduction for AdamW.
• ZeRO Stage 2: Also shards gradients. ~8× reduction.
• ZeRO Stage 3: Shards parameters, gradients, and optimizer states. Memory scales linearly with GPU count — a 70B model that needs 560 GB can train on 8×80GB GPUs.

DeepSpeed integrates with PyTorch and Hugging Face Trainer with minimal code changes.

Configuration Checklist for 8-GPU Setup

NCCL Environment Variables

NCCL (NVIDIA Collective Communications Library) handles GPU-to-GPU communication. Set these for optimal performance:

export NCCL_DEBUG=INFO                    # Verbose logging for debugging
export NCCL_IB_DISABLE=0                 # Enable InfiniBand if available
export NCCL_SOCKET_IFNAME=eth0           # Specify network interface
export NCCL_NSOCKS_PERTHREAD=4          # Socket count per thread
export NCCL_SOCKET_NTHREADS=2           # CPU threads for socket management

Containerized Setup

Use the official NVIDIA PyTorch container for consistent CUDA and NCCL versions:

docker run --gpus all -it --rm \
  -v /data:/data \
  nvcr.io/nvidia/pytorch:24.06-py3 \
  torchrun --nproc_per_node=8 train.py

The --gpus all flag exposes all GPUs. Use --gpus '"device=0,1,2,3"' to restrict to specific GPUs.

Validation Commands

After launching a multi-GPU job, verify all GPUs are active:

# Check GPU utilization
nvidia-smi dmon -s pucv -d 2

# Verify NCCL communication
python -c "import torch; torch.distributed.init_process_group('nccl'); print('NCCL ready')"

# Check memory usage per GPU
nvidia-smi --query-gpu=index,name,memory.used,memory.total --format=csv

Common Pitfalls

Data Loading Bottleneck

With 8 GPUs processing batches in parallel, data loading must keep pace. Use num_workers in PyTorch DataLoader (typically 4–8 per GPU), store datasets on local NVMe rather than network storage, and pre-process data before training to avoid on-the-fly transformations.

Gradient Synchronization Imbalance

If one GPU consistently finishes its batch faster than others, all GPUs wait. Balance batch sizes, ensure uniform GPU models (don't mix A100 and RTX 3090 in the same DDP group), and pin processes to specific GPUs with CUDA_VISIBLE_DEVICES.

Shared Storage Contention

When 8 GPUs write checkpoints simultaneously to a shared NFS mount, throughput collapses. Use local NVMe for checkpoint writes and sync to object storage asynchronously, or stagger checkpoint writes across GPUs.

Mixed Precision Mismatch

FP16 and BF16 training behave differently on different GPU architectures. A100 and H100 support BF16 natively; RTX 3090 does not. If mixing GPU types (not recommended), verify precision settings produce identical results.

Cost Analysis: 1 vs 4 vs 8 GPUs

Key insight: scaling GPU count reduces wall-clock time but total cost per job stays relatively flat with good data-parallel scaling (3.7× speedup for 4× GPUs, not 4×, due to communication overhead). The real value is time-to-result, not cost savings.

Frequently Asked Questions