GPU Cloud Storage Architecture: NVMe, Object Storage, and Network File Systems

Every GPU workload depends on storage. Training reads datasets; inference loads model weights; checkpointing writes snapshots. A slow storage layer turns a fast GPU into an expensive heater — the 3090 waits idle while data trickles in at HDD speeds. This guide maps the three storage tiers in GPU cloud, explains when to use each, and provides reference architectures for common AI workloads.

The Three Storage Tiers

Tier 1: Local NVMe — Performance at the GPU

NVMe SSDs attached directly to the GPU instance via PCIe. Typical performance: 3–7 GB/s sequential read, 500K–1M random read IOPS, sub-100-microsecond latency.

This is where active work happens. Training datasets, model weights during fine-tuning, temporary checkpoints, and Docker image layers all live here during GPU-active periods.

When to use: any data the GPU actively accesses. Training data must be on NVMe — reading 100 GB from object storage at 1 Gbps takes 13 minutes per epoch, while local NVMe delivers it in under 30 seconds.

Pricing trap: most providers charge for NVMe storage whether the GPU is running or stopped. A 500 GB NVMe volume at $0.10/GB/month costs $50/month even if the GPU ran for only 20 hours. Always delete or snapshot NVMe volumes when stopping instances for more than a few hours.

Tier 2: Network-Attached Storage — Shared Access

Network file systems (NFS, Lustre, WEKA) or block storage volumes accessible from multiple GPU instances simultaneously. Performance: 1–10 GB/s depending on the filesystem and network interconnect.

This tier solves the multi-GPU problem: when four GPUs across two nodes all need the same dataset, copying it to each node's local NVMe wastes time and storage. A shared network filesystem makes the dataset available to all GPUs with a single copy.

When to use: multi-node training where the dataset exceeds single-node NVMe capacity, or when multiple instances need concurrent read access to the same data.

Performance considerations: network-attached storage adds 100–500 microseconds of latency per I/O operation — negligible for sequential reads during training, but problematic for random-access patterns. File locking and metadata operations (listing directories with millions of files) can become bottlenecks at scale.

Tier 3: Object Storage — Scale and Durability

S3-compatible object storage. Performance: 100–500 MB/s per object, eventual consistency, 10–50 ms latency per request. Unlimited capacity, 99.999999999% durability (11 nines) on major providers.

This is the cold storage layer and the source of truth. Datasets, model checkpoints, training logs, and deployment artifacts live here permanently.

When to use: everything that is not actively being read by a GPU. Datasets staged to NVMe before training. Checkpoints pushed after each epoch. Model weights archived for later deployment.

Pricing advantage: object storage costs $2–$6/TB/month. The same 500 GB that costs $50/month on persistent NVMe costs $1–$3/month on object storage — a 20–50× difference.

Reference Architecture: Single-GPU Training

┌─────────────────────────────────┐
│         GPU Instance            │
│  ┌─────────────────────────┐    │
│  │   Local NVMe (500 GB)   │    │
│  │  • Active dataset       │    │
│  │  • Model weights        │    │
│  │  • Checkpoints (temp)   │    │
│  └─────────────────────────┘    │
└─────────────────────────────────┘
           │
           │ Stage / Push
           ▼
┌─────────────────────────────────┐
│   Object Storage (Unlimited)    │
│  • Datasets (source of truth)   │
│  • Checkpoints (archived)       │
│  • Model weights (versioned)    │
│  • Training logs                │
└─────────────────────────────────┘

Workflow:

1. Before training. Pull dataset from object storage to local NVMe. 500 GB over 10 Gbps ≈ 7 minutes.
2. During training. GPU reads from local NVMe at 3–7 GB/s. Checkpoints saved to NVMe.
3. After training. Push latest checkpoint and logs to object storage. Delete NVMe volume (or stop instance).
4. Resume. New instance pulls latest checkpoint from object storage to NVMe. Training continues from checkpoint.

Reference Architecture: Multi-Node Training (4 GPUs)

┌──────────────────┐  ┌──────────────────┐
│  Node 1 (2 GPU)  │  │  Node 2 (2 GPU)  │
│  NVMe: temp data │  │  NVMe: temp data │
└────────┬─────────┘  └────────┬─────────┘
         │                     │
         └──────────┬──────────┘
                    │
         ┌──────────▼──────────┐
         │  Network Filesystem │
         │  (NFS / Lustre)     │
         │  • Shared dataset   │
         └──────────┬──────────┘
                    │
         ┌──────────▼──────────┐
         │   Object Storage    │
         │  • Cold datasets    │
         │  • Checkpoints      │
         └─────────────────────┘

Workflow:

1. Dataset staged from object storage to network filesystem.
2. All nodes mount the same network filesystem path.
3. Each node reads data from the shared filesystem; intra-epoch shuffling is coordinated.
4. Checkpoints written to network filesystem, then pushed to object storage for durability.

Network filesystem performance is the bottleneck in this architecture. At 1 GB/s shared across 4 GPUs, each GPU gets 250 MB/s — sufficient for image datasets but potentially limiting for video or large text corpora.

Storage Costs for Common Workloads

The single largest cost optimization is not keeping data on expensive storage tiers when GPUs are idle. A team running 20 hours of GPU training per week with 500 GB of data saves $43/month by moving from persistent NVMe to a staged-from-S3 architecture.

Optimizing Storage for AI Workloads

Dataset Formats

Sequential reads (training). Use formats optimized for streaming: WebDataset (tar archives of tensors), MosaicML StreamingDataset, or TFRecord. These formats minimize random I/O and maximize throughput.

Random access (debugging, data exploration). Use formats with indexing: Lance, Parquet, or Zarr. These allow querying specific samples without loading the entire dataset.

Avoid storing millions of small files (individual JPEGs). Filesystem metadata operations become a bottleneck. Archive into sharded tar files (WebDataset) or columnar formats.

Checkpoint Strategy

• Frequency. Save checkpoints every 500–1,000 training steps. More frequent = less wasted compute on restart, more storage cost.
• Retention. Keep the last 3–5 checkpoints on NVMe during training. Archive every Nth checkpoint (e.g., every 5,000 steps) to object storage.
• Size. A 7B-parameter model checkpoint is approximately 14 GB in FP16. Ten checkpoints = 140 GB. At $2.49/TB/month, storing 100 checkpoints costs about $3.50/month.

Caching

For repeated training runs on the same dataset, keep a cached copy on object storage in an optimized format. The first run converts raw data to the training format; subsequent runs skip conversion. This eliminates 10–30% of training startup time.

Frequently Asked Questions