High-Performance Storage for AI and ML Workloads
AI workloads are I/O-intensive in ways that traditional enterprise applications are not. Training reads terabytes of data sequentially; checkpointing writes gigabytes in bursts; inference requires microsecond-latency model weight access. Standard cloud block storage — designed for databases and web servers — is often the wrong tool. This guide covers the storage technologies purpose-built for AI/ML and how to assemble them into a cost-effective architecture.
The AI I/O Profile
AI workloads have a distinct I/O pattern that differs sharply from traditional applications:
| Characteristic | AI Training | AI Inference | Traditional DB/Web |
|---|---|---|---|
| Read pattern | Sequential, large blocks (1–100 MB) | Random, small blocks (4–64 KB) | Mixed random |
| Write pattern | Burst (checkpoints, logs) | Minimal | Continuous |
| Throughput requirement | 1–10 GB/s | 100–500 MB/s | 10–100 MB/s |
| Latency tolerance | Moderate (10–100 ms OK) | Low (<1 ms) | Low (<5 ms) |
| Working set size | 100 GB–10 TB | 1–50 GB | 1–100 GB |
| Data persistence needs | Ephemeral (reproducible) | Critical (model weights) | Critical |
Training is a throughput game — optimize for GB/s, not IOPS. Inference is a latency game — optimize for microseconds, not GB/s. The storage architecture must serve both profiles.
Storage Technologies for AI
NVMe SSDs
How they work. Flash storage connected directly to the PCIe bus. No SATA/SAS controller, no HBA bottleneck. Gen4 NVMe drives deliver 7 GB/s sequential read and 1M random read IOPS.
AI fit. The default choice for GPU-attached storage. Training datasets, model weights, temporary checkpoints, and Docker images all belong on NVMe during active use.
Limitations. Capacity caps at 2–4 TB per drive in typical cloud instances. Not shared across instances without a network layer. Expensive to keep provisioned when GPUs are idle.
NVMe-oF (NVMe over Fabrics)
How it works. NVMe commands encapsulated over RDMA (InfiniBand or RoCE) or TCP networks. Remote NVMe drives appear as local block devices with near-local latency (<100 microseconds over RDMA).
AI fit. Multi-node training where multiple GPU servers need shared high-performance storage. Replaces NFS for latency-sensitive shared storage. Delivers 90–95% of local NVMe performance with centralized management.
Limitations. Requires RDMA-capable networking (InfiniBand or 100+ GbE with RoCE). More expensive than local NVMe. Overkill for single-node workloads.
Parallel Filesystems (Lustre, GPFS, WEKA)
How they work. Distributed filesystems that stripe data across multiple storage servers, aggregating throughput. Lustre routinely achieves 1 TB/s at scale in HPC deployments.
AI fit. Large-scale training clusters (16+ GPUs) where the dataset exceeds single-node capacity and throughput must scale with GPU count. WEKA and Lustre are common in frontier AI labs.
Limitations. Complex to deploy and manage. Expensive at small scale (<8 GPUs). WEKA is simpler but proprietary; Lustre is open-source but operationally demanding.
S3-Compatible Object Storage
How it works. HTTP-based storage for immutable objects. Unlimited capacity, high durability, eventual consistency. Throughput: 100–500 MB/s per object, scales horizontally with parallel requests.
AI fit. Cold storage for datasets, archived checkpoints, model weights, logs. The source of truth from which NVMe is populated. The cost floor for data at rest.
Limitations. Latency too high (10–50 ms) for direct training access. Not a filesystem — cannot be mounted and used like a local drive without a translation layer (s3fs, which adds latency and reduces throughput).
Memory-Mapped Storage
How it works. Files mapped directly into process virtual memory. The OS pages data from storage to RAM on demand. Zero-copy access — no read() syscalls.
AI fit. Pre-processed tensor datasets. A 500 GB memmap file of pre-tokenized text embeddings loads instantly (virtual memory allocation is near-instant); pages are faulted in as the GPU reads them.
Limitations. Requires the dataset to fit in the virtual address space (not a problem on 64-bit systems). Random access patterns cause page thrashing if the working set exceeds RAM.
Storage Architecture by Workload Type
Single-GPU Training
NVMe (active) ← staged from ← S3 Object Storage (source of truth)
Dataset staged to NVMe before training. Training reads from NVMe. Checkpoints pushed to S3 for durability. NVMe deleted when GPU is stopped.
Cost for 500 GB dataset, 80 GPU-hours/month: $2.50 (S3) + $1.50 (NVMe hours) = $4/month.
Multi-Node Training (4–8 GPUs)
NVMe per-node (hot data) + Network FS (shared dataset) ← S3 (cold data)
Frequently accessed data cached on per-node NVMe. Full dataset on network filesystem accessible by all nodes. Cold data on S3.
Cost for 2 TB dataset, 320 GPU-hours/month: $12 (S3) + $200 (network FS) + $6 (NVMe) = $218/month.
LLM Inference (24/7)
NVMe (model weights) + RAM (KV-cache)
Model weights loaded into GPU VRAM at startup from NVMe. KV-cache lives in GPU VRAM during generation. No storage I/O during inference except for logging.
Cost for 14 GB model, 730 hours/month: $50 (NVMe persistent) + $1 (S3 backup) = $51/month.
Batch Inference (Burst)
S3 → NVMe (staged) → GPU → S3 (results)
Input data staged from S3 to NVMe. Inference runs. Results pushed to S3. NVMe released. No persistent storage cost.
Cost for 100 GB input, 10 GPU-hours/month: $0.50 (S3) + $1.50 (GPU) + $0 (NVMe, released) = $2/month.
Storage Performance Tuning
I/O Scheduler
Linux I/O schedulers optimized for HDDs (mq-deadline, bfq) add unnecessary latency for NVMe. Set the scheduler to none (no-op) for NVMe devices:
echo none > /sys/block/nvme0n1/queue/scheduler
This reduces per-I/O latency by 20–50 microseconds — small in isolation but meaningful at millions of I/Os during training.
Read-Ahead
The Linux kernel prefetches data it predicts will be read next. For sequential training workloads, increase read-ahead:
blockdev --setra 8192 /dev/nvme0n1
This tells the kernel to prefetch 8,192 sectors (4 MB) ahead of the current read position — enough to cover the next batch while the current batch processes.
Filesystem Choice
| Filesystem | Strengths | Weaknesses | AI Fit |
|---|---|---|---|
| ext4 | Mature, reliable, low overhead | No snapshots, no compression | Good for single-drive NVMe |
| XFS | Excellent large-file performance, parallel I/O | Slightly higher CPU overhead | Best for training datasets |
| ZFS | Compression, snapshots, checksums | Memory-hungry (1 GB RAM per TB) | Good for checkpoint storage |
| btrfs | Compression, snapshots, CoW | Performance variability under load | Avoid for training data |
For training dataset volumes, XFS provides the best balance of throughput and CPU efficiency. For checkpoint and model storage, ZFS compression (lz4) reduces storage footprint 20–40% for FP16 tensors.
Cost Optimization Strategy
Tier Data by Access Pattern
| Access Frequency | Storage Tier | Cost per TB/Month |
|---|---|---|
| Active (hourly) | Local NVMe | $70–$150 |
| Warm (daily) | Network filesystem | $50–$100 |
| Cold (weekly/monthly) | S3-compatible object | $2–$6 |
| Archive (rarely) | S3 Glacier-equivalent | $1–$2 |
Move data down the tiers as access frequency decreases. Automate the lifecycle: data not accessed for 30 days moves from NVMe to S3.
Rightsizing
Do not provision storage for peak capacity if average usage is much lower. A team that needs 2 TB for one week per month during a big training run should not pay for 2 TB of persistent NVMe all month. Stage the dataset from S3 only when needed.
Compression
Enable filesystem compression (ZFS lz4) for checkpoint storage. FP16 model weights compress by 20–40% — a 14 GB checkpoint becomes 8–11 GB. Over 100 checkpoints, compression saves 300–600 GB.
Frequently Asked Questions
Do I need a parallel filesystem for 4 GPUs?
No. For single-node 4-GPU training, local NVMe provides more than enough throughput (7 GB/s shared across 4 GPUs = 1.75 GB/s per GPU). A parallel filesystem adds cost and complexity with no benefit at this scale. Parallel filesystems become valuable at 16+ GPUs across 4+ nodes.
How fast does object storage need to be for dataset staging?
Aim for 500+ MB/s aggregate throughput. At this speed, a 500 GB dataset stages in under 17 minutes. Most S3-compatible providers achieve this with 4–8 parallel download threads. If your provider caps at 100 MB/s per bucket, staging 500 GB takes 83 minutes — consider a different provider or parallelize across multiple buckets.
Can I train directly from object storage?
Not recommended. The 10–50 ms latency per read operation destroys throughput for sequential training workloads. A training loop reading 1 MB chunks at 50 ms latency achieves 20 MB/s — 100× slower than NVMe. Some streaming dataset libraries (MosaicML, WebDataset) use aggressive prefetching to hide object storage latency, but staging to NVMe before training is simpler and more reliable.
What RAID level should I use for training NVMe drives?
None. In cloud GPU environments, NVMe drives are typically single virtual devices backed by provider-managed redundancy. If you are configuring bare metal with multiple physical NVMe drives, use RAID-0 (striping) for maximum training throughput. RAID-1 or RAID-5 sacrifices capacity and performance for redundancy — unnecessary for training data that can be re-staged from object storage.
BHK Cloud offers NVMe SSD-equipped RTX 3090 instances at $0.15/hr and S3-compatible storage at $2.49/TB/month with zero egress fees. Right-size your storage: NVMe for active work, S3 for everything else. Rent a GPU or explore S3 storage.