S3-Compatible Object Storage for AI Datasets and Model Weights

Object storage — specifically the S3 API pioneered by AWS — has become the universal data layer for AI. Datasets, model checkpoints, training logs, and deployment artifacts all live in S3-compatible buckets. At $2–$6 per terabyte per month, it costs less to store a 500 GB dataset for a year ($12–$36) than to provision the equivalent NVMe for a single month ($35–$100). This guide covers how to use S3-compatible storage effectively in AI workflows: what the S3 API provides, which features matter for AI, how to structure buckets and objects, and how to integrate with training and inference pipelines.

What "S3-Compatible" Means

S3 compatibility means the storage service implements the AWS S3 REST API — the HTTP verbs, headers, response codes, and XML structures that S3 clients expect. Any tool built for S3 (AWS CLI, boto3, s5cmd, rclone, MinIO client, TensorFlow I/O, PyTorch S3 integration) works with an S3-compatible service.

S3 compatibility comes in two levels:

• Core S3 API. PUT, GET, DELETE, HEAD, ListObjects on buckets and objects. Multipart uploads. Pre-signed URLs. This is the minimum viable set.
• Extended S3 API. Versioning, object locking, lifecycle policies, bucket policies, CORS, event notifications, replication. These matter for production AI pipelines.

For AI workloads, the core API is sufficient. Versioning and lifecycle policies become valuable when managing model registries and automated checkpoint cleanup.

Bucket Structure for AI Projects

A well-structured bucket layout separates concerns and enables lifecycle automation:

s3://ai-projects/
├── datasets/
│   ├── imagenet-1k/
│   │   ├── train/
│   │   │   ├── shard-000000.tar
│   │   │   ├── shard-000001.tar
│   │   │   └── ...
│   │   └── val/
│   ├── custom-dataset-v3/
│   │   ├── metadata.json
│   │   └── samples/
│   └── common-crawl-2025/
├── checkpoints/
│   ├── experiment-42/
│   │   ├── step-000500.pt
│   │   ├── step-001000.pt
│   │   └── best.pt
│   └── experiment-43/
├── models/
│   ├── llama-3-8b-instruct/
│   │   ├── config.json
│   │   ├── model-00001-of-00004.safetensors
│   │   └── ...
│   └── fine-tuned-medical-v2/
└── logs/
    └── training/
        ├── 2026-07/
        └── 2026-08/

Key principles: - Separate datasets, checkpoints, models, and logs into distinct prefixes. This enables different lifecycle policies (e.g., delete old logs after 30 days, keep model checkpoints for 1 year). - Version datasets. Do not overwrite. A custom-dataset-v3 prefix tells you what data was used. When you improve the dataset, create v4 — old experiments remain reproducible. - Shard large datasets. Individual objects should be 100 MB–5 GB. Smaller than 100 MB and LIST/GET overhead dominates. Larger than 5 GB and multipart upload is mandatory. WebDataset tar shards of 1–2 GB are ideal.

Tools for S3 in AI Workflows

s5cmd: Fast Bulk Operations

s5cmd is a Go-based S3 client optimized for throughput. It parallelizes operations across multiple connections and saturates a 10 Gbps link:

# Download dataset shards in parallel
s5cmd cp --concurrency 16 "s3://bucket/datasets/imagenet-1k/train/*.tar" ./data/

# Upload checkpoints
s5cmd cp ./checkpoints/step-*.pt s3://bucket/checkpoints/experiment-42/

For staging datasets to NVMe before training, s5cmd with 16 concurrent connections achieves 800–1,200 MB/s on most S3-compatible services.

rclone: Universal Sync

rclone supports 40+ storage backends and is the Swiss Army knife for data movement:

# Sync local directory to S3 bucket
rclone sync ./datasets s3:ai-projects/datasets --progress

# Mount S3 bucket as local filesystem (read-only)
rclone mount s3:ai-projects ./mnt/s3 --vfs-cache-mode full

The mount feature is useful for exploration but should not be used for training — the added latency degrades throughput.

AWS CLI with S3-Compatible Endpoint

The standard AWS CLI works with any S3-compatible service by setting the endpoint:

aws configure set aws_access_key_id YOUR_KEY
aws configure set aws_secret_access_key YOUR_SECRET
aws s3 ls --endpoint-url https://s3.your-provider.com s3://bucket/

Python (boto3)

import boto3
from botocore.config import Config

s3 = boto3.client(
    's3',
    endpoint_url='https://s3.your-provider.com',
    aws_access_key_id='YOUR_KEY',
    aws_secret_access_key='YOUR_SECRET',
    config=Config(max_pool_connections=50)
)

# Upload with automatic multipart for large files
s3.upload_file('model.pt', 'bucket', 'models/model.pt')

# Download
s3.download_file('bucket', 'datasets/shard-000000.tar', '/data/shard-000000.tar')

For PyTorch training, use the s3:// protocol in WebDataset or MosaicML's StreamingDataset — both handle S3 authentication and parallel prefetching natively.

Integration with Training Pipelines

Stage-Before-Training Pattern

1. s5cmd cp s3://bucket/dataset/*.tar /data/nvme/
2. python train.py --data-dir /data/nvme/
3. python upload_checkpoints.py --to s3://bucket/checkpoints/
4. # Terminate GPU instance (NVMe data deleted)

This is the simplest and most reliable pattern. It costs a few minutes of staging time but eliminates storage-related training failures. A 500 GB dataset stages in 7–14 minutes on a 10 Gbps link.

Streaming Pattern

For datasets too large to stage (multi-TB), stream directly from S3:

import webdataset as wds

dataset = wds.WebDataset(
    "s3://bucket/datasets/large-corpus/shard-{000000..000999}.tar",
    shardshuffle=True
).decode("rgb").to_tuple("jpg", "cls")

WebDataset handles S3 authentication, parallel prefetching, and shard shuffling. The library prefetches upcoming shards while the GPU trains on the current shard, hiding network latency.

Checkpoint Management

import boto3
import torch
import os

def save_checkpoint(model, optimizer, step, s3_path):
    local_path = f"/tmp/checkpoint-{step}.pt"
    torch.save({
        'step': step,
        'model': model.state_dict(),
        'optimizer': optimizer.state_dict(),
    }, local_path)

    s3.upload_file(local_path, BUCKET, s3_path)
    os.remove(local_path)

def load_checkpoint(s3_path):
    local_path = "/tmp/resume.pt"
    s3.download_file(BUCKET, s3_path, local_path)
    return torch.load(local_path)

Push checkpoints to S3 immediately after saving. In the event of instance preemption or failure, the latest checkpoint is safe in S3 and can be loaded on a new instance.

Cost Modeling

S3 vs Persistent NVMe

Data Transfer Costs

Most S3-compatible providers charge for API requests (PUT, GET, LIST). Typical pricing: $0.005 per 1,000 PUT requests, $0.0004 per 1,000 GET requests. For AI workloads, storage costs dominate — API request costs are typically under 1% of total storage spend.

Egress — downloading data from S3 to the internet or to another provider — is where costs can surprise. AWS S3 charges $0.09/GB for egress. S3-compatible alternatives often charge $0.00/GB. Downloading a 500 GB dataset from AWS costs $45; from a zero-egress S3 provider it costs $0.

Frequently Asked Questions