Migrating from AWS S3 to S3-Compatible Storage: Cost and Performance

AWS S3 is the industry standard for object storage, but its pricing model — particularly data egress fees — creates a cost structure that penalizes AI workloads. Downloading a 1 TB dataset from S3 costs $90 in egress fees. Running monthly training cycles on that dataset adds $1,080/year in egress alone. S3-compatible alternatives with zero egress fees eliminate this cost entirely. This guide covers the migration process: cost comparison, data transfer methods, compatibility validation, and cutover planning.

The Cost Case for Migration

AWS S3 Pricing

S3-Compatible Alternative (Typical)

Annual savings for 1 TB with 500 GB monthly egress: $786. For teams managing 10 TB of AI datasets with frequent downloads, annual savings exceed $7,800.

The savings compound when egress volume is high — which it always is for AI workloads that download datasets for training runs, download model checkpoints for evaluation, and distribute models to inference endpoints.

Migration Planning

Phase 1: Inventory

List all S3 buckets, their sizes, and access patterns:

aws s3 ls --summarize --human-readable

# Check bucket sizes
aws s3 ls s3://bucket-name --summarize --human-readable | grep "Total Size"

# Check lifecycle policies
aws s3api get-bucket-lifecycle-configuration --bucket bucket-name

# Identify public buckets (no migration needed if read-only)
aws s3api get-bucket-policy --bucket bucket-name

Prioritize buckets by egress volume — the highest-egress buckets produce the greatest savings. Archive-only buckets with zero egress can be migrated last or left on AWS.

Phase 2: Compatibility Validation

Not all S3-compatible services implement every API feature. Test the following against the target service:

Required for AI workflows: - PUT, GET, DELETE, HEAD operations - Multipart upload (objects > 5 GB) - Range GET (partial object reads for streaming datasets) - ListObjects and ListObjectsV2 with prefix/delimiter - Pre-signed URLs (for sharing checkpoint downloads)

Nice to have: - Versioning (for model registry) - Lifecycle policies (for automated checkpoint cleanup) - Object locking (for compliance) - CORS configuration (for web-based tools)

Validation script:

import boto3

def validate_s3_compat(endpoint, key, secret, bucket):
    s3 = boto3.client('s3', endpoint_url=endpoint,
                      aws_access_key_id=key, aws_secret_access_key=secret)

    # Test PUT + GET
    s3.put_object(Bucket=bucket, Key='test.txt', Body=b'hello')
    resp = s3.get_object(Bucket=bucket, Key='test.txt')
    assert resp['Body'].read() == b'hello'

    # Test multipart upload
    mpu = s3.create_multipart_upload(Bucket=bucket, Key='large.bin')
    parts = []
    for i in range(2):
        part = s3.upload_part(Bucket=bucket, Key='large.bin',
                              PartNumber=i+1, UploadId=mpu['UploadId'],
                              Body=b'x' * (5 * 1024 * 1024))
        parts.append({'PartNumber': i+1, 'ETag': part['ETag']})
    s3.complete_multipart_upload(Bucket=bucket, Key='large.bin',
                                  UploadId=mpu['UploadId'],
                                  MultipartUpload={'Parts': parts})

    # Test Range GET
    resp = s3.get_object(Bucket=bucket, Key='large.bin', Range='bytes=0-99')
    assert len(resp['Body'].read()) == 100

    # Test List
    resp = s3.list_objects_v2(Bucket=bucket, Prefix='test', MaxKeys=10)
    assert len(resp['Contents']) >= 2

    # Test pre-signed URL
    url = s3.generate_presigned_url('get_object',
                                     Params={'Bucket': bucket, 'Key': 'test.txt'},
                                     ExpiresIn=3600)
    assert url.startswith('http')

    print("All S3 compatibility tests passed")

Phase 3: Data Transfer

For datasets under 1 TB, direct transfer is straightforward:

# Option A: rclone (recommended — handles retries, verification, parallelism)
rclone sync s3_aws:bucket-name s3_new:bucket-name \
  --progress \
  --transfers 16 \
  --checkers 32 \
  --s3-upload-concurrency 8

# Option B: AWS CLI to local, then to new provider
aws s3 sync s3://source-bucket ./temp-data/
aws s3 --endpoint-url https://new-provider.com sync ./temp-data/ s3://dest-bucket/

For datasets over 1 TB, use an intermediate compute instance to avoid egress charges on the transfer itself:

1. Provision a GPU instance on the new provider (since the GPU instance has fast, free connectivity to the new provider's S3).
2. On that instance, pull data from AWS S3 and push to the new provider's S3 simultaneously.
3. The AWS egress fee applies once for the initial transfer. After that, all subsequent access is on the zero-egress provider.

Alternatively, if you already have the dataset locally or on an existing GPU instance, push directly to the new provider — no AWS egress involved.

Phase 4: Application Cutover

Update environment variables in all training scripts, CI/CD pipelines, and inference deployments:

# Old
export AWS_ENDPOINT_URL_S3="https://s3.amazonaws.com"
export AWS_DEFAULT_REGION="us-east-1"

# New
export AWS_ENDPOINT_URL_S3="https://s3.your-provider.com"
export AWS_DEFAULT_REGION="auto"

Update boto3 client initialization:

# Old
s3 = boto3.client('s3')

# New
s3 = boto3.client(
    's3',
    endpoint_url=os.environ['AWS_ENDPOINT_URL_S3'],
    aws_access_key_id=os.environ['AWS_ACCESS_KEY_ID'],
    aws_secret_access_key=os.environ['AWS_SECRET_ACCESS_KEY'],
)

Phase 5: Validation and Decommission

Before decommissioning AWS S3 buckets:

1. Run a full training cycle end-to-end using only the new provider.
2. Verify checkpoint save/load works correctly.
3. Verify model export and deployment reads from the new storage.
4. Keep AWS S3 buckets in read-only mode for 30 days as a safety net.
5. After 30 days with no issues, delete AWS S3 buckets and verify the final AWS bill shows zero S3 charges.

Performance Comparison

S3-compatible services vary in throughput. Benchmark before committing:

# Write test: upload 10 GB of random data
dd if=/dev/urandom bs=1M count=10240 of=test.bin
time s5cmd cp --concurrency 16 test.bin s3://new-bucket/benchmark/test.bin

# Read test: download and discard
time s5cmd cp --concurrency 16 s3://new-bucket/benchmark/test.bin /dev/null

# Metadata test: list 10,000 objects
time aws s3 ls s3://new-bucket/benchmark/ --endpoint-url https://new-provider.com --recursive | wc -l

Expected results for a good S3-compatible service on a 10 Gbps link:

For AI workloads, write throughput is less critical (datasets are written once, read many times). Read throughput matters for staging data to NVMe before each training run.

Common Migration Pitfalls

1. Virtual Hosted-Style vs Path-Style URLs

AWS S3 uses virtual hosted-style URLs (bucket.s3.amazonaws.com). Many S3-compatible services use path-style URLs (s3.provider.com/bucket). Most S3 clients auto-detect, but some libraries require explicit configuration. If you see "NoSuchBucket" errors after migration, switch to path-style addressing:

s3 = boto3.client('s3', config=Config(s3={'addressing_style': 'path'}))

2. Signature Version

AWS S3 supports Signature v2 and v4. Most S3-compatible services require v4. This is the default in modern SDKs but may need explicit configuration in older tools.

3. ETag Differences

Some S3-compatible services compute ETags differently for multipart uploads. If your application uses ETags for integrity verification, test multipart ETag behavior before migrating production data.

4. Rate Limiting

S3-compatible services may impose per-bucket rate limits (requests per second). Check the provider's limits and configure your clients to stay within them. s5cmd and rclone have built-in rate-limiting flags.

Frequently Asked Questions