Spot and Preemptible GPU Instances: Save 60–90% on AI Compute

Spot instances — also called preemptible instances — are unused GPU capacity that cloud providers sell at steep discounts, typically 60–90% below on-demand rates. The trade-off: the provider can reclaim the GPU with as little as 30 seconds of notice. For the right workloads, spot instances turn a $900/month GPU bill into $90–$180/month. This guide covers how spot pricing works, which workloads qualify, and how to build fault-tolerant pipelines that survive preemption.

How Spot Pricing Works

Cloud GPU providers have excess capacity. GPUs sit idle between customer reservations, during off-peak hours, and when hyperscalers over-provision. Rather than let hardware run empty, providers auction this spare capacity as spot instances.

The mechanics:

1. You place a bid at or above the current spot price (or accept the market price).
2. The instance launches immediately if capacity is available.
3. When demand rises and the provider needs the GPU back for a full-price customer, you receive a preemption notice — typically 30–120 seconds.
4. Your instance is terminated. You are not charged for the partial hour in most cases.

Spot pricing is dynamic — it fluctuates with supply and demand. During a GPU shortage, spot prices can spike above on-demand rates. During off-peak hours (nights, weekends), spot prices often hit their floor.

Spot vs. On-Demand vs. Reserved: Cost Comparison

At 500 GPU-hours per month, switching from on-demand to spot saves $1,800–$7,200 annually on a single GPU. For a team running four GPUs, annual savings exceed $28,000.

Workloads That Work on Spot

Ideal: Fault-Tolerant Batch Workloads

Distributed training with checkpointing. Save model checkpoints every N steps. If the instance is preempted, restart from the latest checkpoint. PyTorch and TensorFlow both support resumable training with minimal code changes:

# Save checkpoint every 500 steps
if step % 500 == 0:
    torch.save({
        'step': step,
        'model_state_dict': model.state_dict(),
        'optimizer_state_dict': optimizer.state_dict(),
        'loss': loss,
    }, f'checkpoints/step_{step}.pt')

On restart, load the latest checkpoint and resume. A preemption costs only the GPU time since the last checkpoint — typically a few minutes.

Hyperparameter sweeps. Running 100 independent training runs with different parameters. If 15 get preempted, the remaining 85 complete. No single run is critical; resubmit the preempted configurations to a new spot instance.

Batch inference and data processing. Processing a million images through a model. Track which images are done. Preemption means reprocessing the last few uncheckpointed images — negligible cost.

Rendering and simulation. Rendering animation frames or running Monte Carlo simulations. Each frame or simulation run is independent. A preemption loses at most one unit of work.

Poor Fit: Stateful and Latency-Sensitive Workloads

Real-time inference APIs. Users expect sub-100 ms responses. A 30-second preemption notice followed by a 2-minute cold start creates unacceptable downtime. Use reserved or on-demand instances for production inference.

Interactive development. Jupyter notebooks, debugging sessions, and exploratory data analysis. Losing an hour of in-progress work to a preemption destroys productivity. Use on-demand instances for development.

Databases and stateful services. Any workload where data lives only in memory and cannot be reconstructed quickly. Preemption means data loss.

Long-running single jobs without checkpointing. A 72-hour training run with no checkpointing. A preemption at hour 71 wastes 71 hours of GPU time. Always add checkpointing before using spot.

Building a Spot-Ready Pipeline

Checkpointing Strategy

Checkpoint frequency determines your maximum wasted compute on preemption:

Store checkpoints on object storage (S3-compatible), not local NVMe. If the instance is preempted, local storage is gone. Object storage survives and is accessible from the replacement instance.

Instance Replacement Automation

Do not manually restart preempted instances. Automate the recovery flow:

1. Monitor for preemption signals (cloud provider webhook, or SIGTERM handler).
2. On preemption notice: save final checkpoint, upload to object storage, log progress.
3. Request a new spot instance (possibly from a different availability zone or GPU type).
4. On new instance boot: pull latest checkpoint from object storage, resume training.

Tools like Kubernetes with spot node pools, Slurm with preemption hooks, or simple bash scripts with provider CLIs handle this automatically.

Multi-Zone and Multi-Provider Diversification

Spot capacity varies by availability zone and provider. If us-east-1 has no spot A100s, us-west-2 might. Diversifying across zones increases the probability of finding spot capacity at any given moment.

Similarly, maintaining accounts with two GPU cloud providers doubles your spot capacity surface area. When Provider A has no spot availability, Provider B might.

Hybrid Architecture: Spot Baseline + On-Demand Fallback

For workloads where throughput matters more than cost (e.g., a training run that must finish by Monday morning), use a hybrid approach:

• Primary: spot instances for as long as they last (cheapest).
• Fallback: if spot instances are unavailable for more than 30 minutes, launch on-demand instances to meet the deadline.
• Cease fallback: once spot capacity returns, terminate on-demand and resume on spot.

This caps the worst-case cost at on-demand rates while capturing spot savings most of the time.

Spot Instance Availability Patterns

Spot availability follows predictable rhythms:

• Weekday business hours (9 AM–5 PM local time). Highest demand from enterprise customers. Spot capacity is tightest and prices are highest.
• Nights and weekends. Enterprise usage drops. Spot capacity increases, prices reach their floor.
• GPU generation transitions. When a new GPU generation launches (e.g., H200), demand for the previous generation (H100) drops temporarily, creating spot surplus on older hardware.
• End of quarter / end of year. Cloud providers clear capacity before financial reporting periods. Spot availability spikes.

Schedule flexible workloads for nights and weekends to maximize spot availability and minimize cost.

Frequently Asked Questions