Spot instances without the sleepless nights: a practical playbook
Spot capacity can cut compute costs by 60–90%, but only if your workloads are built to survive interruptions. Here is how to capture the savings safely.
Spot instances are the cloud's best-kept discount: spare capacity sold at 60–90% off on-demand prices. The catch is that the provider can reclaim it with as little as two minutes' notice. Used carelessly, spot is an outage waiting to happen. Used deliberately, it is the single largest compute saving available to a scaleup.
Which workloads belong on spot
The rule is simple: anything stateless, checkpointable, or retryable. Good candidates are CI runners, batch processing, rendering, data pipelines, and containerised web services behind a load balancer with enough replicas. Bad candidates are primary databases, single-instance stateful services, and anything that cannot finish a unit of work in a few minutes.
Build interruption tolerance in layers
Start with diversification. Request spot capacity across multiple instance types and availability zones — a spot pool with ten instance families rarely dries up completely. Add automatic fallback: when spot capacity is reclaimed, your orchestrator should be able to shift the workload to on-demand or reserved capacity rather than dropping it. Kubernetes makes this tractable with node groups, taints, and priority classes; the managed spot offerings from the big providers handle much of the fencing for you.
Checkpoint everything that takes time
Any job that runs longer than a few minutes should save progress and resume from the last checkpoint. This turns an interruption from a lost hour into a lost minute. Queue-based architectures get this almost for free: the message returns to the queue and another worker picks it up.
Keep a stable floor
Never run your minimum viable capacity on spot. A common pattern is 60–70% of baseline load on commitments or on-demand, with spot absorbing the peaks. That way a total spot outage degrades performance rather than taking the product down.
Measure what spot actually saves you
Track the blended compute rate per workload, not the headline discount. Include the cost of interruptions, retries, and the engineering time spent maintaining the setup. For most scaleups the honest number is still a 40–60% reduction on the eligible fleet — well worth the discipline.
bluebill helps teams identify which parts of their fleet are safe spot candidates and watches the blended rate so the savings show up in the invoice, not just the spreadsheet.