Resilient GPU Compute on DigitalOcean Kubernetes: Surviving Spot Interruptions

Spot GPU Droplets are capacity that’s not currently being used by customers and, rather than sitting idle and wasting electricity and cooling, is offered to customers at a discounted price with the understanding that DigitalOcean can reclaim that resource at any time. The reclamation will be preceded with 2 hour notifications (best effort) by email.

In order to maximize the value of using Spot GPU Droplets, customers will benefit greatly from using them in stateless, decoupled, fault tolerant workloads, like inference and other short-lived processes that don’t persist connections or data.

For longer-running work on Spot GPU Droplets, techniques like checkpointing and offloading are recommended so that work can be performed in blocks. After an interruption, work can continue when using techniques like this, rather than starting over completely.

Running GPU workloads on Kubernetes often presents a tough trade-off between cost and reliability. Spot GPU Droplets offer substantial cost savings for compute-heavy tasks like AI inference, batch image processing, and model training. But Spot capacity can be reclaimed by DigitalOcean during periods of high platform demand — and when it is, DOKS reclaims the entire Spot node pool, not one node at a time. Your infrastructure has to be designed around that fact, not around a “node dies, autoscaler quietly relaunches it” assumption.

In this post, we demonstrate how to build a DOKS cluster that fails over cleanly when that happens: a Spot GPU pool for cost-effective steady-state work, and a pre-provisioned On-Demand GPU pool that stands ready to take over the moment Spot capacity is reclaimed.

To validate this behavior, we’ll walk through an automated reclaim simulation demo available in the companion Github repository.

To handle a reclaim gracefully, we split our DOKS cluster into specialized node pools:

+-----------------------------------------------------------------+
|                          DOKS Cluster                           |
|                                                                 |
|  +----------------+   +------------------+   +----------------+ |
|  |  System Pool   |   |     Spot GPU     |   | On-Demand GPU  | |
|  |                |   |                  |   |                | |
|  |                |   |                  |   |                | |
|  |                |   |                  |   |                | |
|  +----------------+   +------------------+   +----------------+ |
|         |                      |                      |         |
|   apps & addons                 -->  GPU workload  <--          |
+-----------------------------------------------------------------+

• Dedicated System Pool: An On-Demand pool reserved strictly for core system services, ingress controllers, and operational tooling (like kube-ops-view). System processes should never run on Spot hardware.
• Spot GPU Pool: A node pool of Spot GPU Droplets, labeled gpu-pool-type=spot and tainted so only the GPU workload can land there. This is the pool DOKS reclaims — in full — when it needs the capacity back.
• On-Demand GPU Pool (fallback): A second pool of matching On-Demand GPU Droplets, labeled gpu-pool-type=on-demand and left untainted. It’s created up front, at 0 running nodes, so it costs nothing until a reclaim actually happens.

Designing for Spot resilience requires matching your application architecture to ephemeral compute mechanics — and to how DOKS specifically reclaims Spot capacity.

1. Statelessness & Checkpointing

Your application pods must tolerate sudden, whole-pool eviction. For API inference or batch queues, stateless worker pods are ideal. For long-running ML training, ensure your application periodically saves checkpoints to DigitalOcean Block Storage or Spaces — a reclaim doesn’t leave you a node to finish a job on.

2. Matching Hardware Specs

Map your On-Demand fallback pool to identical GPU hardware as best possible. Sometimes the exact droplets will be available in both Spot and On-Demand, when that’s not possible you’ll need to ensure the two pools both meet the minimum requirements of the workload.

3. The Autoscaler “Fixed-Pool” Catch

A key operational detail of Cluster Autoscaler on any Kubernetes installation: it completely ignores fixed-size node pools. Provision your Spot pool without autoscaling and any ordinary node loss just leaves evicted pods Pending indefinitely, Cluster Autoscaler logs a NotTriggerScaleUp verdict and does nothing, because a fixed-size pool isn’t something it manages at all.

The fix is --auto-scale --min-nodes N --max-nodes N: min equal to max keeps the steady-state node count fixed during normal operation, while still giving Cluster Autoscaler visibility into the pool. This matters for ordinary node churn within an intact pool — it does not help when the whole pool is reclaimed, because at that point the pool itself no longer exists for the autoscaler to manage. That’s what the On-Demand fallback pool is for.

4. A Reclaimed Spot Pool Doesn’t Come Back

DOKS reclaims a Spot pool via its DeleteNodePool API — cordoning, draining (respecting PodDisruptionBudgets), and deleting every node in the pool, all in one action. Nothing relaunches that pool automatically afterward. Getting Spot capacity back means creating a brand-new Spot pool, at whatever rate is in effect at that time. Design your fallback around a separate, independent pool — not around waiting for the original one to reappear.

The doks-spot-demo repository includes everything needed to launch the cluster, simulate a reclaim, and watch the infrastructure fail over automatically.

Prerequisites

• doctl authenticated to your DigitalOcean account (doctl auth init).
• kubectl and jq installed on your local PATH.

Step 1: Launch the Cluster Infrastructure

Run launch.sh to provision the project, DOKS cluster, system pool, Spot GPU pool, an idle On-Demand fallback pool, the visual dashboard (kube-ops-view), and a test deployment:

# Usage: ./launch.sh [system-size] [spot-gpu-size] [ondemand-gpu-size]
./launch.sh

(By default, this provisions lightweight compute nodes for testing the reclaim/failover mechanics without incurring GPU costs. You can pass real `gpu` size slugs as arguments once capacity or the GPU Spot preview program allows.)*

The demo workload’s pod spec carries a toleration for the Spot pool’s taint, plus node affinity that makes it eligible on either pool but prefers Spot (preferredDuringSchedulingIgnoredDuringExecution) — the label/taint/affinity pattern DigitalOcean recommends for a Spot-with-fallback setup, applied directly rather than left as an exercise for the reader.

Step 2: Open kube-ops-view

Once launch.sh completes, open the visual cluster dashboard to monitor node state in real time:

kubectl -n doks-spot-demo-ops get svc kube-ops-view
# Open http://<EXTERNAL-IP> in your browser

You’ll see two active node groups at this point — the System pool hosting system pods and kube-ops-view, and the Spot GPU pool carrying the demo workload. The On-Demand fallback pool exists in the cluster’s pool list but has 0 nodes, so it won’t show up in kube-ops-view until a reclaim brings it to life.

Step 3: Simulate a Spot Reclaim

To test the resilience setup, run simulate-interruption.sh.

Rather than deleting a single Droplet, this script calls the DOKS node-pool API to delete the entire Spot pool — the same mechanism a real reclaim uses. There’s no manual cordon/drain step: DeleteNodePool already cordons, drains (respecting PodDisruptionBudgets), and deletes every node in the pool as part of the same call.

./simulate-interruption.sh

What Happens Under the Hood

1. Pool-wide reclaim: The script calls doctl kubernetes cluster node-pool delete on the Spot pool. DOKS cordons, drains, and deletes every node in it — the whole pool, not one node.
2. Reclaim signal, immediately observable: The script prints the Node events fired the moment the reclaim starts — the same signal a third-party scheduler (Kueue, Volcano, Ray) could react to without polling DO’s side.
3. Nowhere left to run: With the Spot pool gone, the workload’s pod goes Pending — there’s no partial pool left for an autoscaler to patch.
4. On-Demand fallback comes online: The script brings the pre-provisioned On-Demand pool up from 0 to its target node count. (Cluster Autoscaler can’t scale an autoscaling pool up from zero on its own today, so the script performs that scale-up directly — the end state is the same one a fully hands-off setup would reach.)
5. Cluster restored: Once the On-Demand node is Ready, Kubernetes’ own scheduler places the pod there via its toleration and affinity — no manual pod placement. The script prints total recovery time (typically within a few minutes).

Restoring Spot capacity afterward means creating a brand-new Spot pool — the reclaimed one is gone for good, and the new one locks in whatever rate is in effect then.

Step 4: Tear Down Resources

When testing is complete, clean up all created resources (cluster, load balancers, and volumes):

./teardown.sh --yes

Note that this script includes an automated sweep to guarantee no worker Droplets are left running out-of-band.

A DOKS Spot reclaim takes down an entire node pool at once — there’s no partial loss to autoscale around. The resilient pattern is a pre-provisioned, independent On-Demand pool with the right labels, taint, and Spot-preferred affinity already in place, so the workload fails over the moment Spot capacity disappears rather than waiting on a pool that isn’t coming back. Treating reclaims as routine infrastructure events — and practicing the drill — lets you keep the cost advantages of Spot compute without sacrificing reliability.

Explore the complete automation scripts and launch your own test cluster by visiting the doks-spot-demo repository.