Fault-Tolerant Training on Spot GPU Droplets: Checkpoint, Resume, and Keep Your Run Alive

• Spot is the self-serve, hourly path to B300, MI350X, and MI355X on DigitalOcean. Size slugs end in -spot; both 1-GPU and 8-GPU shapes exist. The rate is locked at creation for the life of the Droplet, so a running job’s price never moves.
• Two hours of notice is generous, and the loop does not depend on it. DigitalOcean targets a two-hour email before a reclaim, which is enough time to finish a checkpoint calmly and even use the remaining GPU time. Periodic checkpoints to Spaces make the job safe regardless; the notice and the SIGTERM handler turn a reclaim into a non-event.
• Measured on MI355X, 201M-param model with AdamW: each checkpoint is 2,417 MB and takes 9 to 12 seconds to upload to Spaces NYC3 from MEM1. Checkpointing every 600 steps at 3.5 steps per second costs about 6 percent of wall clock; every 1,200 steps costs about 3 percent.
• The reclaim drill, end to end: systemctl stop to clean exit with a final checkpoint uploaded took 13.7 seconds. Destroying the Droplet and creating a replacement with cloud-init took 59 seconds to “active” and 5 minutes 24 seconds to “training again from the saved step.” Zero steps were lost.
• The whole drill, two Droplets and a full 3,000-step run, cost about $2.50 at the locked $4.50 per hour rate.
• Code is on GitHub: anishsingh20/spot-gpu-checkpoint-resume. Clone it, swap in your model, run it.

GPU Droplets come in two capacity tiers that serve the same GPU configurations. The product comparison draws the line simply: on-demand is guaranteed capacity at a fixed rate that is never reclaimed; spot is as-available idle capacity at a rate that may change daily for new Droplets but locks at creation for each one, and it can be reclaimed with a two-hour notice target.

The Public Preview lineup on September 9, 2026, from the sizes API (doctl compute size list) and the console:

Three practical notes from running this:

1. Capacity is live, so check two regions. Spot draws from whatever is idle right now. During my run RIC1 showed “GPUs in this datacenter are currently unavailable” while MEM1 had plenty. If your GPU is listed in two regions, script your create command to try both.
2. The 8-GPU shapes are on spot too. An 8x MI355X node at $36 per hour with 40 TB of scratch is a real option for a one-afternoon data-parallel run, and the loop in this tutorial works the same way on it.
3. The scratch disk is 5 TB (1x) or 40 TB (8x) of local NVMe. It is the right place for a dataset copy and staging, and it is local to the Droplet, so anything you want to keep goes to Spaces. On the gpu-amd-base image it shows up as an unformatted /dev/vdc; format and mount it yourself.

Spot is built for batch model training, batch or asynchronous inference, hyperparameter tuning, and rendering. Production inference, latency-sensitive applications, and stateful services belong on on-demand GPU Droplets, which are SLA-backed and never reclaimed. Most AI teams have both kinds of work, and the two tiers are meant to be used together.

Every spot product makes the same trade; the difference is what happens when the capacity is pulled back.

Sources: DigitalOcean Public Preview terms, sections 2.3, 2.4, 3.3; AWS EC2 Spot interruption notices; GCP Spot VM preemption.

Most “handle spot interruptions” guides are written for a two-minute or thirty-second panic window. Two hours is a different design space. It is long enough to let the current checkpoint interval finish, upload, and then keep training until the Droplet actually goes. The notice arrives by email rather than as a metadata signal on the instance, and the terms are careful to say the two hours is a target. The design is therefore layered: periodic checkpoints to Spaces make the job safe on their own, whatever the timing, and the email plus a SIGTERM handler make an ordinary reclaim cost nothing.

This is the life of one spot job. Most of it is normal training with a checkpoint uploaded every N steps, about 10 seconds each on the MI355X. When the reclaim email lands, you have a comfortable window to flush by hand with touch /tmp/checkpoint-now and keep going, or systemctl stop if you would rather bring the run down cleanly. At T zero the Droplet and its local disks are gone and billing stops. Minutes later a replacement running the identical command reads a small LATEST pointer in Spaces and continues from the saved step. The cards underneath sort out what stays with the Droplet (the local disks and any steps since the last upload) and what does not (your checkpoints in Spaces, your code in git). The loop is designed around the checkpoints, so it works whether or not the email arrives on time.

Let’s create one Spot GPU Droplet, confirm we got the GPU we asked for and the price we saw, and install PyTorch on it. By the end you have a working MI355X machine ready to run the training loop in the next section. Along the way you will notice that almost nothing is spot-specific: the size slug ends in -spot, the console has a purchasing toggle, and everything else is a normal Droplet. That is the point. Spot does not ask you to learn a new product.

I used the console for the first Droplet so you can see every choice, and doctl for the replacement so the whole thing can be scripted. The screenshots are from my session on September 9, 2026.

Creating a Spot GPU Droplet using the Console

Go to GPU Droplets and click Create GPU Droplet. The console tells you where spot lives before you pick anything.

Pick your datacenter first. The banner under the region picker names the four spot regions, ATL1, MEM1, RIC1, and MKC1, and greyed-out regions are simply out of GPU capacity at that moment. I chose Memphis. The Summary card on the right already shows the plan I am about to pick, an MI355X (Spot) at $4.50 per hour, along with the reclaim note in plain text.

Choose a purchasing option is the new control: On-demand for SLA-backed availability, Spot for cost-effective compute on workloads that can tolerate interruptions. With Spot selected and AMD as the platform, MEM1 offers two MI355X plans, the 8-GPU shape with 2.25 TB of VRAM and 40 TB of scratch, and the 1-GPU shape with 288 GB of VRAM and 5 TB of scratch, both at $4.50 per GPU-hour. I took the single GPU.

SSH keys, public IPv4, and the optional metrics agent are the same as any Droplet. Keep public IPv4 on for this tutorial so you can SSH in and watch the log. The rate in the Summary card is the one that gets locked when you click Create.

A few seconds after Create, the Droplet page shows provisioning in progress and the cost card already reads $4.500 per hour. This is your locked rate. Tomorrow’s spot price for new Droplets can move; this Droplet’s will not.

About a minute later the Droplet is Active with a public IP to SSH to and a private IP on the default MEM1 VPC. Billing starts here, per second with a five-minute minimum.

Configuration Details confirms the order: one GPU with 288 GB of VRAM, 24 vCPUs, 256 GB of RAM, a 720 GB boot disk, and a 5 TB scratch disk, all at the locked $4.500 per hour. Worth a screenshot for your own records.

How to create a Spot GPU Droplet using doctl

The command that created the replacement Droplet in this tutorial, verbatim except for the key fingerprint:

doctl compute droplet create spot-trainer-02 \
  --region mem1 \
  --size gpu-mi355x1-288gb-spot \
  --image gpu-amd-base \
  --ssh-keys $SSH_KEY_FINGERPRINT \
  --user-data-file /tmp/user-data.yaml \
  --tag-names spot-tutorial \
  --wait

gpu-amd-base is the slug for the “AMD AI/ML Ready Image” (Ubuntu 24.04, ROCm 7.14 preinstalled, no PyTorch). For NVIDIA use gpu-h100x1-base or gpu-h100x8-base, the “NVIDIA AI/ML Ready” images; the driver stack is preinstalled and the same slugs work on B300. AMD also publishes framework images (amd-pytorchrocm7, amd-vllmrocm7, amddeveloperclou-rocm714software) if you would rather not install torch yourself. List them with doctl compute image list --public | grep -i "ai/ml\|rocm\|nvidia"; slugs change over time, so check before you hardcode one.

How to create a Spot GPU Droplet using the API

The same create call as any Droplet, with the spot size slug (API reference):

curl -X POST "https://api.digitalocean.com/v2/droplets" \
  -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "spot-trainer-01",
    "region": "mem1",
    "size": "gpu-mi355x1-288gb-spot",
    "image": "gpu-amd-base",
    "ssh_keys": ["'"$SSH_KEY_FINGERPRINT"'"],
    "tags": ["spot-tutorial"]
  }'

The response for my first Droplet is saved as evidence/droplet-create-response.json in the repo. The size slug is how you tell a spot Droplet apart, which is useful when you audit your fleet by tag or name.

Sanity check the hardware

On the AMD image, the first things I ran after SSH:

$ rocm-smi --showproductname
GPU[0]  : Card Series:    AMD Instinct MI355X VF
GPU[0]  : GFX Version:    gfx950
$ amd-smi version
AMDSMI Tool: 26.5.0 | ROCm version: 7.14.0 | amdgpu version: 6.19.14
$ lsblk -d -o NAME,SIZE,TYPE
vda   720G disk      # boot
vdc     5T disk      # scratch, unformatted
$ nproc; free -g | head -2
24
Mem: 251 total
$ python3 -c "import torch"
ModuleNotFoundError: No module named 'torch'

On the NVIDIA image the equivalent is nvidia-smi. Confirm the GPU model line matches what you ordered. Billing is per second with a five-minute minimum, and a powered-off Droplet still bills, so destroy Droplets you are done with rather than powering them off (pricing FAQ).

The gpu-amd-base image ships ROCm but not PyTorch. Match the wheel index to the ROCm version amd-smi reports:

python3 -m venv /root/venv
/root/venv/bin/pip install --index-url https://download.pytorch.org/whl/rocm7.14 torch
/root/venv/bin/pip install boto3
/root/venv/bin/python -c "import torch; print(torch.cuda.get_device_name(0), torch.version.hip)"
# AMD Instinct MI355X VF 7.14.60850

ROCm exposes the GPU through the torch.cuda API, so the training script below is vendor-neutral. You now have a GPU machine at a locked hourly rate. Next, we make the job running on it immune to losing that machine.

Here is the goal of this section: a training job that you can kill at any moment, on any machine, and restart on a fresh machine with one command, losing at most a few minutes of work and needing nobody to babysit it.

To get there we do three things, cheapest first:

1. Save progress to Spaces on a schedule. Every N steps, the script writes the model weights, the optimizer state, and the step counter to Spaces object storage, which lives outside the Droplet. This alone makes the job safe: however the Droplet goes away, the most you lose is the work since the last save.
2. Save once more on the way out. When the process gets a SIGTERM, from systemctl stop, a shutdown, or the reclaim, it writes one final checkpoint and exits cleanly. Five lines of code, and a graceful stop now loses zero steps.
3. Give yourself a button for the reclaim email. touch /tmp/checkpoint-now on the Droplet makes the script save immediately and keep training. When the two-hour notice lands, you press it and carry on using the GPU you are paying for.

Resume is the mirror image and it is automatic. On start, the script looks in Spaces for a small pointer file called LATEST. If it exists, it downloads the checkpoint it names and continues from that step. If not, it starts from zero. Because a replacement Droplet runs the identical command with the identical --run-id, it finds LATEST and picks up on its own.

A note on where checkpoints go. The spot regions (ATL1, RIC1, MKC1, MEM1) are new AI datacenters and do not have a Spaces region yet, so I used NYC3 from MEM1 and measured 9 to 12 seconds per 2.4 GB upload. Check Spaces availability and pick the nearest.

The diagram is the loop as one Droplet lives it. It starts by asking Spaces for LATEST and either resumes or begins fresh. It trains, and every N steps it uploads a checkpoint and only then rewrites LATEST to point at it, so the pointer can never name a half-uploaded file. An interrupt, whether a SIGTERM or the flag file, triggers an immediate save. When the Droplet is reclaimed, the arrow loops back to the start on a new machine. The four numbers along the bottom are what this cost in practice on the MI355X: 2,417 MB per checkpoint, about 10 seconds to upload, 13.7 seconds from systemctl stop to a clean exit, and zero steps lost across the whole drill.

Get the code

Everything below is in github.com/anishsingh20/spot-gpu-checkpoint-resume:

What the code tests. That a PyTorch job on a spot GPU can save its full state to Spaces on a schedule and what that costs in time. That a SIGTERM produces one last save and a clean exit. That a brand-new Droplet, given only the same command, finds the latest save and continues with nothing lost. And that LATEST is always safe to follow.

What it does not test. A reclaim triggered by DigitalOcean itself; there is no button for that, so the drill simulates the two halves separately, a graceful stop and a hard destroy-and-recreate, and the code path is the same either way. Model quality; the stand-in model trains on an unlearnable task on purpose, so the loss sits near 1.0 and the step counter is what matters. Multi-node training; this is one Droplet, one process. Swap in your own model and the rest of the file does not care.

Set up the bucket and key

Create a bucket in the console under Spaces Object Storage, or with any S3 client against the regional endpoint. Then create an access key scoped to that one bucket, because cloud-init user data is readable from inside the Droplet:

doctl spaces keys create spot-xx-xxxx-xx-xx-key \
  --grants "bucket=spot-xx-xxxx-xx-xx;permission=readwrite"

Put the four values in /root/.spaces.env as plain KEY=VALUE lines (no export, because systemd’s EnvironmentFile does not understand it; that mistake cost me one failed service start):

SPACES_KEY=DO00...
SPACES_SECRET=...
SPACES_REGION=nyc3
SPACES_BUCKET=spot-xx-xxxx-xx-xx

chmod 600 it. Rotate the key when the job is done.

The training script

One file, PyTorch and boto3. The model is a stand-in sized so one step is real GPU work (201M parameters, six MLP blocks at d=2048). Replace it with yours; nothing else in the file cares.

#!/usr/bin/env python3
"""
Checkpoint-and-resume training loop for DigitalOcean Spot GPU Droplets.

Three layers of protection, cheapest first:
  1. Periodic checkpoint every --every-steps to Spaces. This is what keeps
     the run safe no matter how or when the Droplet goes away.
  2. SIGTERM/SIGINT trap -> checkpoint now and exit cleanly. Costs five lines
     and makes `systemctl stop` (or any graceful shutdown) lose zero steps.
  3. Drain trigger for the two-hour reclaim email: touch /tmp/checkpoint-now
     (flush, keep going) or send SIGTERM (flush, exit). Same code path.

Resume is automatic. On start the script reads checkpoints/<run-id>/LATEST
from Spaces, restores model + optimizer + step, and continues. No checkpoint
means a fresh run. Run the identical command on the replacement Droplet.

Env (put these in /root/.spaces.env, chmod 600, `source` it before running):
  SPACES_KEY, SPACES_SECRET   Spaces access key pair
  SPACES_REGION               Spaces region, e.g. nyc3
  SPACES_BUCKET               bucket name

Run:
  python3 spot_train.py --run-id sweep-01 --total-steps 2000 --every-steps 400

Tested 2026-09-09 on gpu-mi355x1-288gb-spot (MEM1), ROCm 7.14, torch rocm7.14.
"""
import argparse
import io
import os
import signal
import sys
import time

import boto3
import torch
import torch.nn as nn

FLUSH_FLAG = "/tmp/checkpoint-now"
_exit_requested = False


def _on_signal(signum, frame):
    global _exit_requested
    _exit_requested = True
    print(f"[signal] {signal.Signals(signum).name} received, will checkpoint and exit",
          flush=True)


def spaces_client():
    region = os.environ["SPACES_REGION"]
    return boto3.client(
        "s3",
        region_name=region,
        endpoint_url=f"https://{region}.digitaloceanspaces.com",
        aws_access_key_id=os.environ["SPACES_KEY"],
        aws_secret_access_key=os.environ["SPACES_SECRET"],
    )


def save_checkpoint(s3, bucket, run_id, step, model, optimizer, keep):
    t0 = time.time()
    buf = io.BytesIO()
    torch.save({"step": step,
                "model_state": model.state_dict(),
                "optimizer_state": optimizer.state_dict()}, buf)
    size_mb = buf.tell() / 1e6
    buf.seek(0)
    key = f"checkpoints/{run_id}/step-{step:09d}.pt"
    s3.upload_fileobj(buf, bucket, key)
    # Write the pointer only after the object is fully uploaded, so LATEST
    # never names a half-written checkpoint.
    s3.put_object(Bucket=bucket, Key=f"checkpoints/{run_id}/LATEST", Body=key.encode())
    dt = time.time() - t0
    print(f"[checkpoint] step {step} -> s3://{bucket}/{key} "
          f"({size_mb:,.0f} MB in {dt:.1f}s)", flush=True)
    prune_old(s3, bucket, run_id, keep)
    return dt


def prune_old(s3, bucket, run_id, keep):
    prefix = f"checkpoints/{run_id}/step-"
    resp = s3.list_objects_v2(Bucket=bucket, Prefix=prefix)
    keys = sorted(o["Key"] for o in resp.get("Contents", []))
    for old in keys[:-keep]:
        s3.delete_object(Bucket=bucket, Key=old)
        print(f"[prune] deleted {old}", flush=True)


def load_latest(s3, bucket, run_id, model, optimizer):
    try:
        latest = s3.get_object(Bucket=bucket, Key=f"checkpoints/{run_id}/LATEST")
    except s3.exceptions.NoSuchKey:
        print(f"[resume] no checkpoint for run-id={run_id}, starting fresh", flush=True)
        return 0
    key = latest["Body"].read().decode()
    t0 = time.time()
    buf = io.BytesIO()
    s3.download_fileobj(bucket, key, buf)
    buf.seek(0)
    ckpt = torch.load(buf, map_location="cpu", weights_only=False)
    model.load_state_dict(ckpt["model_state"])
    optimizer.load_state_dict(ckpt["optimizer_state"])
    print(f"[resume] restored {key} in {time.time() - t0:.1f}s, "
          f"continuing from step {ckpt['step']}", flush=True)
    return ckpt["step"]


class StandInModel(nn.Module):
    """A stack of MLP blocks sized so one step is real GPU work (~200M params).
    Replace with your model; nothing else in this file cares what it is."""

    def __init__(self, d_model=2048, d_hidden=8192, layers=6):
        super().__init__()
        self.blocks = nn.ModuleList([
            nn.Sequential(nn.LayerNorm(d_model), nn.Linear(d_model, d_hidden),
                          nn.GELU(), nn.Linear(d_hidden, d_model))
            for _ in range(layers)
        ])

    def forward(self, x):
        for blk in self.blocks:
            x = x + blk(x)
        return x


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--run-id", required=True)
    ap.add_argument("--total-steps", type=int, default=2000)
    ap.add_argument("--every-steps", type=int, default=400)
    ap.add_argument("--keep", type=int, default=3, help="checkpoints to retain in Spaces")
    ap.add_argument("--batch", type=int, default=64)
    ap.add_argument("--seq", type=int, default=512)
    args = ap.parse_args()

    signal.signal(signal.SIGTERM, _on_signal)
    signal.signal(signal.SIGINT, _on_signal)

    if not torch.cuda.is_available():
        sys.exit("no GPU visible to torch (on AMD: check rocm-smi and the rocm wheel index)")
    device = "cuda"  # ROCm exposes the GPU through the same torch.cuda API
    print(f"[gpu] {torch.cuda.get_device_name(0)}, torch {torch.__version__}, "
          f"hip {torch.version.hip}", flush=True)

    torch.manual_seed(0)
    model = StandInModel().to(device)
    n_params = sum(p.numel() for p in model.parameters())
    print(f"[model] {n_params / 1e6:,.0f}M params", flush=True)
    optimizer = torch.optim.AdamW(model.parameters(), lr=1e-4)
    loss_fn = nn.MSELoss()

    s3 = spaces_client()
    bucket = os.environ["SPACES_BUCKET"]
    step = load_latest(s3, bucket, args.run_id, model, optimizer)
    start_step, t_start, ckpt_time = step, time.time(), 0.0

    while step < args.total_steps:
        x = torch.randn(args.batch, args.seq, 2048, device=device)
        target = torch.roll(x, 1, dims=1)
        loss = loss_fn(model(x), target)
        optimizer.zero_grad(set_to_none=True)
        loss.backward()
        optimizer.step()
        step += 1

        if step % 50 == 0:
            torch.cuda.synchronize()
            rate = (step - start_step) / (time.time() - t_start)
            print(f"[train] step {step}/{args.total_steps} loss {loss.item():.4f} "
                  f"{rate:.2f} steps/s", flush=True)

        flag = os.path.exists(FLUSH_FLAG)
        if (_exit_requested or flag or step % args.every_steps == 0
                or step == args.total_steps):
            torch.cuda.synchronize()
            ckpt_time += save_checkpoint(s3, bucket, args.run_id, step, model,
                                         optimizer, args.keep)
            if flag:
                os.remove(FLUSH_FLAG)
            if _exit_requested:
                break

    wall = time.time() - t_start
    done = step - start_step
    print(f"[summary] ran steps {start_step}->{step} in {wall:.0f}s wall, "
          f"{ckpt_time:.0f}s of that in checkpoints "
          f"({100 * ckpt_time / max(wall, 1e-9):.1f}% tax), "
          f"{done / max(wall - ckpt_time, 1e-9):.2f} steps/s while training",
          flush=True)
    if _exit_requested:
        print("[exit] checkpointed and exiting cleanly; rerun the same command to resume",
              flush=True)
        sys.exit(0)
    print(f"[done] {args.total_steps} steps complete", flush=True)


if __name__ == "__main__":
    main()

Four details that matter more than they look:

• Upload first, then move the pointer. If the Droplet dies mid-upload, LATEST still names the previous complete checkpoint. A resume never loads a truncated file.
• Save the optimizer. AdamW keeps two moment tensors per parameter, so the checkpoint is roughly three times the weights (201M params, 2,417 MB). Skipping it makes resume silently worse, not faster.
• Prune. --keep 3 deletes older checkpoints after each upload. Without it a 10-hour run at this cadence would leave 200 checkpoints and 480 GB in the bucket.
• Fail loudly without a GPU. The first draft of this script fell back to CPU. On a $4.50 per hour machine that is an expensive way to find out the wheel index was wrong.

Run it as a service

A systemd unit gives you SIGTERM forwarding, a stop timeout long enough for the final upload, a log file that survives your SSH session, and a start-on-boot hook for cloud-init. This is spot-train.service:

[Unit]
Description=Checkpoint-and-resume training on a Spot GPU Droplet
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
WorkingDirectory=/root
EnvironmentFile=/root/.spaces.env
# Same --run-id on every Droplet that runs this job; that is what makes resume work.
ExecStart=/root/venv/bin/python /root/spot_train.py --run-id sweep-01 --total-steps 3000 --every-steps 600
# On shutdown or `systemctl stop`, systemd sends SIGTERM and waits this long
# for the final checkpoint to upload before it escalates to SIGKILL.
KillSignal=SIGTERM
TimeoutStopSec=600
Restart=no
StandardOutput=append:/root/train.log
StandardError=append:/root/train.log

[Install]
WantedBy=multi-user.target

cp spot-train.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now spot-train
tail -f /root/train.log

What the first Droplet logged, trimmed (full file: evidence/02-run1-train-drill.log):

[gpu] AMD Instinct MI355X VF, torch 2.14.0+rocm7.14, hip 7.14.60850
[model] 201M params
[resume] no checkpoint for run-id=sweep-01, starting fresh
[train] step 50/3000 loss 1.1003 3.05 steps/s
[train] step 600/3000 loss 1.0026 3.57 steps/s
[checkpoint] step 600 -> s3://spot-ckpt-anish-0909/checkpoints/sweep-01/step-000000600.pt (2,417 MB in 10.2s)
[train] step 1200/3000 loss 1.0026 3.49 steps/s
[checkpoint] step 1200 -> s3://spot-ckpt-anish-0909/checkpoints/sweep-01/step-000001200.pt (2,417 MB in 10.4s)
[train] step 1800/3000 loss 1.0023 3.46 steps/s
[checkpoint] step 1800 -> s3://spot-ckpt-anish-0909/checkpoints/sweep-01/step-000001800.pt (2,417 MB in 12.4s)
[train] step 1850/3000 loss 1.0010 3.38 steps/s

You cannot ask DigitalOcean to reclaim a Droplet on cue, so I simulated the two halves separately: a graceful interrupt (what the email gives you time to do, or what a shutdown does) and a hard loss (destroy the Droplet with no warning, then bring up a replacement). Together they cover everything a real reclaim can do to you.

Half one: the graceful flush

With training at step 1850, systemctl stop spot-train from a second SSH session:

[train] step 1850/3000 loss 1.0010 3.38 steps/s
[signal] SIGTERM received, will checkpoint and exit
[checkpoint] step 1892 -> s3://spot-ckpt-anish-0909/checkpoints/sweep-01/step-000001892.pt (2,417 MB in 9.3s)
[prune] deleted checkpoints/sweep-01/step-000000600.pt
[summary] ran steps 0->1892 in 568s wall, 42s of that in checkpoints (7.4% tax), 3.60 steps/s while training
[exit] checkpointed and exiting cleanly; rerun the same command to resume

time systemctl stop reported 13.7 seconds: one in-flight step, a 9.3 second upload, and the pointer write. Spaces afterwards, from evidence/03-spaces-listing-after-drill.txt:

 2,417.0 MB  06:11:42Z  checkpoints/sweep-01/step-000001200.pt
 2,417.0 MB  06:14:41Z  checkpoints/sweep-01/step-000001800.pt
 2,417.0 MB  06:15:15Z  checkpoints/sweep-01/step-000001892.pt
LATEST -> checkpoints/sweep-01/step-000001892.pt

The same flush without exiting is touch /tmp/checkpoint-now; the script uploads at the next step boundary, deletes the flag, and keeps training. That is the command to run when the reclaim email arrives and you would rather keep using the remaining two hours of GPU.

Half two: destroy and replace

For the hard half I destroyed the Droplet outright, which is what a reclaim does to your filesystem, and created a replacement whose cloud-init installs the stack and starts the same unit. Because the unit carries the same --run-id, the new Droplet finds LATEST and resumes on its own.

The cloud-init template is cloud-init-resume.yaml. It writes /root/.spaces.env, /root/spot_train.py, and the unit file, then installs torch and enables the service. Secrets never go in the repo: make_user_data.py fills the template from a local env file and inlines the script.

python3 make_user_data.py --env ~/.spaces.env > /tmp/user-data.yaml
doctl compute droplet delete spot-trainer-01 --force        # the "reclaim"
doctl compute droplet create spot-trainer-02 --region mem1 \
  --size gpu-mi355x1-288gb-spot --image gpu-amd-base \
  --ssh-keys $SSH_KEY_FINGERPRINT --user-data-file /tmp/user-data.yaml \
  --tag-names spot-tutorial --wait

Timeline, from evidence/04-reclaim-and-recreate.log, journalctl, and evidence/05-run2-resume-complete.log:

Four of those five minutes are pip install torch pulling the ROCm 7.14 wheel. Bake a snapshot of a Droplet with the venv installed and create replacements from the snapshot, and cloud-init only has to write three files and start the unit; the gap drops to about a minute.

What the replacement logged:

cloud-init finished 2026-09-09T06:23:31Z
[gpu] AMD Instinct MI355X VF, torch 2.14.0+rocm7.14, hip 7.14.60850
[model] 201M params
[resume] restored checkpoints/sweep-01/step-000001892.pt in 13.2s, continuing from step 1892
[train] step 1900/3000 loss 1.0009 1.30 steps/s
[train] step 2400/3000 loss 1.0012 3.53 steps/s
[checkpoint] step 2400 -> s3://spot-ckpt-anish-0909/checkpoints/sweep-01/step-000002400.pt (2,417 MB in 10.1s)
[train] step 3000/3000 loss 1.0008 3.47 steps/s
[checkpoint] step 3000 -> s3://spot-ckpt-anish-0909/checkpoints/sweep-01/step-000003000.pt (2,417 MB in 10.3s)
[summary] ran steps 1892->3000 in 330s wall, 20s of that in checkpoints (6.2% tax), 3.58 steps/s while training
[done] 3000 steps complete

Steps 0 to 1892 ran on one Droplet, 1892 to 3000 on another, and the job did not notice. The loss values sit near 1.0 because the stand-in task is unlearnable by design; the step counter and the timings are the result.

Two minutes after the drill, the GPU Droplets list has one row, spot-trainer-02, with a new IP. The first Droplet is gone. This is what your fleet looks like after a reclaim plus replacement: a different name and address, the same shape, and the thread that connects them lives in Spaces.

Here is that thread. The bucket holds checkpoints/sweep-01/ with three 2.25 GiB checkpoints for steps 1200, 1800, and 1892, plus LATEST, a 38-byte file whose whole content is the name of the newest one. Step 600 was already pruned. When the replacement booted, this folder was all it needed.

The replacement one minute after going active: same region, same image, same locked rate. The graphs are empty because cloud-init is still installing torch; four minutes later the GPU was busy at step 1900.

Both Droplets were destroyed when the run finished. Total GPU time for the whole tutorial, including a smoke test and the drill: about 33 minutes, roughly $2.50 at the locked $4.50 rate. The bucket sits inside the Spaces base plan.

The overhead of the loop is the checkpoint time as a share of wall clock, and it depends on three things you control: checkpoint size, upload bandwidth to Spaces, and the interval N. From the run: 2,417 MB takes about 10 seconds (9.3 to 12.4 across six uploads), and training runs at about 3.5 steps per second.

The rule: pick the largest N whose “work at risk” you can shrug at. For a 10-hour job, losing 6 minutes is nothing and 3 percent is the tax; every 1,200 steps is the right answer for this model. For a 40-minute job, a tighter cadence is cheap insurance. Larger models move the whole table: a 7B-parameter model with AdamW is roughly 84 GB per checkpoint, about six minutes at this upload speed, and you would checkpoint hourly and consider sharding the upload.

One reclaim-and-resume cycle costs the re-provision gap (5 minutes 24 seconds here, about one minute with a baked snapshot), one restore (13 seconds), and at most one interval of recompute. Put that against the price: on an MI350X at $4.00 spot versus $4.76 reserved, one reclaim a day costs about 6 minutes of a 24-hour day, or 0.4 percent, against a 16 percent lower hourly rate and no 12-month term. For B300, the case for spot is access and flexibility rather than the hourly rate; you get the GPU this week, by the hour, with no commitment, and the loop makes sure that flexibility never costs you a run.

For batch inference the checkpoint is not model state, it is a done-list. Track which items are complete in a manifest object in Spaces and skip them on resume:

def load_done(s3, bucket, run_id):
    try:
        obj = s3.get_object(Bucket=bucket, Key=f"batch/{run_id}/done.txt")
        return set(obj["Body"].read().decode().split())
    except s3.exceptions.NoSuchKey:
        return set()

def mark_done(s3, bucket, run_id, done_ids):
    s3.put_object(Bucket=bucket, Key=f"batch/{run_id}/done.txt",
                  Body="\n".join(sorted(done_ids)).encode())

Process items in a loop, add each finished ID to the set, and rewrite the manifest every K items with the same flush triggers as the training loop. A reclaim costs at most K items of rework. Write outputs to Spaces as you go, keyed by item ID, so the manifest and the outputs cannot disagree.

If your batch job targets catalog models rather than your own weights, you can skip the GPU entirely and use Batch Inference on the Inference Engine, which owns the job lifecycle for you. The spot pattern is for models you serve yourself.

The same preview covers Spot GPU node pools on DOKS, and the mechanics differ enough to note. Nodes are labelled and tainted as spot capacity so pods must opt in. Reclamation is per node pool, not per node; DigitalOcean cordons and drains the pool, honours PodDisruptionBudgets up to a drain timeout, and emits Kubernetes Events and node conditions when it starts. Spot pools can be shrunk but not grown after creation; new capacity means a new pool at the then-current rate. The cluster still needs a CPU pool. Details in terms section 3. The checkpoint loop above runs unchanged as a Job with a toleration for the spot taint, and the Kubernetes signals give you a programmatic hook to trigger the flush.

Three questions decide it. Can the job resume from saved state without a person watching? Is losing the steps since the last checkpoint, plus about five minutes to replace the Droplet, acceptable? Do you want this GPU this week, by the hour, without a sales call or a 12-month term? Three yeses and Spot GPU Droplets is the right call, and the good-fit list is long: training with checkpoints, hyperparameter sweeps where each trial is its own run-id, batch inference and evals, data preprocessing, and testing kernels on B300 or MI355X silicon. Any “no” points you to on-demand or reserved capacity, and that list is also clear: anything serving live traffic, anything a person is typing into, tightly coupled multi-node training that cannot restart elastically, and long-running B300 workloads where the 12-month reserved rate is the better deal. The short version: if the job can save its progress, Spot GPU Droplets will save you money, time, or both.

1. Is there an on-demand price for MI355X or B300 I can compare spot against?

Not today. On September 9, 2026, the on-demand list stops at H200 ($4.47) and MI325X ($3.80). B300 and MI350X have 12-month reserved rates ($7.94 and $4.76); MI355X is available on spot only. Compare spot to reserved, or to what it is worth to have the GPU this week without a term.

2. Will I get a SIGTERM when the Droplet is reclaimed?

The documented notice is the two-hour email, and for DOKS pools, Kubernetes Events and node conditions. The loop handles SIGTERM so a graceful stop loses nothing, and it does not depend on one: the periodic checkpoints keep the job safe on their own.

3. Does the spot rate on my running Droplet change when the posted rate changes?

No. The rate is locked at creation for the life of that Droplet (terms 2.6, and the console shows it on the Droplet page). A replacement Droplet locks whatever the rate is when it is created.

4. How big should the checkpoint interval be?

Measure one checkpoint upload, divide by the share of wall clock you are willing to spend (3 percent is a reasonable default), and that is the training time per interval. For this run, 10 seconds at 3 percent gives 330 seconds, or about 1,200 steps at 3.5 steps per second. Then check that losing one interval would not ruin your day.

5. Why not save checkpoints to the 5 TB scratch disk? It is faster

Because the scratch disk is local to the Droplet and goes with it on reclaim (terms section 2.5). Use scratch for the dataset and for staging a checkpoint before upload if you want, and keep the durable copy in Spaces.

6. What about Volumes Block Storage instead of Spaces?

A Volume survives its Droplet and can be re-attached to the replacement in the same region, which avoids the upload entirely. Check that Volumes are offered in your spot region first; the new AI datacenters (MEM1, RIC1, MKC1) are still adding products. Spaces works from any region, which is why this tutorial uses it.

7. Is there an SLA or support for Spot GPU Droplets?

Spot GPU Droplets are in Public Preview and supported on a reasonable-effort basis during business hours (terms section 2.7). They are built for workloads that save progress and restart, which is exactly what this loop gives you. For SLA-backed capacity, use on-demand GPU Droplets.

8. Can I run multi-node training on spot?

Yes, if the framework can restart elastically. Reclamation of one Droplet in a tightly coupled job stalls the others, so for most teams the simplest path is one 8-GPU spot Droplet rather than eight 1-GPU ones. On DOKS, the whole pool is reclaimed together, which keeps things consistent. You can read more on Resilient GPU Compute on DigitalOcean Kubernetes: Surviving Spot Interruptions

For a long time the newest GPUs came with a choice: sign a term or wait. Spot GPU Droplets remove that choice for the work that makes up most of an AI team’s week. You can have a B300 or an MI355X in about a minute, by the hour, at a rate that is locked the moment you click Create, and hand it back when you are done. The sweep you wanted to run this afternoon, the eval on your customer’s exact silicon, the kernel you want to test on gfx950 before you commit: all of that is now one API call away.

The only thing spot asks of you is that your job can save its progress, and this tutorial shows how small that ask really is. One script uploads a checkpoint every N steps and moves a pointer. Five lines catch SIGTERM for a free final save. One flag file turns the two-hour reclaim email into a button you press and then get back to work. And one cloud-init template means the replacement Droplet resumes on its own, with the same command and the same run-id, before you have finished reading the email. On the MI355X that loop cost 5.5 percent of wall clock, flushed and exited in 13.7 seconds, and came back from a destroyed Droplet in 5 minutes 24 seconds with zero steps lost. The whole drill, two Droplets and a complete 3,000-step run, cost about $2.50.

That is the trade Spot GPU Droplets offer: an afternoon of engineering once, and latest-generation GPUs on your terms from then on. The code is ready to clone at github.com/anishsingh20/spot-gpu-checkpoint-resume. Swap in your model, point it at a bucket, and create your first Spot GPU Droplet. Then read the comparison doc and the Public Preview terms, and check the availability matrix for your GPU’s regions. When the model is trained and it is time to serve it, on-demand GPU Droplets and the Inference Engine pick up the production side.

• Spot GPU Droplets vs On-Demand GPU Droplets, DigitalOcean Docs (last verified by DO August 11, 2026; read September 9, 2026)
• GPU Droplet Pricing, Spot Plans (prices captured September 9, 2026; they change daily)
• GPU Droplet Availability by Region
• DigitalOcean Public Preview Terms: Spot GPU Droplets and Node Pools (last updated August 10, 2026)
• Now Available: Spot GPU Droplets in Public Preview, Changelog
• Droplets API: create and Sizes API
• Droplet metadata service and providing user data (cloud-init)
• Spaces Object Storage, Spaces availability, Spaces access keys
• Batch Inference how-to
• AWS EC2 Spot Instance interruption notices
• GCP Spot VMs documentation
• Code and evidence for this tutorial: github.com/anishsingh20/spot-gpu-checkpoint-resume