You finished a userspace package update. Libraries changed. Daemons need a clean boot graph. The box is “up,” but half the stack is still running the old generation. The reflex is reboot. That means firmware, bootloader, kernel bring-up, initrd, and every cold-start cost that comes with them—even when the kernel itself did not change. systemd has a narrower tool for that case: soft-reboot. It tears down and restarts userspace while the kernel keeps running. Downtime shrinks from “full machine cycle” to “service manager reexec + new boot transaction.” This is a practical operator guide: when soft-reboot is the right hammer, how /run/nextroot/ root switches work, what can survive the cut, verification, and the hard boundaries where you still need a real reboot or kexec. What soft-reboot actually is From systemd-soft-reboot.service(8) and systemctl(1): Trigger with systemctl soft-reboot (do not start systemd-soft-reboot.service by hand). That isolates toward soft-reboot.target, which requires systemd-soft-reboot.service. Near the end, leftover processes get SIGTERM, then SIGKILL (no wait for polite exit on the final kill wave). If /run/nextroot/ exists (directory, mount, or symlink to either), the root filesystem is switched to it. PID 1 reexecutes the service manager from the (possibly new) root and enqueues a new boot transaction, similar to a normal reboot’s userspace phase. It deliberately does not go through: the second phase of regular shutdown (systemd-shutdown) return to the initrd context hardware reboot firmware initialization bootloader initialization kernel initialization initrd initialization systemctl soft-reboot was added in systemd 254. On this host’s generation it ships as part of modern systemd (Debian 13 ships 257.x; upstream man pages track 261). # Presence / version systemctl --version | head -1 systemctl cat soft-reboot.target systemctl cat systemd-soft-reboot.service man 8 systemd-soft-reboot.service On a stock install you should see something like: soft-reboot.target: Requires=systemd-soft-reboot.service, AllowIsolate=yes, JobTimeoutSec=30min, JobTimeoutAction=soft-reboot-force systemd-soft-reboot.service: DefaultDependencies=no, ordered after shutdown.target / umount.target / final.target, SuccessAction=soft-reboot-force That last line is the fail-closed escape hatch: if the soft path cannot complete cleanly within policy, systemd escalates to a forced soft-reboot action rather than hanging forever in a half-shut state. Mental model: three reboot depths Depth Command Kernel Firmware / bootloader Typical use Userspace only systemctl soft-reboot stays skipped userspace refresh, A/B root flip via /run/nextroot/ New kernel, skip firmware systemctl kexec (after load) replaced mostly skipped kernel update without full POST Full cycle systemctl reboot replaced full path firmware/kernel/initrd/device topology changes Soft-reboot is not “reboot but faster magic.” It is a different contract: kernel state is continuous; userspace state is not (unless you deliberately pin survivors). Prerequisites systemd ≥ 254 with soft-reboot.target present. Root (or polkit rights equivalent to other power commands). A reason that is userspace-shaped: package refresh, config generation that needs a full dependency graph, image-based root handoff. Acceptance that sysctl /sys kernel knobs are not reset and the running kernel is unchanged. ArchWiki’s systemd page adds two operator notes that match the man page: unlocked dm-crypt devices can remain attached across soft-reboot (no full teardown through the usual shutdown path) do not soft-reboot as your only action after updates that changed kernel + initramfs—those need kexec or a full reboot Lab 0: inspect before you pull the lever On a spare VM or lab host, record a baseline. Soft-reboot will disconnect your SSH session the same way a reboot does for interactive work—plan console or out-of-band access. # Boot identity uname -r cat /proc/sys/kernel/random/boot_id cat /proc/sys/kernel/osrelease # Userspace manager systemctl show -p Version -p UserspaceTimestamp -p FirmwareTimestamp -p LoaderTimestamp -p KernelTimestamp # What soft-reboot will isolate toward systemctl list-dependencies soft-reboot.target systemctl show soft-reboot.target -p Requires -p After -p JobTimeoutUSec -p JobTimeoutAction Useful expectations: boot_id: changes on full reboot / kexec paths that start a new boot; treat soft-reboot as a userspace cycle—verify with timestamps and service start times rather than assuming every “boot counter” UI agrees. FirmwareTimestamp / LoaderTimestamp / KernelTimestamp: should stay continuous across soft-reboot; UserspaceTimestamp advances when PID 1 starts a new userspace boot transaction. Kernel release from uname -r must be unchanged after soft-reboot (if it changed, you did not soft-reboot). # After soft-reboot, compare: systemctl show -p UserspaceTimestamp -p KernelTimestamp -p FirmwareTimestamp journalctl -b -o short-iso | head systemctl --failed The default operation: plain soft-reboot For “restart all of userspace on the same root”: # Optional wall / schedule knobs work like other power commands: # systemctl soft-reboot --when=2026-09-07 23:30:00 # systemctl soft-reboot --when=cancel sudo systemctl soft-reboot From systemctl(1): equivalent to starting soft-reboot.target with --job-mode=replace-irreversibly --no-block asynchronous: the command returns after enqueue honors --force and --when= similarly to halt Never do: # Wrong: do not start the implementation unit directly # sudo systemctl start systemd-soft-reboot.service The man page is explicit: trigger via systemctl soft-reboot (or the target isolation path it implements), not by hand-starting the helper service. What still happens / what does not Still happens (userspace shutdown graph toward soft-reboot): normal units stop via the usual stop jobs as the transaction isolates mounts that are part of the shutdown graph can unmount journal and unit state for the new userspace cycle start fresh from PID 1 reexec Does not happen: systemd-shutdown second phase executables under /usr/lib/systemd/system-shutdown/ (skipped because systemd-shutdown is not run) firmware / bootloader / kernel / initrd automatic reset of /proc/sys and /sys policy If you rely on shutdown hooks in /usr/lib/systemd/system-shutdown/ for disk flush gymnastics or LED scripts, soft-reboot will not run them. Put critical work in normal service ExecStop= / ExecStopPost= instead. Lab 1: switch root with /run/nextroot/ This is the power feature for image-based and A/B setups: prepare a complete root filesystem hierarchy at /run/nextroot/, then soft-reboot into it without losing kernel-held state (routes, some device setup, non-reset sysctls, unlocked LUKS mappings as applicable). /run/nextroot/ may be: a plain directory on the existing /run tmpfs a mount point (bind mount, loop, image) a symlink to either systemd will turn a non-mount /run/nextroot/ into a mount point automatically on soft-reboot when needed. Minimal directory-based next root (teaching lab) This is intentionally small and not a full distro root. Use it only to prove the switch mechanism on a disposable VM. # WARNING: disposable lab only. A real next root needs a bootable userspace: # /usr, /etc, /var skeleton, device nodes or devtmpfs policy, and a systemd binary. sudo mkdir -p /run/nextroot # Example: bind an already-prepared root tree (snapshot, unpacked image, ostree checkout) # sudo mount --bind /var/lib/images/root-b /run/nextroot # Or, for image-based hosts, mount the inactive slot: # sudo mount /dev/disk/by-partlabel/root-b /run/nextroot find /run/nextroot -maxdepth 2 -type d | head test -x /run/nextroot/usr/lib/systemd/systemd \ -o -x /run/nextroot/lib/systemd/systemd \ && echo "systemd binary visible in next root" Then: sudo systemctl soft-reboot After reconnecting: findmnt -no SOURCE,FSTYPE,OPTIONS / # Confirm you landed on the intended slot/device lsblk -o NAME,LABEL,PARTLABEL,MOUNTPOINTS Important interaction with plain reboot From both systemctl reboot and systemctl soft-reboot docs: if /run/nextroot/ is set up, systemctl reboot performs a soft-reboot instead, unless you set: export SYSTEMCTL_SKIP_AUTO_SOFT_REBOOT=1 sudo systemctl reboot That is easy to miss in automation. Image updaters that stage /run/nextroot/ and then call reboot are often depending on this auto-soft-reboot behavior. Operators who stage a next root for testing and then expect a full reboot can be surprised—set SYSTEMCTL_SKIP_AUTO_SOFT_REBOOT=1 when you truly need firmware/kernel bring-up. Resource pass-through (use sparingly) Soft-reboot can carry selected runtime resources into the next userspace cycle. The man page lists the official mechanisms—and warns to use them sparingly, because mixing old and new generations is how you get “half updated” systems. 1) /run stays mounted /run is shared across the soft-reboot boundary. It is the natural place for: next-root staging (/run/nextroot/) one-shot flags for the next boot transaction ephemeral coordinator state that should not hit disk Do not treat /run as durable storage; treat it as kernel-lifetime memory backed state that happens to survive userspace recycle. 2) File descriptor store Services can stash file descriptors with the manager (FileDescriptorStoreMax= / fdstore protocol). Across soft-reboot: FDs in the store of units that remain active until the end can be handed to the same unit after the next cycle alternatively, set FileDescriptorStorePreserve=: Value Behavior (from systemd.service(5)) no release store when the service stops restart (default) keep while unit is not inactive/failed, or a job is queued, or restart is expected yes keep until the unit is removed from memory—useful to pin until the manager exits FileDescriptorStorePreserve= was added in version 254 (same generation as soft-reboot). # Inspect a unit's fdstore (when in use) systemd-analyze fdstore some.service sudo systemctl clean --what=fdstore some.service Units that must keep the store without staying “up” the entire shutdown usually combine preserve mode with careful dependency design—or stay alive with the survivor pattern below. 3) .socket units that never stop Socket FDs remain open and connectible if the .socket unit is not stopped during the transition. Achieved with DefaultDependencies=no and by avoiding conflicts that pull the socket into the stop set. Clients can keep connecting while userspace recycles—powerful for load balancers and local brokers, easy to get wrong if activation policy disagrees after the new boot transaction. 4) Surviving service processes A process may keep running across soft-reboot only if all of the following hold (paraphrased from the man page example): # /etc/systemd/system/my-survivor.service [Unit] Description=My Surviving Service SurviveFinalKillSignal=yes IgnoreOnIsolate=yes DefaultDependencies=no After=basic.target Conflicts=reboot.target kexec.target poweroff.target halt.target rescue.target emergency.target Before=shutdown.target rescue.target emergency.target [Service] Type=oneshot ExecStart=/usr/bin/sleep infinity # Real services: your long-running binary; oneshot+sleep is the man-page sketch only RemainAfterExit=no For templated units (foo@instance.service), also ship a slice that survives, because instances default into a template slice that would otherwise be stopped: # /etc/systemd/system/system-foo.slice [Unit] SurviveFinalKillSignal=yes IgnoreOnIsolate=yes DefaultDependencies=no Official guidance is blunt: prefer not to survive. Survivors: pin old filesystem mounts and libraries in memory skip code updates for that process require D-Bus reconnect logic if they talk to the system bus (the broker does restart; connections drop) If you need isolation from the host OS tree, the man page points at Portable Services—and warns not to BindPaths= host OS files, or the old tree stays pinned for the life of the unit. 5) Mounts and complex storage Mounts can remain mounted if configured with DefaultDependencies=no and without Conflicts=umount.target. That is how complex storage can stay attached while userspace turns over—aligned with the “unlocked LUKS stays” operational reality. Homelab pattern: userspace refresh after apt without a full reboot A common safe pattern on servers where the kernel did not change: # 1) Update packages sudo apt-get update sudo apt-get upgrade # 2) If a new kernel/initrd landed, do NOT soft-reboot as the only step if [ -n "$(find /boot -name 'vmlinuz-*' -newer /proc/1/root 2>/dev/null | head -1)" ]; then echo "Kernel payloads changed under /boot — schedule kexec or full reboot" fi # 3) Otherwise recycle userspace cleanly # sudo systemctl soft-reboot Pair with: needrestart or package hooks to see which daemons still need attention a scheduled full reboot window for kernel CVEs (livepatch if you use it) monitoring that distinguishes “userspace boot age” from “kernel boot age” Verification checklist # Before KTS=$(systemctl show -p KernelTimestamp --value) UTS=$(systemctl show -p UserspaceTimestamp --value) echo "kernel=$KTS userspace=$UTS" uname -r | tee /run/soft-reboot-pre-uname # Perform soft-reboot, then after login: # Kernel identity unchanged diff -u <(cat /run/soft-reboot-pre-uname) <(uname -r) # Userspace transaction should look like a new boot systemctl show -p UserspaceTimestamp -p KernelTimestamp -p FirmwareTimestamp systemctl is-system-running systemctl --failed # Soft-reboot path evidence in the journal (wording varies by version) journalctl -b -u systemd-soft-reboot.service -u soft-reboot.target --no-pager | tail -50 journalctl -b | head -40 Success signals: uname -r unchanged firmware/loader/kernel timestamps continuous (where exposed) userspace services show fresh start times failed units list is empty (or only known pre-existing failures) if you used /run/nextroot/, findmnt / points at the new backing store Failure / rollback signals: stuck isolating toward soft-reboot.target past the job timeout (watch for soft-reboot-force) services pinned to old trees via accidental survivors / bind mounts expectation of reset sysctls that never reset—re-apply /etc/sysctl.d/ explicitly if needed: sudo sysctl --system Boundaries (what not to expect) Need Use instead New kernel / modules / initramfs systemctl kexec or full systemctl reboot Firmware / UEFI variable / boot-entry change full reboot (--boot-loader-entry=, --firmware-setup, etc.) Reset all sysctls and kernel object state full reboot (or explicit sysctl + module reload discipline) Run /usr/lib/systemd/system-shutdown/ helpers full shutdown path; or move logic into unit stop jobs Crash dump / kdump path testing real panic/kdump flow (separate stack) Cold device re-probe that depends on boot firmware full reboot “Survive everything, zero disruption” not soft-reboot; use rolling services, FD store carefully, or cluster failover Related tools that are not substitutes: systemctl daemon-reexec: reexec PID 1 only, without a full userspace boot transaction systemctl isolate rescue.target: maintenance mode, not an update cycle kexec: replaces the kernel; still skips much of firmware, but it is not soft-reboot nspawn / containers: recycle a machine without touching the host kernel—different layer Suggested roll-out Confirm systemd ≥ 254 and read systemctl cat soft-reboot.target. Try plain systemctl soft-reboot on a disposable VM; compare kernel vs userspace timestamps. Document whether your fleet stages /run/nextroot/ and whether reboot auto-promotes to soft-reboot. Keep survivors and FD-store pass-through off by default; add them per service with an explicit threat/availability model. After package updates: soft-reboot for userspace-only changes; kexec/full reboot when /boot kernel artifacts change. Re-apply sysctl policy if you depend on clean defaults after the cycle. Monitor systemctl --failed and boot-age metrics that understand two clocks (kernel uptime vs userspace boot). Quick reference # Userspace-only reboot (same root) sudo systemctl soft-reboot # Stage a new root, then soft-reboot into it sudo mount /dev/disk/by-partlabel/root-b /run/nextroot sudo systemctl soft-reboot # Force a *full* reboot even if /run/nextroot exists sudo SYSTEMCTL_SKIP_AUTO_SOFT_REBOOT=1 systemctl reboot # Inspect implementation units systemctl cat soft-reboot.target systemd-soft-reboot.service # FD store preserve (unit drop-in sketch) # [Service] # FileDescriptorStoreMax=128 # FileDescriptorStorePreserve=yes References systemd-soft-reboot.service(8) — Debian man page / man7 systemctl(1) — soft-reboot, auto-soft-reboot on reboot when /run/nextroot/ exists, SYSTEMCTL_SKIP_AUTO_SOFT_REBOOT systemd.special(7) — soft-reboot.target systemd.service(5) — FileDescriptorStoreMax=, FileDescriptorStorePreserve= (254+) ArchWiki — Systemd § Soft reboot Portable Services overview — systemd.io/PORTABLE_SERVICES (survivor isolation option cited by the soft-reboot man page) Full reboots are for kernels, firmware, and cold topology. When only userspace needs a clean graph, soft-reboot is the proportionate tool: same kernel, new service manager transaction, optional root flip through /run/nextroot/, and a short list of deliberate pass-through knobs you should almost always leave alone until you have a specific reason not to.