Failure semantics & supervision:
“stopped” is not “crashed”

OrphanedOrphaned02Exit paths vs exit codes — “stopped” is not “crashed”No restart

The plain explanation

A service can stop existing in exactly three ways, and the difference between them decides whether anything should bring it back:

1. It exits. Either cleanly (it finished, returned 0) or by crashing (unhandled error, non-zero exit).
2. It is killed. Something outside it sends a signal — SIGTERM from a stop request, SIGKILL from the OOM killer, SIGKILL after a timeout.
3. It is deliberately stopped. A supervisor (systemd), a script (hermes gateway stop), or a person runs the stop path. The process still dies by signal or exit — but the death was requested, and that changes everything.

Restart=always — the setting nearly everyone puts on a long-running unit and then forgets — covers (1) and most of (2). It explicitly does not cover (3). The man page is unambiguous; this is the sentence to remember:

That is the whole lesson in one clause.

Exit codes are a separate contract

Once "stopped" and "crashed" are different events, you want to say which non-zero exits deserve a restart. systemd gives you three independent layers. Confusing them is the most common supervision bug:

The two overrides, verbatim from man systemd.service:

The sentence that ties them together:

The two magic numbers are not invented: they are the BSD sysexits conventions, in /usr/include/sysexits.h on this host:

#define EX_TEMPFAIL     75      /* temp failure; user is invited to retry */
#define EX_CONFIG       78      /* configuration error */

• 75 = "temporary failure; user is invited to retry." The program tells its supervisor: not my fault, it will probably work next time — relaunch me. So RestartForceExitStatus=75 means "even if my restart policy says no, restart me on 75."
• 78 = "configuration error." The program tells its supervisor: I am misconfigured; relaunching will fail the same way forever and fill your logs. So RestartPreventExitStatus=78 means "never restart me on 78" — a deliberate restart-storm circuit breaker owned by the program itself: the child gets a vote in its own supervision.

The concrete case, from this host's journal

Sep 18 08:56:31 kagami-core systemd[1]: Stopping hermes-gateway.service - Hermes Agent Gateway - Messaging Platform Integration...
Sep 18 08:56:34 kagami-core systemd[1]: hermes-gateway.service: Main process exited, code=exited, status=1/FAILURE
Sep 18 08:56:34 kagami-core systemd[1]: hermes-gateway.service: Failed with result 'exit-code'.
Sep 18 08:56:34 kagami-core systemd[1]: Stopped hermes-gateway.service - Hermes Agent Gateway - Messaging Platform Integration.
...
Sep 21 00:28:42 kagami-core systemd[1]: Started hermes-gateway.service - Hermes Agent Gateway - Messaging Platform Integration.

Read those four lines in order, because the order is the whole diagnosis:

1. Stopping … — systemd initiated a stop. This is the line that decides everything that follows. From this moment on, the death is "a result of systemd operation".
2. Main process exited, code=exited, status=1/FAILURE — the process did come back with exit status 1. It exited, by exit code, not by signal. (Worth knowing: a stop request defaults to SIGTERM, and KillSignal= here is SIGTERM — a program that handles SIGTERM and then exits 1 is a program whose own shutdown code ran and returned non-zero.)
3. Failed with result 'exit-code'. — systemd classifies the unit as failed. Different final state from inactive (dead).
4. Stopped … — the unit is now stopped by request.

The unit on this host is configured, in /etc/systemd/system/hermes-gateway.service:

[Unit]
StartLimitIntervalSec=0

[Service]
Type=simple
Restart=always
RestartSec=5
RestartForceExitStatus=75
RestartPreventExitStatus=78
KillMode=mixed
KillSignal=SIGTERM
TimeoutStopSec=210

Restart=always is there. RestartForceExitStatus=75 is there. Yet nothing restarted the service for 2 days 15 hours 32 minutes, because Restart=always was never consulted. The question was never "should this be restarted?" It was "was this stopped?" The answer was yes, and systemd did what it was asked.

journalctl -u hermes-gateway --since "2026-09-18 08:00" --until "2026-09-18 09:30"
systemctl show -p ActiveState -p SubState -p Result -p NRestarts -p ExecMainStatus hermes-gateway.service

The lifecycle ledger — the third kind of stop

systemd can tell you an exit happened. It cannot tell you, at the moment it happened, that no exit path ran: that is a statement about code that never executed. So a running process leaves itself a note, and the next process reads it. That is the Hermes lifecycle ledger: on boot it detects a previous life terminated without an exit path, in the only words that fit:

Three things to notice:

• The four stop-kinds are now named: clean exit, exit with an error, killed (SIGKILL/OOM), deliberately stopped. The ledger distinguishes the third from all the others because the others leave evidence behind.
• The ledger can only report. It runs inside the process that just started, so it is a historian, not a supervisor. Keep that distinction — it is the bridge into §3.
• It carries last_heartbeat_at and memory figures, turning "it died" into "it died at 01:21 while holding 310 MB" — an anecdote becomes a diagnosis. (Hermes' web-dashboard surfaces these two conditions: memory critical, and "Your agent restarted unexpectedly, most likely because it ran out of memory".)

One more real-world detail, easy to miss and expensive to learn the hard way: Hermes' docs describe what happens when a requested restart does not line up with the supervisor's restart policy:

That last sentence is §2 in six words. A restart request is only a restart if somebody has agreed to catch the exit code. That is why RestartForceExitStatus=75 is in the unit file: the child says "start me again", and the supervisor promised to honour it.

Why it matters

• "Is it running?" is the wrong question. The right one is "is it supposed to be running?" — and only the log's Stopping/Started lines, not the process table, can answer it.
• Restart=always on a unit whose stop path is used by tooling (hermes gateway stop, hermes update, a /restart command, an installer) means each of those paths opens a window where nothing rescues the service. On a headless box, that window is measured in days.
• Exit codes let a program vote on its own supervision. Use them: 75 for "retry me" (once the listener is back up), 78 for "I am misconfigured, do not loop".

OutsideOutside03Who supervises whomSupervisor

The plain explanation

Supervision is not a property of a program. It is a relationship, and the relationship has a direction. Somebody watches somebody else. Three properties decide whether that relationship can actually save you:

1. Liveness vs readiness. Liveness = "the process exists." Readiness = "the process can do its job right now." These are different questions with different answers, and a supervisor that only asks the first one is blind to the most common failure mode there is: a process that is alive and useless.
2. Fate-sharing. Does the watcher share the fate of the watched? If the watchdog runs inside the process it watches, then the failure it exists to detect also disables the detector. This is the single most common supervision mistake, and it is invisible during normal operation because a watchdog that never fires looks identical to a watchdog that works.
3. Give-up policy. A supervisor that retries forever is a restart storm. A supervisor that gives up is a silent outage. Both are policies, and the defaults are chosen for servers, not for your laptop.

What this host actually had (and why it was not supervision)

Three findings:

(a) The unit cannot distinguish "alive" from "working".

Type=simple          # systemd's liveness model = "the main PID exists"
WatchdogUSec=0       # systemd's watchdog is disabled

Type=simple means systemd considers the service started as soon as the process is forked, and healthy as long as the main PID is alive. WatchdogSec= — the mechanism that would catch a wedged process — reads, verbatim:

— is set to 0, i.e. off. So systemctl status printing active (running) is a statement about a PID, not about health. A process deadlocked on a database lock looks identical to a healthy one. This is the default shape of most systemd units, and why the app-level 30-second heartbeat exists.

(b) The watchdog that existed was a detector, and it shared fate with the thing it watched.

~/.hermes/cron/jobs.json contains a job named Restart Watchdog (7afae45c813e), no_agent: true, interval 5 minutes, 14,249 completed runs. Its entire script is this:

#!/usr/bin/env bash
# Restart detector — checks main daemon PID changes
# Silently notifies when the core process restarts.
PID_FILE="/tmp/.hermes_restart_pid"
CURRENT_PID=$(pgrep -f "hermes_cli" 2>/dev/null | head -1)
if [ -z "$CURRENT_PID" ]; then
  exit 0
fi
...
if [ "$CURRENT_PID" != "$STORED_PID" ]; then
  echo "$CURRENT_PID" > "$PID_FILE"
  echo "🔄 Hermes core restarted"
fi

Three separate lessons in twenty-five lines:

• It detects restarts; it does not cause them. It is named "watchdog" and it can never save anything. Naming is not supervision.

• It runs from Hermes' own cron scheduler, i.e. inside the gateway. When the gateway stopped, the watchdog stopped with it. That is fate-sharing, and it is why its last_run_at is 2026-09-21T00:36 — five minutes after the gateway came back, silent for the whole outage.

(c) There was no recovery layer underneath.

• The root crontab contains five entries (backup, mempalace capture, drei data generators) and no gateway check.
• systemctl list-timers --all shows no unit that starts or verifies the gateway.
• No Upholds= (the systemd setting that continuously restarts a unit — "as long as this unit is up, all units listed in Upholds= are started whenever found to be inactive or failed"), no OnFailure=, no StartLimitAction=.

The fix pattern: recovery lives one layer down

The rule:

Cheapest to most correct:

1. A root crontab line or a systemd timer that asks the OS, not the app: systemctl is-active --quiet hermes-gateway || systemctl start hermes-gateway. This is one line, it lives outside the process, and it cannot fate-share. (A timer is slightly better: it gets its own lifecycle, logging in the journal, and can be inspected with systemctl list-timers.)
2. Let systemd do it, but tell it how to tell "alive" from "working": add Type=notify + WatchdogSec= so the app must ping systemd, and use Restart=on-failure, RestartForceExitStatus=75, RestartPreventExitStatus=78. Now a wedged process is a failed process, and the supervisor can act on it.
3. Make the stop path impossible to hit by accident, and be explicit about the deliberate one. systemctl stop is a request that says "stay down". If your tooling wants "restart", it must exit 75 and let the supervisor relaunch — and the supervisor must be configured to honour it (the --external-supervisor contract quoted in §2).

Why a supervisor gives up — start rate limiting vs restart intensity

Anyone who has written a watchdog has met the storm: the service is broken in a way restarting cannot fix (bad config, missing file), and the supervisor restarts it forever at full speed, filling the disk with identical error logs. Every serious supervisor has a give-up rule. Two compared show how universal it is.

systemd (man systemd.unit):

Defaults, from man systemd-system.conf on this host: "DefaultStartLimitIntervalSec= defaults to 10s. DefaultStartLimitBurst= defaults to 5." — i.e. 5 starts in 10 seconds, then start request repeated too quickly and a unit that stays down. With RestartSec=5, a few attempts and then silence.

Erlang/OTP (about twenty years earlier, and the terminology is better):

The Erlang version is strictly better for one reason: when the supervisor gives up, it propagates upward — "the next higher-level supervisor takes some action. It either restarts the terminated supervisor or terminates itself." That is a ladder of supervision, each rung able to save the one below. systemd's give-up is a leaf: the unit goes failed, and that branch is dead. That is what a 2.5-day outage looks like.

One more detail, on unintentional storms:

Why it matters

• A monitor you cannot rely on during the failure is decoration. The test for every watchdog: "if the thing I am watching died right now, would I still find out?" If the watchdog is a cron job inside it, no.
• Liveness is cheap, readiness is what you actually want, and conflating them means a deadlocked service reports healthy forever.
• Every supervisor's give-up rule is a policy you inherited by not setting it. Know your numbers: 5 starts per 10s, RestartSec, and TimeoutStopSec=210 here — how long a stop may take before SIGKILL.