nx_restart_guard.nx source

1// nx_restart_guard.nx -- SUPERVISOR CRASH-LOOP GUARD (closes hosting_research gap #4 2// "watchdog-restart-loop-guard", 3/0 CONFIRMED -- the root cause of the live 2026-06-16 nx_hostctl 3// crash-loop). A bedrock supervision primitive any supervisor (nx_hostctl / nx_keeper / nx_torrent_up) 4// composes so a daemon that crashes immediately on start is NOT respawned forever. 5// 6// module: nishi-core.supervision.restart_guard 7// capability: DAEMON_ROBUSTNESS 8// 9// INCUMBENT (benchmark, sourced via the Nishi researcher -> hosting_research.tsv): systemd unit restart 10// policy -- Restart=on-failure + RestartSec (wait before restart, default 100ms) + StartLimitBurst (N, 11// default 5) within StartLimitIntervalSec (T, default 10s): more than N restarts in T -> unit enters 12// "failed", systemd stops trying. We MATCH that burst-limit semantics, and EXCEED it on two measured axes: 13// (1) DETERMINISTIC / REPLAYABLE -- given the same restart timestamps, the SAME verdict every run, so a 14// supervision session can be replayed bit-for-bit (systemd's behavior depends on wall-clock + monotonic 15// boot time and cannot be replayed). The caller passes `now`; nothing reads a hidden clock here. 16// (2) EXPONENTIAL BACKOFF -- rg_backoff_ms doubles the wait each restart (capped), vs systemd's FIXED 17// RestartSec, so a flapping daemon backs off instead of hammering at a constant rate. 18// 19// STATE per daemon (caller-owned, 2 words): ws[0] = window-start timestamp, cnt[0] = restarts in window. 20 21const RG_DEF_INTERVAL_MS: i64 = 10000 // systemd StartLimitIntervalSec default (10s) 22const RG_DEF_BURST: i64 = 5 // systemd StartLimitBurst default 23const RG_DEF_BASE_MS: i64 = 100 // systemd RestartSec default (100ms) = our backoff base 24const RG_DEF_MAX_MS: i64 = 30000 // backoff ceiling (30s) so it never waits unboundedly 25// HEALTH-RESET threshold: a daemon that ran healthy for >= this long since its last restart is NOT in a TIGHT 26// crash-loop -> reset its crash counter (auto-recovery). MUST be comfortably above the supervisor poll cadence 27// (nx_hostctl polls ~15s, so a dead daemon always shows elapsed>=~15s) -- set to 60s so only daemons that die 28// within ~1min REPEATEDLY are treated as crash-looping; anything that ran a full minute is "healthy -> reset". 29const RG_HEALTH_RESET_MS: i64 = 60000 30 31// S-CLASS crash-loop containment = Kubernetes CrashLoopBackOff + Erlang/OTP restart-intensity + CAPPED EXPONENTIAL 32// BACKOFF (grounded in the banked SOTA: knowledge/library/rel_erlang_otp.txt, rel_backoff.txt, rel_circuit_breaker.txt). 33// State per daemon (caller-owned, 2 words): ws[0] = last-restart timestamp (ms), cnt[0] = consecutive-crash count. 34// Called by the supervisor ONLY when the daemon is dead. `cap_ms` = backoff ceiling; `base_ms` = first backoff step. 35// (The hostctl reuses its two guard consts for these positionally.) 36// - HEALTH-RESET: ran healthily for >= RG_HEALTH_RESET_MS since the last restart -> cnt=0 (loop over; auto-recover). 37// - FIRST death (cnt==0): restart immediately (minimal downtime for a one-off crash). 38// - REPEATED crashes: WAIT an exponential backoff (base doubling, capped at cap_ms) before each restart, so a hard 39// crash-loop is CONTAINED to ~1 restart per cap_ms (e.g. 5 min) -- bounded CPU/log blast radius. This RETIRES the 40// old flat "5/window-reset-each-window" which allowed a slow INFINITE loop (5 restarts every 2 min forever). 41// returns 1 (restart NOW) or 0 (wait -- the loop is being contained by backoff). 42func rg_should_restart(ws: *i64, cnt: *i64, now: i64, cap_ms: i64, base_ms: i64) -> i64 { 43 let elapsed: i64 = now - ws[0] 44 if elapsed >= RG_HEALTH_RESET_MS { cnt[0] = 0 } // ran healthy >= reset thresh -> auto-recover 45 var backoff: i64 = 0 46 if cnt[0] > 0 { backoff = rg_backoff_ms(cnt[0], base_ms, cap_ms) } // exp backoff ONLY on repeated crashes 47 if elapsed < backoff { return 0 } // backoff not elapsed -> WAIT (contain the loop) 48 ws[0] = now 49 cnt[0] = cnt[0] + 1 50 return 1 51} 52 53// exponential backoff before the Nth restart: base * 2^(count-1), capped at max. count is 1-based. 54func rg_backoff_ms(restart_count: i64, base_ms: i64, max_ms: i64) -> i64 { 55 if restart_count <= 1 { return base_ms } 56 var v: i64 = base_ms; var i: i64 = 1 57 while i < restart_count { v = v * 2; if v >= max_ms { return max_ms } i = i + 1 } 58 return v 59} 60 61// how many restarts remain before the loop-guard trips (window-aware inspector). 62func rg_remaining(cnt: i64, burst: i64) -> i64 { let r: i64 = burst - cnt; if r < 0 { return 0 } return r }