Platform stability, root-caused from the first byte up

One shared storage primitive with no close explains nine leaking daemons, 92% swap exhaustion, and 400 wedge-kills. Measured 2026-07-25 — every number below came from a live read, none from inference.

The symptom, and why every instrument missed it

The platform had been unstable for days: services restarting, probes hanging, the box occasionally dying outright. The health plane reported {overall:OK, degraded:0, down:0} throughout.

Baseline, measured

The causal chain: thrash slows every probe → the supervisor returns HUNG(up-but-not-serving) → it SIGKILLs a service that was healthy but merely slow → respawn adds load → repeat. The restart storm was a symptom of memory pressure, not a fault in the services being killed.

Layer 1 — the keystone: a storage primitive with no close

Nine unrelated daemons all carried the same signature in /proc/<pid>/status: VmSize == VmPeak with VmData ≈ VmSize. Address space that only ever grows and never shrinks — the mechanical fingerprint of allocate-without-free.

Nine organs, one signature. That is not nine bugs — that is one bug in something they all share. Tracing the hottest one down:

nx_hub_gw  → hgw_ncfg_to_buf()   called TWICE per HTTP request
           → ncfg_open()         nx_native_config.nx:79
           → ss_open()           nx_seg_store.nx:1494
           → ss_open2(prefix, 0) "read every segment file fully into anon RAM"

$ grep "func ss_close" buildroot/runtime   →  0 matches
$ grep "func sys_munmap" buildroot/runtime →  nx_syscalls.nx:148  (it exists)

This is the same defect class our own nx_segguard.sh header documented after the 2026-07-20 host death (“re-read the WHOLE store inside per-row loops and never munmap … grew to 16.4GB and OOM-killed the box”). Segguard bounded the symptom, segment count. The primitive underneath was never given a close.

Layer 2 — a guard that queued forever FIXED

nx_seg_compact_cli takes a blocking flock(LOCK_EX). Proven from the kernel, not guessed:

/proc/17774/syscall  →  73 0x3 0x2 ...     (73 = flock, 0x2 = LOCK_EX)
/proc/17774/wchan    →  locks_lock_inode_wait

One long compaction meant every subsequent 10-minute tick blocked forever, pinning three processes each (guard + subshell + CLI), none ever exiting. Measured: 48 nx_segguard.sh + 24 nx_seg_compact_cli stacked, oldest ~6 hours, thread count climbing 1868→1976 in twenty minutes.

Fix: a non-blocking mkdir singleton lock — a sweep that cannot run because one is already running must skip, not queue — plus a stale-lock break and a watchdog SIGKILL past compact-timeout-seconds. Both bounds live in segguard.conf, not as literals.

Gate: nx_segguard_lock_gate.sh — GREEN 6/6, non-vacuity proven. T5 strips the lock block, rebuilds, and requires the skip behaviour to disappear. A gate that passes with and without the fix proves nothing.

Live-verified: the next tick ran the edited script clean — SWEEP scanned=916 compacted=3 folded-segments=29 failed=0.

A negative result we are publishing rather than burying

A memory-safety bug found on the way

In nx_hub_gw, bodybuf was allocated at 256 KiB (HGW_MAGIC_262144) while all six writers are handed HGW_MAGIC_2097152 — 2 MiB — as its capacity, including hgw_read_file on arbitrary docroot files. Any page over 256 KiB wrote up to 1.75 MiB past the end of the mapping, into the neighbouring reused buffers. The allocation was simply never raised when the cap constant was. Fixed in source; build GREEN at 408,877 bytes.

The remedy already existed — three times — and was never generalised

Before building anything we searched for prior art. It was there, and it changes the plan entirely.

nx_docportal_search_seg.nx:128

func dss_open_maybe_cached(prefix) -> *i64 {
    if not the web prefix { return ss_open(prefix) }      // uncached fallback
    dsc_manifest_sig(prefix, cur)                          // (st_size, st_mtime) of the manifest
    if handle != 0 and cur == cached_sig { return handle } // HIT
    handle = ss_open2(prefix, 1)                           // MISS -> reopen
    cached_sig = cur
    return handle
}

It was built for latency — a per-request ss_open measured ~0.7s per query — not for memory. But it incidentally eliminates the leak, because you cannot leak what you never allocate twice.

And the same remedy has been independently rediscovered at least three times, each time locally:

The correct fix, and why it beats ss_close

Promote the pattern into nx_seg_store as ss_open_cached(prefix) — a small prefix-keyed memo of {prefix, signature, handle} — and migrate consumers onto it. That is decisively better than building ss_close first:

• Purely additive — no munmap anywhere. It therefore cannot produce the partial-unmap or use-after-free hazard the merged-.idx interior-pointer case creates.
• Fixes every consumer at once, rather than one organ at a time.
• Manifest-mtime invalidation preserves live-edit-without-restart — the exact property nx_hub_gw depends on.

A cache miss still re-opens without freeing, so ss_close stays worth building eventually — but it drops from urgent to a rare-path cleanup, because re-opens fall from per request to per store edit. The oracle is unchanged and sharp: hub_gw's measured 1,336 kB/request must fall to ~0 between edits.

Why ss_close is still not a one-afternoon fix

Having characterised it precisely, the danger is specific rather than vague. ss_load_aux2 carries two incompatible ownership models behind one out-array:

MERGED .idx (returns 1)   ONE mapping ib of size szp[0], then publishes
                          outs[0]=ib+16   outs[2]=ib+16+kl   outs[4]=ib+16+kl+tl
                          → outs[2] and outs[4] are INTERIOR POINTERS

LEGACY 3-file (returns 0) outs[0], outs[2], outs[4] ARE three independent
                          mappings with sizes outs[1], outs[3], outs[5]

And ss_open2 discards that return value — so the handle has no record of which model is in force, and in the merged case neither the base pointer nor its true size is stored anywhere. A ss_close that frees three regions is correct for legacy stores and catastrophic for merged ones: a partial unmap of a live mapping at a non-base address.

What is already safely recoverable from the handle, with no ledger change: the docs blob and its exact size, the live-doc map (size recomputable), the six query-scratch arenas, and the handle itself. What must be added first: ss_load_aux2 must emit base, total size and mode; ss_open2 must persist those per segment — which needs three more slots per segment against a handle that has only nine spare slots in total, so the allocation and every reader index must migrate together.

What we built this round — and what it is not yet

ss_open_cached(prefix) now exists in nx_seg_store: a 16-slot prefix-keyed memo of {prefix, st_size, st_mtime, handle}, using static pointers to lazily-mapped tables (a BSS static array silently crashes handler modules on startup, so that shape is deliberate), with one reused scratch buffer so the manifest probe allocates nothing per call, and a fail-open fallback to plain ss_open when the table is full — caching must never be able to break a caller.

It is wired in one line:

nx_native_config.nx:88
func ncfg_open(prefix: *u8) -> *i64 { return ss_open_cached(prefix) }

That routes all 23 ncfg_open call sites through the cache at once — including nx_hub_gw's three per-request opens. It compiles: BUILT 414,100 bytes, up from 408,877 before the cache, which also proves the compiler is not dead-stripping it.

Gated GREEN 5/5, and mutation-proven

The gate was built, promoted, registered and executed live on the NAS against real seg-stores:

=== nx_seg_cache_gate -- ss_open_cached memoises without ever freeing ===
  T1-cached-open-same-handle         PASS
  T2-NONVACUITY-bare-open-differs    PASS
  T3-manifest-change-refreshes       PASS
  T4-value-byte-correct-via-cache    PASS
  T5-table-full-fails-open-works     PASS
verdict=GREEN passes=5/5

A built-in control tooth is good but not sufficient, so we also ran the mutation. Disabling only the cache-hit comparison — swapping the signature check for an impossible sentinel — and rebuilding gave:

  T1-cached-open-same-handle         FAIL
  T2-NONVACUITY-bare-open-differs    PASS
  T3-manifest-change-refreshes       PASS
  T4-value-byte-correct-via-cache    PASS
  T5-table-full-fails-open-works     PASS
verdict=RED passes=4/5

Exactly one tooth died, and it was the right one; T2–T5 correctly survived because they do not depend on the hit path. Reverting restored GREEN 5/5. T1 is causally testing the cache and cannot pass for free.

The gate it needs, stated before writing it so it cannot be quietly weakened: T1 repeated cached opens of an unchanged store return the same handle; T2 non-vacuity — bare ss_open on the same store returns different handles, proving T1 tests the cache rather than a coincidence; T3 after a commit that changes the manifest, a new handle is returned, preserving live-edit-without-restart; T4 values read through a cached handle are byte-correct; T5 a full table falls back to a working handle.

One bound worth stating plainly: st_mtime has one-second granularity, so a rewrite landing in the same second that leaves the manifest byte-length identical can serve a single stale read. That is the same bound the already-proven dss_open_maybe_cached has run under in production, and the gate should assert it rather than pretend it away.

We built the missing instrument — and it indicted itself

nx_resmon is built, promoted, registered and running live on the NAS. It gives the health plane the resource axis it never had, on two axes: pressure (swap consumed, memory available) and a leak census — processes whose VmSize == VmPeak above a committed-memory floor. The census is the valuable half, because it is a leading indicator: it fires while swap is still healthy. Every threshold lives in resmon.conf; the exit code is the verdict, so callers can gate on it.

The first live run closed the gap it was built for, in one call:

nx_resmon  -> verdict=RED sev=2   swap_used_permil=934
nx_health  -> {"overall":"OK","degraded":0,"down":0}

Same box, same moment. One instrument can see the outage class; the other cannot.

Then it reported something impossible

leak_suspects=63   worst_anon_kb=42581844   (= 40.6 GiB)

The box has 36.9 GB of RAM. A single process cannot have committed 40.6 GiB. The screen was wrong: VmData counts reserved address space, not touched pages — a tool that maps a large arena and never faults it in is not leaking. (That same reservation is what makes Committed_AS 179 GB against a 42.7 GB limit — a second finding that falls straight out of the correction.)

Re-screening on committed memory, VmRSS + VmSwap:

Worth noting the corrected census finds 23 leak suspects — more than the nine we had found by hand, because the manual sweep only examined the top of the root cgroup. The instrument beats the analyst, which is the whole point of building one.

Gated, and the gate is mutation-proven too

Before gating we refactored: the two policy predicates — rm_is_leaker and rm_verdict — moved into nx_resmon_lib.nx, so the gate exercises the same code the organ runs rather than a reimplementation, and needs no /proc and no fixtures on disk. Fully deterministic.

=== nx_resmon_gate -- pressure + leak-census predicates ===
  T1-verdict-ladder-is-threshold-driven  PASS
  T2-NONVACUITY-unreachable-stays-GREEN  PASS
  T3-each-axis-can-raise-alone           PASS
  T4-reserved-not-leak-committed-is      PASS
  T5-field-parse-status-conf-and-absent  PASS
verdict=GREEN passes=5/5

Then we deleted only the committed-memory floor from rm_is_leaker and rebuilt:

  T4-reserved-not-leak-committed-is      FAIL
verdict=RED passes=4/5     (T1, T2, T3, T5 correctly survived)

Reverting restored GREEN 5/5 and the gate binary returned to exactly 16,958 bytes, byte-identical to the pre-mutation build — so the revert is provably clean.

The instability showed up while we were measuring it

Mid-session the sovereign edge on :8443 refused connections outright. The host stayed ARP-reachable with 3/3 ping and ports 443/5000/22 up throughout, and :8443 recovered on its own inside a minute — the guard respawning a crash-looped edge. That is a live instance of exactly the class this workstream is about: a box under sustained memory pressure drops its public surface and comes back, and every instrument except the resource axis reads it as fine.

It is also why the discipline is to ping and ARP first. A refused port is not automatically a transport flake; sometimes it is the outage you are writing the page about.

The instrument earned its keep: a live incident, caught and mitigated

Hours after building nx_resmon, it caught a real emergency that every other instrument reported as healthy. Two readings, same moment:

nx_health :  {"overall":"OK","degraded":0,"down":0}
nx_resmon :  swap_used_permil=999   SwapFree = 1,136 kB of 24,252,332
             mem_avail_permil=146   verdict=RED sev=2

Swap was 99.995% exhausted — 1.1 MB free. That single number explained symptoms we had been treating as unrelated transport noise: POST /mcp intermittently returned HTTP 200 carrying the static site homepage instead of routing to the tools daemon, and the edge had refused connections outright minutes earlier and self-recovered.

The supervisor already knew. Its own snapshot flags:

SVC nx_tools_api_serve.elf        18096 UP 1 1 1 0    <- dup=1 loop=1
SVC nx_docportal_admin_daemon.elf 18456 UP 1 1 1 0    <- dup=1 loop=1
SVC (every other service)               UP 1 0 0 0

Two services flagged duplicated and crash-looping — and the health rollup discarded that and returned OK. Wiring those flags into health is no longer a theoretical improvement; it is evidenced.

Mitigation, and an honest reading of it

Applied in increasing order of blast radius, all documented and guard-respawned: restart survey, then office (zero downtime), then sites (dropped the in-flight connection; back to HTTP 200 on the first probe after ~40s).

MemAvailable nearly tripled; SwapFree barely moved, because swapped-out cold pages are not faulted back until something touches them. MemAvailable, not SwapFree, is the near-term OOM guard — judge recovery on that. The verdict correctly remains RED: we bought headroom, we did not fix the defect.

Two sessions, one primitive, and a fix of mine that was wrong

While this work was in flight, a parallel session was editing the same file. They built ss_close — correctly handling the merged-.idx interior-pointer case — and we had independently built ss_open_cached. That combination creates a hazard neither session could see alone: the cache hands the same handle to many callers, so closing a cached handle strands a freed pointer in the cache.

We filed that as urgent and proposed a fix: evict the slot at the top of ss_close. That fix was wrong. Eviction removes the cache entry, but a caller that has already taken the handle is still dereferencing it — the free still faults. We would have shipped a real fault while believing we had closed one.

The sibling had already solved it, and better. Their design is retire-then-explicitly-reap:

• On invalidation the old handle is remembered, not freed — retiring cannot fault, by construction.
• Reclamation is an opt-in ss_cache_reap(), called only where a consumer structurally holds no handle — the top of a request loop, between batches.
• If the retire ring overflows it drops the pointer rather than freeing it. Overflow degrades to exactly today's behaviour, never to a bad free.

They also added the tooth that makes the claim honest, to our gate: T7 reads the original key back through the retired handle. If retire ever silently becomes a free, that tooth faults or returns garbage.

The merged state, verified

Neither session had run the combined result. We rebuilt and ran it live:

  T1-cached-open-same-handle         PASS
  T2-NONVACUITY-bare-open-differs    PASS
  T3-manifest-change-refreshes       PASS
  T4-value-byte-correct-via-cache    PASS
  T5-table-full-fails-open-works     PASS
  T6-invalidated-handle-RETIRED      PASS
  T7-SAFETY-retired-still-readable   PASS
  T8-reap-frees-and-empties-ring     PASS
verdict=GREEN passes=8/8

Honest status

nx_hub_gw cannot currently be shipped by the API-pure loop, but a second correction is owed here, because the first version of this paragraph named the wrong cause. nx_hub_gw.elf is already in md_upload_target_ok — the upload/deploy allowlist. That claim was inferred from a summary list in our own doctrine notes rather than read out of the function, and it was wrong. Probing /api/deploy target=hubgw returned the actual answer: unknown target (not in deploy_targets.conf allowlist). The genuinely missing pieces are (a) a row in the data-driven deploy_targets.conf, and (b) a real hubgwdeploy/hubgwrollback sub in nx_hostctl — every existing target names a bespoke sub, and grep hubgw and grep cmd_kickhub both return zero. (b) requires rebuilding and self-swapping the supervisor, which is the highest-stakes operation on the platform, so it is scheduled deliberately rather than tacked on. Correction to an earlier version of this page: it is not unsupervised — nx_hostctl does launch it (HC_HUB_GW_CMD, dispatched at nx_hostctl.nx:665) and it demonstrably respawned during this investigation. It is absent from the 14-service status and health inventory, which is a monitoring gap, not a supervision gap. The original claim was inferred from PPID=1 plus its absence from nx_status; direct evidence refined it.

The finding that only appears when two sessions are cross-checked

A parallel workstream found that seven supervised services were launched with budget=5000 against while served < budget { accept } sys_exit(0) — they could retire themselves, and the supervisor's crash-loop backoff then left the surface genuinely down. That was swept to 1000000000. The fix is correct and necessary.

But nx_hub_gw is one of the seven, and it leaks through the ss_close keystone above. We measured the rate directly against the live daemon:

cmdline : ./nx_hub_gw.elf 18792 ... 1000000000 65536 3 4   (new unbounded budget, live)
BEFORE  : VmData 4,279,572 kB
          25 requests to /hub
AFTER   : VmData 4,312,972 kB      VmSize == VmPeak throughout

LEAK RATE = 1,336 kB PER REQUEST

At the old budget, the daemon retired after 5000 × 1.336 MB = 6.37 GiB — which independently reproduces the 6.58 GiB measured on the previous PID before it retired mid-investigation. The model is confirmed from two directions.

Method

Run API-first over the sovereign MCP plane and the mgmt API — zero ssh. Techniques worth reusing: cgroup memory.memsw.usage_in_bytes per slice to localise consumption without a process table; /sys/fs/cgroup/memory/cgroup.procs for an exact PID list; curl --parallel over batched tools/call bodies for a 575-process sweep in about a minute; and /proc/<pid>/wchan + /proc/<pid>/syscall to turn “compaction is slow” into “compaction is blocked on LOCK_EX”, which is an entirely different fix.