NISHI ENGINEERING · EVIDENCE · 2026-07-25

The self-retiring service

One constant took the gallery down, wedged the control plane, and hid behind a status page that said UP.

1 · What the gallery was actually doing

Every sovereign service runs the same accept loop. Here it is, unedited, from nx_gallery_gateway.nx:269:

while served < budget {
    let cfd = sys_accept(lfd)
    if cfd >= 0 { ...fork-per-request... }
    served = served + 1        // increments even on a FAILED accept
}
sys_close(lfd); sys_exit(0)    // the service retires itself

budget arrives as a launch argument. The supervisor was passing 5000. A gallery page load spends dozens of connections on thumbnails and range requests, so the gateway reached its budget and called sys_exit(0) after minutes of browsing.

That alone would have been a visible restart. What made it look like an intermittent login bug is the second half:

// nx_hostctl.nx:798
[guard] nx_gallery_gateway.elf crash-looping -> BACKING OFF

The supervisor cannot distinguish “exited cleanly at budget” from “crashed on startup.” Repeated exits read as a crash loop, so the guard stopped restarting it. The gallery went genuinely down — while nx_status, which probes liveness between respawns, kept printing UP. The instrument agreed with the wrong answer.

2 · The measurement that changed the scope

Grepping the built supervisor binary for the corrected constant surfaced every sibling launch line in the same string table. The gallery was not special:

The management API exiting every 5000 requests is the long-documented “MCP transport flake” and the recurring “mgmt wedged” incidents. It is also why the MCP client failed to connect at the start of this session — the diagnosis began by tripping over the thing being diagnosed.

The gen orchestrator was left at 5000 on purpose. Its budget argument sits in an unverified position, and changing a control value that has not been read is how you trade one outage for another.

This had already been found once

// nx_docportal_search_serve.nx:303
// 5000 requests -- the ~15s respawn gap was the "takes forever"
// the operator hit (2026-07-03). With the leak...

The same failure was diagnosed three weeks earlier on a different service, fixed there, and never swept. A fix applied to one call site and not to its class is a fix with a half-life.

3 · The vault stall was something else entirely

The media-vault migration had stopped at 21,521 files against a 76,004-recording catalogue, with no fatal counter and no verdict=DONE — the signature of a process killed from outside rather than one that failed.

// nx_mvault_walk.nx -- bg worker, before
sys_close(0)
sys_close(1)
sys_close(2)          // detaches the pipe, NOT the session

Closing the standard descriptors detaches the capture pipe but leaves the worker in the tools-daemon’s session. When that daemon restarted, the worker died with it. sys_setsid appears in zero organs under the build tree, while nx_setsid is called about sixty times by the supervisor alone and by every durable daemon in the ecosystem. The survival pattern was already there; one long-running worker had skipped it.

Fixed, rebuilt, promoted over the API, selftest 7/7 GREEN, and relaunched.

The guard that proved itself on first use

Two concurrent walkers both read-modify-write a lock-free segment store — that corrupts a 20,000-record plane rather than merely duplicating work. The old control for this was operator memory. It is now structural:

{"action":"REFUSED","verdict":"SINGLE-WALKER",
 "reason":"another nx_mvault_walk is already running;
           not starting a second (seg-store RMW would corrupt)"}

That refusal is the first live output of the new guard — and it doubled as the liveness proof that the session-detach fix had worked, because the only way to earn it was for the previous worker to still be running.

4 · Regression proof

A fix that relies on nobody re-typing the number is not a fix. nx_svc_budget_gate is built, promoted, registered with an MCP contract, and carries ten teeth:

• T1–T7 — each launch constant must carry an unbounded budget.
• T8 — the live deployed binary must carry the fix, not merely the source. This session produced two separate instances of source-versus-deployed drift, including a carve-out recorded in memory as “landed in source” that was not in source at all. Source-GREEN plus binary-RED means not shipped.
• T9–T10 — mutation teeth: they fail if the exact literal 5000 ever returns, so a revert cannot pass quietly.

T8 is independently confirmed live — the deployed supervisor now execs galx_gw_store 1000000000 18090.

5 · Debt eaten along the way

Two blockers stood between the fix and a shell-free deploy. Both are now permanently closed:

• The rename bridge only matched one shape. It handled <t>.elf.new but not the extensionless nx_hostctl artifact (nx_hostctl.new), so every hostctl deploy aborted with “nothing staged.” Generalized to strip either suffix.
• The promote deny blocked the media vault. A substring rule protecting credential vaults also blocked nx_mvault_*, forcing an ssh rename on every rebuild. Now an enumerated exact-match carve-out — every other *vault* name stays denied by construction.

5b · A second instrument caught lying

Chasing the first fix surfaced another. nx_sov_build_run creates /tmp/<name>.sov.elf even when the compiler fails — an empty .s yields an empty ELF. The supervisor’s build handler only checked whether that file was missing, so a zero-byte build walked straight through:

BUILT /tmp/nx_proc_kill.sov.elf size=0
STAGED -> nx_proc_kill.sov.elf.new
BUILDRUN OK

A build that produced nothing reported success and staged the nothing. This was load-bearing: nx_dedup_migrate.nx:172 keys its ACCEPT on the literal string “BUILDRUN OK”, so a failed compile was scoring as a successful one inside the self-emitting-code lane.

The handler now validates the artifact — size plus ELF magic — never the exit code alone, and refuses to stage garbage. Proven on the same input, before and after:

BUILT /tmp/nx_proc_kill.sov.elf size=0
BUILDRUN FAIL: artifact is not a valid ELF (compile produced nothing) -- NOT staged

A known-good target still builds and stages unchanged, so the happy path is intact.

How the failing targets got there

They were collateral from a concurrent session’s duplicate-file reconcile. runtime/nx_proc_ctl.nx was renamed aside while three consumers still imported it by name and called its API, so nx_proc_ctl_gate, nx_servicectl and nx_proc_kill all now fail with expand_imports failed. Notably the casualty includes the module’s own 5/5 gate — the proof cannot build while the refactor is open. Left uncorrected on purpose: another session is mid-refactor on those exact files, and racing it would clobber. Filed instead.

6 · What this cost, and what it bought

6b · The keystone underneath: a leak with no free

The vault never resumed, and the reason was not the walker. The host sat ~20.6 GB into 24.3 GB of swap, which parks a process doing large sequential I/O in uninterruptible sleep — where it makes no progress and ignores SIGKILL. That is a leak presenting as a hang.

The cause: ss_open loads every live segment of a store into memory, and func ss_close grepped to zero matches against roughly 180 ss_open call sites. Nothing could ever give the memory back. Every seg-store consumer leaked by construction.

Building the missing half took four rounds, and each round found something the previous one had not:

• the handle also owns a query-scratch arena the first cut missed;
• ss_open2 allocated scratch inside its per-segment loop — unreachable from the handle, so no correct ss_close could have reclaimed it;
• segment names each took a 4 KiB page for a 128-byte string (~468 KiB per open on the live 117-segment store);
• and a genuine correctness defect: in the merged .idx layout, keys/terms/post are interior pointers into one mapping, not three. Freeing them independently is wrong — and it is path-dependent, because the legacy fallback really does return three. ss_close now branches, checking adjacency first and only then the header magic, so the header is never read off the front of a legacy blob.

The instrument was lying, in the other direction

Through all four rounds the leak tooth reported FAIL. It was measuring nothing: /proc/self/status separates label from value with a tab, not a space, so the parser parked on the separator, read zero digits, returned 0, and failed its own > 0 guard unconditionally.

It survived four rounds of review for an uncomfortable reason — red matched what was already believed. The usual warning is about optimistic instruments; this was the mirror image, a pessimistic one confirming an assumed problem and never being questioned. What exposed it was making the tooth print a number instead of a verdict: no probe branch fired and the reading was zero, which said “label matched, no digits parsed” and pointed straight at the separator.

With the probe repaired, the measurement is trustworthy and the earlier work is vindicated — it had landed all along, it simply could not be seen:

T8 delta_kb=800 vm0_kb=1330000 vm1_kb=1330800
   iters=200 bytes_per_open=4096 probe=LIVE   -> 9/9 GREEN

Not fully closed. One page per open still leaks; the tooth passes because 800 KB sits under the slack, not because it is zero. The slack has been ratcheted from 1024 KB to 850 KB so the gain cannot quietly erode — any regression worse than ~4.3 KB per open now fails immediately. The rule attached to it: never loosen the ratchet to make a build pass.

7 · Laws banked