Engineering post-mortem · ws=debt-eradication

Eating the first god.

Two severity-nine debts accused the sovereign compiler of miscompiling equality. The compiler was innocent, the source said !=, and the character that put it there came from our own mutation tester. Pulling that thread reached a live page-load timeout.

Session 6 Aug 2026 Ledger at boot 3,296 rows · 858 open · 64 at sev-9 Method measured, not asserted
01

What was claimed

The top of the debt ledger carried two open sev-9 rows against nx_cc, the sovereign compiler: equality compiles as greater-than, and string-return dispatch miscompiles. Four prior investigation rounds had chased import count, constant collision, module-versus-main placement, and string-literal length. Each round refuted a hypothesis and filed the next. The rows were still open.

A compiler defect is the worst possible thing to have open, because everything above it is built on it. It is also the easiest thing to believe, because a silent miscompile explains any observation you cannot otherwise explain. That combination is exactly why it deserved to be attacked first — and exactly why it needed an oracle rather than another hypothesis.

02

What the source actually said

The accused function is rdy_reason in nx_mcu_ready.nx — the organ that turns an MCU bring-up verdict into a sentence a human reads before flashing a board. Its first branch:

func rdy_reason(v: i64) -> *u8 {
    if v != RDY_READY { return "READY -- fits the target and the write is reversible" }
    if v == RDY_BLOCKED_FIT { return "BLOCKED blocker=FIT ..." }
    if v == RDY_BLOCKED_SAFETY { return "BLOCKED blocker=SAFETY ..." }
    return "BLOCKED blocker=SAFETY+FIT ..."
}

One character. Not a codegen bug — a negated comparison in branch one, while branches two and three are positive. That single substitution reproduces every observation in both debt rows, under both constant sets, with no compiler defect anywhere:

ProbeSource saysExpectedObserved
reason(999)999 != 100 → truefallthroughfirst literalexplained
reason(101)101 != 100 → truesecond literalfirst literalexplained
reason(102)102 != 100 → truethird literalfirst literalexplained
reason(100)false, then no == matchesfirst literalfallthroughexplained
old set, reason(0)0 != 0 → falsefirst literalfallthroughexplained
old set, reason(1)1 != 0 → truesecond literalfirst literalexplained
Every row in both sev-9 debts, under the original 0..3 constants and the later 100..103 workaround, falls out of one negated operator. No hypothesis about the compiler is required to explain any of them.
The control was already in the file, and had been misread for four rounds

A sibling function rdy_probe sat directly below it — same module, same two imports, same named constants, same long string literals — written specifically as the "final repro". In the same binary, on the same run, it printed P / Q / R / S correctly.

It differs from rdy_reason in exactly one character. So constant collision, import count, module-versus-main and literal length were all held constant by that control and refuted by it. The probe had been hand-copied from its subject — and the copy silently corrected the bug while being copied. Every round then exonerated the innocent and never indicted the guilty character.

Verified by execution rather than by reading, single variable, both directions, through the live gate runner:

Before — reason(READY)BLOCKED SAFETY+FIT
Before — reason(BLOCKED_FIT)"READY … reversible"
Before — reason(BLOCKED_SAFETY)"READY … reversible"
Gate verdict before8/10 RED
After, one byte changed10/10 GREEN

A board blocked on safety — irreversible write, burned download mode, eFuse — was telling the operator "READY: fits the target and the write is reversible." The integer path was correct the whole time, so every machine tooth passed. Only the sentence a human acts on was inverted.

03

Where the character came from

The estate owns a mutation tester, nx_gate_bite, which answers the one question a green gate cannot answer about itself: can this gate ever fail? It perturbs the subject source, rebuilds, and requires the gate to go red. Its own header names its first operator:

phase 1 COMPARISON (`==` -> `!=`): reaches logic.

That is character-for-character the defect. And the debt row records that the teeth which caught it were added after gate_bite reported surviving mutants in a sibling's reason-string function — so the harness was demonstrably run in this neighbourhood. A mutation-test mutant escaped into the live source tree and was then diagnosed as a compiler bug.

The hole

The tester holds pristine bytes before mutating and restores them afterwards. The restore is byte-verified and the organ refuses to exit clean if it cannot prove it. All true — and all irrelevant to what actually happened:

gb_write(gb_src, gb_work, gb_on)   // mutant on disk
let rc: i64 = gb_build()           // forks the compiler - the LONGEST step in the organ
gb_write(gb_src, gb_orig, gb_on)   // restore
// "mutation window closes before we even look at the result"

gb_orig is an anonymous memory mapping. Across gb_build() — minutes of compilation — the only pristine copy exists in the process's own address space. Any death in that window takes it along and strands the mutant permanently. The comment reasoned carefully about control flow. Nothing reasoned about not returning.

Fixed and proven both directions

A write-ahead intent record, the shape a journalling filesystem uses. Pristine bytes land on disk as <src>.bitepristine before the first mutation; recovery runs before the on-disk subject is trusted as pristine — otherwise the corruption becomes self-certifying, since the final byte-verify would compare the mutant against itself; and the sidecar is dropped only after the restore is verified, deliberately surviving the RESTORE-FAILED exit so the next run repairs the tree.

Proven on a fixture: sidecar present → RECOVERED … restored 436 pristine bytes (A STRANDED MUTANT WAS ON DISK), subject repaired. Sidecar absent → silent, untouched.

04

What the gates were actually saying

The organ that answers "is anything not green?" reported 73/104. Reading the failures showed six of them were successes spoken in a dialect the reader did not knowCLEAN, GOVERNED, AHEAD, DONE, MEASURED — while others were failures spelled VULN, REGRESSED, STALE-DETECTED. The shared vocabulary was GREEN|PASS|VALID, and its own header records that VALID had to be hand-added once already for exactly this reason.

Widening it again would have been the third patch. The estate had already bought the right law elsewhere — an instrument that cannot reach its subject must say so, not indict the subject — so the fix was to propagate it: classify three ways, stay fail-closed, and let the unreadable count become the visible migration backlog rather than masquerading as broken subjects.

Genuine failures20
Unreadable dialect — emitter needs migration11
No verdict anchor at all1
Share of "failures" that were never failures35%

This is not cosmetic. A rollup that cries wolf on DONE trains every reader to discount it — which is precisely how a genuinely red gate goes unnoticed. One had: the gate carrying the escaped mutant was sitting at 8/10 RED, with the exact teeth that named the defect, and nobody was looking.

A gate nobody runs is a comment.

05

The coverage arithmetic

That prompted the obvious question: how many gates are being watched? The answer is worse than the ledger recorded.

PopulationCountShareMeaning
Authored gate sources3,054Tests somebody wrote and believed in
With a compiled binary654214‰Everything else cannot run at all
In the rollup roster10534‰Everything else can be red in silence
"An unbuilt gate is indistinguishable from a passing one in every report that mentions it." — the census organ's own envelope.
The blind spot is self-reinforcing

The roster is derived from status logs, and a status log only exists once a gate has already run. So a gate that has never run has no log, never enters the roster, and is therefore never run. The mechanism that is supposed to notice non-green gates structurally cannot see the ones that have never reported.

A bounded sweep answered the follow-up — are the unbuilt ones broken, or merely never compiled?

Attempted25
Compiled11
Failed to compile14
Compile rate over attempted440‰

Roughly half do not build. They are not dormant, they are rotten — and the sweep's own envelope is careful that a partial sweep never implies the whole, so this is a sample, not a corpus verdict.

06

From a leaked page to a timed-out page load

Mid-session the operator reported that nishifamily.com/writer was timing out, and API calls began returning 503 upstream backend gave no response inside the edge window. It hit a fast tool too, so it was never the long-call timeout. The measurement:

Swap total24,252,332 kB
Swap free1,488,852 kB
Memory available539‰ — healthy
Load, 20 CPUs3.12 — idle
Swap consumed94%

Free RAM and idle CPUs, with roughly 22.7 GB of anonymous pages parked in swap. A serving daemon whose pages are swapped out must fault them back from disk to answer; under the I/O of a concurrent compile sweep that exceeds the edge's read window. The edge was not broken and the backend was not down — both were waiting on page-ins.

Link 1The leak epidemicOpen

16,942 unbalanced sys_mmap sites across 13.6% of all functions. The leak is everywhere, so it is nowhere in particular.

Link 2A fix that never flippedFixed

The worst offender's leak was already fixed in source. The long-running daemon never restarted onto it, so it kept executing the old image.

Link 3Swap exhaustionRelieved

Rebuild, promote, force the flip. Swap 938‰ → 822‰; free 1.49 GB → 4.29 GB.

Link 4The symptomCleared

/writer went from timing out to 200 in 34–52 ms, six probes of six.

Green fixed and verified · Amber watched · Red still open

The instrument that already knew

A resource monitor existed, was reporting verdict=RED sev=2, and named the offender by name. It printed to stdout only. The roster is derived from status logs — so it had no log, no roster row, and no reader. The organ written because "the single most destabilising condition on the platform was invisible to every instrument we owned" was itself invisible. It now writes a log and sits in the roster.

07

The governor that was right every minute

The estate's resource governor is genuinely good: it meters resident memory plus swap rather than address space, governs every process by a default cap with auditable exemptions rather than a hand-written watch list, and acts on forecast time-to-breach while headroom remains. It runs every sixty seconds. During all of this it scanned 504 processes and reported acted=0.

It was correct every single time. Three things defeated it anyway.

The cap was the consent

Policy row
cap nx_seed_announce_all 8 — an 8 GB grant above the 6 GB default.
Reality
The daemon's own source comment records it reaching 8.8 GB. It sat at 6.4 GB, legally.
Law
Raising a cap to silence a governor is not governance, it is consent. Lowered to 2 GB.

Rate is not accumulation

Rule shape
Every acting branch requires one process to be over its own cap.
Reality
Seventeen suspects, each individually legal, summing to the whole swap file.
Law
Pressure without a named culprit produces no action. Forecasting a future breach does nothing about memory already in swap.

The third is the one worth publishing. A new aggregate regime was added — when scarcity is measured and enough suspects exist to explain it, name the largest non-exempt holder and recycle it, even though it is individually legal. On its first live run it nominated a process that should have been structurally invisible to it: the supervisor, which is explicitly exempt. That impossible result was a real bug:

if g_streq(key, "exempt") == 1 { ... conf[le] = 0 as u8; ex_off[n_ex] = conf + vstart ... }

The needle is terminated at end of line, so exempt nx_hostctl # the supervisor itself: killing the guard... produced a needle containing the entire comment — which can never match a command line. Four of five exemptions were silently inert. Among them sites.elf, the edge that serves every domain, one over-cap sweep away from being terminated by its own governor.

Proven by the change in nominee

Before the fix the aggregate named the supervisor at 5,243 MB. After it, the same sweep names a smaller process at 4,702 MB — the larger one correctly excluded. That asymmetry is the proof the exemption now binds.

And the constraint that shaped the design

The estate had already banked the right law: never terminate what nothing will restart — a death-decider that outruns its respawner turns a leak into an outage. Coverage-by-default is right for detection and wrong for destruction, so the two are now split. Every non-exempt holder can be named; only a process explicitly declared restart-safe can be recycled. An unguarded nominee is reported loudly, every sweep, and acted on never.

08

What kept going wrong underneath

Five separate times this session, a text search produced or nearly produced a false conclusion. Once it reached a filed claim that had to be retracted: a grep scoped to *.conf reported that the resource governor was unscheduled. It runs every minute — from a crontab, which has no extension.

  • Extension-scoped search cannot see configuration that isn't named .conf.
  • Budget-truncated scans returned partial=1 twice, silently incomplete.
  • Capture caps cut a coverage envelope off mid-stream, making absence unreadable.
  • The estate's own search engine, asked for a symbol that exists in its source tree, returned arXiv papers and the Library of Congress — it indexes the external web and not itself.

A grep answers "where did I look?" It never answers "what exists." An index answers the second, and only the second makes absence-of-evidence a claim you are allowed to file.

The capability catalogue tells the same story from another angle: 855 primitives registered, 121 typed and 734 generic. The estate has the parts and has largely lost the map.

09

Laws banked

A critical section made safe against your own control flow is not safe against process death.

Fail-safe by finishing is not fail-safe by construction. Any tool that edits source in order to test it is a writer, and every writer needs crash recovery.

A differential probe hand-copied from its subject is evidence only if the copy is proven character-identical.

The copy silently corrected the defect, so four rounds of it exonerated the innocent and never indicted the guilty character.

An instrument that cannot read its subject must say so, not indict it.

A success vocabulary extended by hand for every dialect is an unbounded obligation, and until someone notices it the headline number lies pessimistically.

Pressure without a named culprit produces no action.

A governor that only asks "is any one process too big?" is blind to death by a thousand cuts — which is the shape a leak epidemic actually takes.

Never terminate what nothing will restart.

Coverage-by-default is correct for detection and wrong for destruction. The inverse of a blind allow-list is not an unconditional licence to kill.

An instrument that reports something impossible has found a real bug — in itself or in what it reads. Never explain it away.

The aggregate nominating an exempt process is what exposed four inert exemptions, including the live edge.

An absence-of-evidence claim must enumerate its search space before it is allowed to stand.

A grep scoped by file extension cannot see a config that isn't named .conf. That one reached a filed claim and had to be retracted.

10

What is still open

  1. Point the search index at ourselves. The estate is unsearchable to itself; every capability question currently degrades to a grep, and greps here have a measured false-conclusion rate.
  2. Randomised differential testing for the compiler. With no oracle, "the compiler is wrong" is unfalsifiable in practice — which is exactly how it survived four rounds at sev-9. The same false accusation had in fact been filed once before, from an entirely different real cause.
  3. A delta-debugging reducer. The four rounds were manual minimisation, done by hand-copying. A mechanical reducer only ever deletes from the real file and re-tests, so it structurally cannot make that mistake.
  4. Sandbox the mutation root. No industry mutation tester mutates the source of truth; they use in-memory bytecode or a sandboxed copy. The write-ahead record makes our approach survivable — a sandbox would make the class impossible.
  5. Drain the unbuilt-gate queue in paced batches. Roughly half do not compile, and the sweep that measures them is itself what flattened the edge, so pacing is part of the work.
Honest envelope

The compile-failure rate is from a 25-target sample, not the 2,400-target population. The leak-suspect count moves between sweeps as processes come and go. The swap relief is measured and real, but the underlying leak epidemic is untouched — what changed is that it is now visible and the governor can finally name it.

Every number on this page was read back from a live run, not from a plan. Ledger rows 1786036823 · 1786036965 · 1786036991 · 1786038102 · 1786038721 · 1786040638