ENGINE EVIDENCE 2026-07-25 · sovereign NishiLang · measured live on nishifamily.com

Impact-ordered postings: the candidacy cap now keeps the best documents, not the first ones

Every per-term read of the inverted index is capped (512 postings per term) so a query over a common word stays bounded. Until today that cap truncated in ascending document-id order — whichever pages happened to be indexed first claimed the slots. A page with the strongest term evidence (an entity's profile page, dense in its own name) could sit past the cap and be unrankable at any score, because it never even became a candidate.

The fix is the classic WAND/impact-ordering move, done integer-only: at index-build time each term's postings are pre-selected top-K by term frequency into a compact sidecar (NXW1 .imp, 32 MB beside a 317 MB corpus segment). At query time candidacy reads at most K entries per term per segment — the highest-evidence documents, with their tf carried along — instead of decoding the whole postings list. Cost per term drops from O(total postings) to O(K); the cap keeps the best.

Measured, same day, same corpus

Why "london metropolitan university" is the signature win: "london" and "university" are common terms whose caps used to admit an arbitrary 512 documents each. Impact order admits the pages densest in those terms — the university's own pages — so the right site finally out-competed the noise.

How it works (integer-only, crash-safe, reversible)

.imp layout   "NXW1" | nterms | per-term offsets | per term: k, then k × (doc, tf) varints
selection     build-time top-K by tf via histogram threshold  (K = 512, tf clamped to 1023 buckets)
read path     per segment: ordinal lookup → ≤K entries → live-doc currency check
              → global counting-sort by tf (strict impact order) → candidates + tf, zero doc reads
fallback      any live segment without .imp ⇒ the exact legacy path (all-or-nothing, gate-proven)
upgrade       additive-only: ss_write_seg emits .imp on every new segment; nx_seg_imp_build (MCP tool)
              rebuilds sidecars for an existing shard in place — 1.4 s for the full live corpus,
              no recompaction, no existing file touched

Everything is proven at three levels before it serves: the seg-store gate grew four discriminating teeth (impact candidacy under cap · true tf values · absent-sidecar fallback · stale-version currency, 9/9 green), the 36-tooth search-module regression gate stayed green, and the deploy went through the never-brick promote-with-rollback watchdog (DEPLOYED-GREEN, 14/14 services healthy).

Why this is the scale rung

Serving is already disk-bound (mmap page-cache serving, shipped earlier this month). With impact-ordered candidacy, per-query index work is bounded by K × query terms — independent of how large a term's postings grow. That is the property that lets the same code ride a corpus 10–100× larger on the same box, and shard cleanly across boxes after that:

Round 2, same day: iterate and learn

Hardening: every organ that writes search segments was rebuilt so new segments always carry the sidecar (crawler, PageRank builder, on-demand ingest), and the daemon's 16-tooth serve gate — red since an earlier scope-default change — was repaired to 16/16 green, restoring the full regression fence.

A refuted experiment, kept honest: we then tried feeding the sidecar's true full-document tf into the BM25 scorer (replacing an 8 KB scan-capped count). The judged ruler said no — MRR@10 fell 701 → 566, because full-document tf rewards long keyword-dense boilerplate: a real-estate listing farm out-scored a university homepage on its own name. The scan cap turns out to double as a head-of-document quality prior. The change was reverted (verified by bytes on disk and by the ruler returning to exactly 701), and the lesson is now part of the design: raw tf must not leave candidacy; the scoring-grade signal belongs to learned-sparse impact weights, computed at ingest. This is what ruler-gated shipping means — the number decides, in both directions.

Round 3: "when do we actually get more than a few hundred results?"

Fair question, and the answer turned out to be two separate defects wearing the same mask — one honesty bug and one coverage regression. Both are fixed; the counts below are live.

Defect 1 — the total was never the match count. Every query reported ≈100–130 "matched" no matter what you asked, which is suspicious the moment you notice it never varies. The reported total was being recomputed after the two-stage retrieval shortlist — so it displayed the shortlist size (offset + page + margin ≈ 128), not how many documents actually matched. Now, when candidacy is truncated, the engine reports the store's own exact per-term document frequency, which is a true lower bound on the match count — and downstream drops (zero-score, duplicate-URL) subtract from it instead of resetting it. When candidacy is not truncated the exact walk count is still used, so small collections stay precise to the document.

The truncation signal is the reader's own knowledge — the impact sidecar reports when it capped — not a derived guess. An earlier attempt used "stored document-frequency > documents returned" as the trigger, which false-positives on any shard holding re-indexed pages (the write-time count includes superseded versions). The regression gate caught it, and the flag now comes from the code that does the truncating.

Defect 2 — the corpus itself had shrunk. A rebuild the previous day left the live index pointing at a ~20,000-document segment while the 150,000-document one sat on disk, unreferenced. Restored by building its impact sidecar (4.0 seconds for 855 MB of index — no recompaction) and committing all three segments to the index manifest. The judged ruler held at MRR@10 705 across the restore: 7× the corpus, zero ranking regression.

A second refuted experiment. With a real corpus underneath, we raised the per-query depth caps 512 → 2048 to page deeper. The ruler said no again — 705 → 534, and latency tripled. The reason is precise: the first-stage shortlist ranks candidates by summed term rarity without term frequency, so quadrupling the pool dilutes it — nsf.gov fell out of the shortlist entirely and stopped being rankable. Reverted. Deeper paging is a real rung, but it needs a frequency-aware first stage (the sidecar already returns the data), not a bigger cap.

Where the ceiling actually is: the index holds ~150,000 pages, so a broad query legitimately matches tens of thousands. Growing that is throughput, not architecture — the bulk ingest lane is built and resumable at roughly 5,500 pages per Common Crawl archive file, and a crawl publishes 100,000 such files. One million pages is ~180 of them: days of polite fetching, not a rewrite. The engine work that had to come first — impact-ordered candidacy, bounded per-query cost, sidecars on every writer — is what makes that ingest safe to run.

Round 4: chasing the depth limit to its actual cause

Round 3 ended with a hypothesis rather than an answer: deeper paging failed because the first-stage shortlist ranks candidates by term rarity alone, ignoring how often each term actually appears. That is a testable claim, so we tested it — by building the missing piece and re-running the same experiment.

The first stage now weights each matched term's rarity by a saturating frequency factor taken from the impact sidecar: a page mentioning a term five times scores about twice a page mentioning it once, and a page mentioning it a hundred times scores barely three times — never a hundred. The saturation is deliberate and is exactly the lesson from the earlier refutation: frequency may decide which documents get fully scored, never how highly they score.

The hypothesis was right, and the feature still didn't ship at depth. The frequency-aware first stage recovered 534 → 648 — two-thirds of the collapse — which confirms the diagnosis was correct. But 648 is still worse than the 705 you get at the shallower depth, so deeper paging remains off. Being right about the cause is not the same as being done, and the ruler doesn't award partial credit.

What that third data point bought is the real cause, now isolated: the shortlist handed to full scoring stayed fixed at 128 documents while the candidate pool grew fourfold. The same-sized sieve simply has to discriminate four times harder. Deep paging needs a wider shortlist — or a second stage cheap enough to score every candidate — not a bigger candidate cap. That is a different, more honest piece of work, and it's now the filed next rung rather than a guess.

The frequency-aware first stage was kept even though it changed nothing at the shipping depth (705 → 705). It is measured-neutral, gate-green, and it is precisely the slot that learned-sparse weights plug into later — the weight simply becomes model-computed instead of a raw count. A neutral change that unlocks the next rung is worth keeping; a negative one never is.

A note on how the revert was verified: rolling the two constants back produced a binary whose hash was byte-identical to the pre-experiment build. Not "close" — identical. Hashing the staged artifact against the live one before every deployment now catches both stale artifacts and proves a revert is a true revert.

Round 5: three knobs that only work together

Round 4 left a specific accusation: the shortlist handed to full scoring stayed fixed at 128 while the candidate pool grew. So we widened it — first on its own, at the shipping depth, to keep it to one variable.

On its own it did nothing. Scoring all 512 candidates instead of 128 left the ruler at exactly 705 and cost ~60% more latency. Useful to know: at the shipping depth the cheap first stage was never actually losing anything — it was already picking the right 128. A wider sieve on a pool that fits is just more work.

Which left one untested combination — and it is the one that wins:

MRR@10 721 — a new high — and hit@10 rose 7 → 8: a query that had never surfaced its answer in the top ten now does. Warm latency stayed acceptable (180–290 ms). Every one of the three changes, raised alone, measured neutral or negative. Raised together they beat the previous best. They are not three knobs; they are one decision, and the code now says so where the constants are declared.

That is the whole value of keeping the refutations. Two of these rows are experiments we shipped and withdrew; a third was a change we kept precisely because it measured neutral. Had we deleted those results as failures, the winning combination would have looked like a lucky guess instead of the last cell of a grid. The earlier "depth hurts quality" conclusion was not wrong — it was incomplete, and the ruler was the only thing that could tell the difference.

One repair worth recording: rewriting those source comments through a Windows shell silently corrupted the file's UTF-8 and prepended a byte-order mark. It compiled and gated clean, so nothing shipped broken — but the fix was verified the same way everything else here is: the repaired source rebuilt to a binary byte-identical to the one already running, proving the change touched nothing but comments. Source files here are ASCII-only now.

Round 6: we widened the ruler, and it indicted us

Every decision on this page rested on a judged set of nine queries. The runs are deterministic, so those numbers were real — but nine queries is a narrow sample of what people actually ask, and a change can flatter those nine while hurting everything else. So before building the next feature, we strengthened the instrument: 27 judged queries across five labelled classes, with the original nine kept verbatim so older numbers stay comparable.

Two rules made it a ruler rather than a mirror. Each expected answer is the objectively canonical site for the query — an official homepage, the subject's encyclopedia entry — never whatever this engine happens to return today, because grading against your own output measures nothing. And the seventeen additions were held out: none were consulted while tuning any of the five rounds above.

721 on nine queries; 316 on twenty-seven. The old set was largely built from queries the engine already handled well, so it was telling us the truth about those nine and nothing about the rest. That is worth stating plainly rather than quietly re-baselining: the five earlier results stand as within-instrument comparisons, but their scope was overstated, and this page said so as soon as we could measure it.

And the failure is not the one we expected. The World Health Organization, the European Space Agency, the IRS, kernel.org, python.org, and Wikipedia's article pages are all simply absent — a 150,000-page sample of Common Crawl contains almost none of the web's canonical sites. The engine cannot rank a page it does not hold. The multilingual class scoring zero is the cleanest proof: that tokenizer is gate-proven and works — there is nothing in the index for it to find.

So the build order just changed, on evidence. The obvious next rung was smarter ranking (learned-sparse term weights). The measurement says that would be optimising relevance over a corpus that cannot answer the questions — measurement theatre. The next rung is ingest breadth: seed the canonical-site set directly, resume bulk crawl ingestion, then re-run this ruler and let it re-rank the work again.

The bench now reports RED and we are leaving it that way. The pass mark stays where it was; lowering a threshold to turn a light green is how an instrument stops being one. The number to beat is 316, and the nine-query 721 is retired rather than quoted. One afternoon spent on the ruler reversed a whole build plan — which is the argument for sharpening your instruments before trusting them.

Round 7: the duplicate we didn't build

The measurement said coverage, so the plan was a seeding tool: a list of canonical sites, fetched and indexed. Before writing it we checked whether the capability already existed. It did, and it was better.

The crawler already in the tree keeps its frontier inside the search index itself — every URL is a row marked pending or done — so a run picks up where the last one stopped, banks newly discovered links, and marks fetched URLs done even when they fail, so a dead link can never wedge the loop. It already had adaptive per-host pacing with backoff, near-duplicate detection, JavaScript rendering for pages that need it, and link filtering. The tool we were about to write would have been a worse version of one paragraph of that — and, critically, it could only ever have fetched homepages. Homepages do not answer "what causes earthquakes"; article pages do, and only a crawler that follows links reaches them.

So the work became fourteen lines instead of a new tool: a scheduled job that runs the existing crawler detached, using a spawner that was already generic. The search pipeline now has all three legs reachable through the API — coverage, authority, and storage — each one deferring politely if another is mid-run.

Round 8: two dead ends, and why the response never counts

Wiring that job through the management API took two wrong turns worth recording, because both were failures of trusting what something told us rather than checking what it did.

The documented restart command for the management service returns "unknown service" — it was never restartable that way. Our own notes said otherwise. Worse, the first attempt ran through a client whose reply was dropped in transit, so that rejection was invisible: the operation looked like it might have worked. And the API's rejection message lists an allowed set that is simply out of date — two jobs that are genuinely permitted don't appear in it. Reading policy off an error message, or success off a response that may never arrive, is how you end up confidently wrong. The correct path was the deploy route, where the control plane replaces and restarts itself; its connection failure is the expected signature of exactly that, and the supervisor had it back in five seconds.

One technique made the whole thing safe: before promoting anything into the control plane, we searched the built artifact for the new setting and confirmed the running one lacked it. That proves the build is genuinely yours and the live one genuinely older — at zero risk, without deploying. The job is now live and verified the only way that counts: by invoking it and watching the crawl start.

A correction to our own record, from the same afternoon: we had written that when the edge times out on a long request, the work still completes on the server. Measurement says otherwise — 44 fired requests produced zero indexed pages, because the process is reaped when the connection tears down. That note has been fixed rather than left to mislead the next session. It is also the whole argument for the detached job: it is not a convenience, it is the only mechanism that actually survives.

Round 9-13: why the crawler only ever fetched one page

With the coverage job finally reachable, we ran it — and it fetched exactly one page, then stopped. The process stayed alive. The log froze. Nothing was indexed. Chasing that down took four rounds and produced two fixes, one refuted theory, and one finding that reaches well past this crawler.

First theory: memory. Wrong. The crawler was still loading the entire search index into memory on open, while two sibling jobs had been converted months earlier to map it on demand — a conversion this one was simply left out of. That was a genuine latent defect and it is now fixed. It did not cure the stall. The converted build hung in exactly the same place, which is worth stating plainly: shipping a correct fix and fixing the bug are different events, and only measurement tells them apart. (The lesson kept: when a sweep converts a class of call sites, enumerate the class and check every member — the one that gets missed is the one that bites months later.)

Second theory, from reading the code rather than guessing: the JavaScript interpreter has no execution budget. Searching the whole 482 KB evaluator for any notion of a step limit, fuel counter, deadline or watchdog returns nothing. The crawler hands it the scripts from every page that has any — and the seed page is dense with them. One page with a long-running or non-terminating script therefore stops the crawler indefinitely. That is not merely a bug; it is a hostile-input property: a single crafted page could wedge any headless consumer by design. It is filed at high severity for the team that owns the interpreter, with the fix shape specified — a step counter plus a wall-clock deadline, and a distinct "budget exhausted" result so callers fall back to the raw page.

We did not fix it here, because that interpreter belongs to another workstream and duplicating their work is its own kind of damage. Instead we defended at our own boundary: page hydration is now off by default, behind a named switch carrying its own justification. The reasoning is an invariant, not a preference — hydration is an enhancement gated behind an unbudgeted interpreter, and a crawler that indexes plain HTML is strictly better than one that wedges on page one. Link discovery and text extraction never needed it.

The crawl now reaches real, varied sources — university admissions, NASA, GitHub, Creative Commons, Wikibooks in five languages, the Internet Archive — which is exactly the breadth the coverage measurement said was missing.

Two process notes, both corrections to ourselves. We built the job's "already running" check as a simple is-it-alive test, which meant a single hang would have wedged the coverage leg permanently with no way to clear it. That was our own defect; the check now judges by progress, and it cleared the wedge twice without anyone asking it to. And we twice mis-described the symptom — first as a hang, then over-correcting to "merely slow" on the strength of a log that had grown. It had grown because the supervisor had quietly restarted the job and a new process wrote those bytes. When a number moves between two observations, confirm it was the same process that moved it. Attribute progress to a process, not to a filename.

Round 14-15: the engine works; the number didn't move, and we said so first

With hydration disabled the coverage job ran to completion for the first time:

The score did not move, and that is the result we predicted in writing before running it. 113 pages onto a 150,000-document index is a 0.075% change, and this crawl started from an encyclopedia hub while the judged queries ask for canonical sites — a health agency, a tax authority, a kernel archive. Almost nothing it fetched is what the measurement asks about. Recording that expectation before taking the reading matters: a prediction that lands means the model of the gap is right, and it makes over-claiming after the fact impossible.

So the honest summary of this round is: a broken engine now works. Coverage itself is not improved in any way the instrument can see, and saying otherwise would be false. The next lever is aiming it — seeding the crawl from the canonical-site list rather than an encyclopedia hub, and letting it compound across runs. Navigational queries should move first; natural-language answers live deep in article pages and will take many more passes.

One more thing worth recording, because it nearly became a wrong diagnosis. Midway through, every route into the system started returning a fallback page — three separate transports, including the one our own notes call deterministic. The tempting label was "flaky network." Reading the control-plane log instead showed a colleague rolling back a bad deployment of the shared front-end proxy — twice — and swapping the supervisor, during exactly that window. Nothing was flaky; the front door was being rebuilt while we tried to walk through it. "Flaky" is usually a placeholder for a log nobody read. No fault was filed, because a colleague reversing their own bad deploy is the safety machinery working correctly.

Honest open gaps

• Semantic ceiling: "how to lower blood pressure" still loses to token-matching pages — measured and classified; the next quality rung is learned-sparse impact weights, which reuse this sidecar.
• Entity ranking: the profile page ranks 4, not 1 — candidacy is fixed; an entity-page prior is a separate ranking rung.
• Coverage: two ruler queries still miss because those pages aren't in the index yet — crawl breadth, not ranking. ~150,000 pages today.
• Cold-index latency: the first queries after a restart take ~3–4 s while the 855 MB index pages into memory, settling to ~170 ms warm. Page-fault bound, not compute bound — the fix is a compact per-document sidecar that stays resident, not more CPU.
• Deep paging: RESOLVED in round 5 — 2048 candidates per term with a matching shortlist, +16 MRR. The next depth step will need the same three-way check, not a single knob.

Method note: every number above came from the live serve through the public edge — the judged ruler writes durable evidence server-side, the before/after ran the same day on the same corpus, and the confounders found (a corpus rebuild that moved the baseline 640→625, and a second that had swapped the live index down to 20,000 pages) are stated rather than hidden. Two changes were shipped and then withdrawn because the ruler fell — that is the mechanism working, not a setback. Ratchet: MRR@10 316 on the 27-query instrument (the 9-query 721 is retired, not comparable) on the ~150,000-page index — raise, never lower.