Web Search and Crawling

Nishi vs the field — every Nishi cell is measured against real organ source at emit time; each gap names the watch contract that will close it.

Nishi vs Mojeek and Brave Search and Google and Marginalia

Layer 1 · Executive

Where we are. Measured 2026-08-19. BM25, inverted and positional indexes, stemming, integer dense retrieval, our own crawler and PageRank, extraction and Common Crawl ingest are present; the exceeds are sovereignty, neutrality by construction and a zero-JS deterministic SERP. The three absent rows are the field's number-one, two and three: a neural reranker (every rerank arm tried so far failed the pre-declared BEIR rule and stayed unwired, which is the referee working), JS-rendered crawling, and web-scale index size.

Where we need to go. Beat BM25 honestly on BEIR before wiring anything neural, render JavaScript pages into the crawl, then shard to web scale with published latency -- never trading the neutral, deterministic SERP for any of it.

The unit. 1 u = one measured session-leg. Calibration from landed rungs: the mangagen panel compositor went from existing substrate to shipped and live-verified in ONE leg (2026-08-13); the citations rung went from 3 to 55 domains in one leg across seven seats (2026-08-18); a greenfield engine with a bite-proven gate has measured 2 to 4 legs. Estimates recalibrate as rungs land and PR7 actuals write back.

Cost to relevance: 3 u. Through M0; the rerank rung is research with a hard accept rule.

Cost to reach: 5.5 u. Through M1.

Cost to scale: 8.5 u. Everything below.

13 of 16 capabilities measured|3 of them measured exceeds|3 open|coverage 812/1000|adoption 9 full / 4 partial

Layer 2 · Roadmap

Do this next — computed by the ranker, never chosen by a seat

Order from nx_compare_rank (nx_dr_ocm: (deficit + cost-of-delay + option + enables) x sponsor x self-sufficiency x momentum / cost). FINISH rows are rungs whose symbol is present but whose organ is short of full adoption: the cheapest closures on this board, listed before any new work. Stamp: # asof=1787883282 domain=search target_version=0.1 rungs=3 done=0 open=3 finish=0 ranker=nx_dr_ocm

#	Stage	Rung	Priority	Derivation
#1	0.1	Neural rerank that beats BM25 (R0) nr_rerank	233	v=7 m=1 c=30
#2	later	JS-rendered crawling (R1) hs_render	600	v=15 m=1 c=25
#3	later	Web-scale sharded index (R2) is_shard	200	v=6 m=1 c=30

Critical path — contract, done-rule, executor, cost

Rung	Closes with	Definition of done (pre-declared)	Executor	Est.
Neural rerank that beats BM25 (R0)	nr_rerank	A trained cross-encoder or dense reranker over the top-k; the accept rule is the lane's standing one: it must EXCEED the BM25 arm (321 permille, at the published baseline) on BEIR through nx_beir_eval on the same queries or it is not wired -- semppmi and RRF already failed this rule and stayed unwired	Local model	3 u
JS-rendered crawling (R1)	hs_render	The sovereign browser's JS execution renders SPA pages into the crawler; the six FETCH-EMPTY SPA records in the browser census flip to FETCHED; shared with the webscraping domain's headless row	Organ	2.5 u
Web-scale sharded index (R2) after R1	is_shard	Segment shards past ten million documents with published p50 and p99 query latency and a partition that sums; composes nx_web_shard_compact and the WAND impact postings already shipped	Organ	3 u

Milestones

Milestone	Rungs	Cumulative
M0 · Relevance	R0	3 u
M1 · Reach	R1	5.5 u
M2 · Scale	R2	8.5 u

Layer 3 · Engineering

How this is scored. Every Nishi mark is measured: the generator reads the real organ source on disk and requires the implementing symbol to exist (no self-grading). A watching tag names the organ and symbol contracted to close a gap — the mark flips itself on the next compare beat when that workstream ships, and the comparewatch- plane row flips with it. The flip is necessary, not sufficient: it proves the symbol exists, never that the capability is good. The bar is the rung's pre-declared done-rule, proven by its gate — a symbol shipped without the behaviour behind it is a defect, and the flip is exactly what makes that defect visible instead of quiet. Competitor marks record documented capability presence — presence, not depth or scale. Adoption is measured too: every measured row carries where its organ stands on the estate's ladder (source → built → promoted → registered → invoked; libraries by importer reach minus validation importers; gates by the execution surfaces that run them). A row is fully adopted only at the top of its ladder; anything short is tagged partial with the exact remedy, so a build nobody promoted can no longer read as shipped. Census stamps: importers asof 1787849099, gate census asof 1787855507 (unix seconds; -1 = census absent).

Capability matrix — measured against source

◉ leads / measured exceed● present◐ partial○ absent · click any capability for its evidence

Capability	Nishi	Mojeek	Brave	Google	Marginalia
BM25 relevance ranking Measured: `idf_q10` exists in `runtime/nx_intlog.nx`, verified at emit. Integer Okapi BM25 (idf_q10); every engine ranks [robertson2009] [thakur2021] Adoption: `LIB-WIRED importers=4 nonval=2` — fully adopted (top of its ladder).	●	●	●	●	●
Inverted index Measured: `nx_search_inverted` exists in `runtime/nx_search_inverted.nx`, verified at emit. Classic posting-list index Adoption: `LIB-WIRED importers=38 nonval=32` — fully adopted (top of its ladder).	●	●	●	●	●
Phrase / positional search Measured: `ss_phrase` exists in `runtime/nx_seg_store.nx`, verified at emit. Positional postings (NXQ1 sidecar) Adoption: `LIB-WIRED importers=441 nonval=321` — fully adopted (top of its ladder).	●	●	●	●	◐
Stemming / morphology Measured: `nx_stem` exists in `runtime/_hdl_build/nx_stem.nx`, verified at emit. Porter-lite query+index stemming [porter1980] Adoption: `LIB-WIRED importers=3 nonval=2` — fully adopted (top of its ladder).	●	●	●	●	◐
Dense / vector retrieval Measured: `nx_vec_index` exists in `runtime/nx_vec_index.nx`, verified at emit. Integer cosine k-NN; Google neural, Brave partial; Mojeek 0 to 3 corrected 2026-08-25 -- our OWN pinned mirror records Mojeek adding Semantic Search in 2024 while this cell still read No. Coded Part not Yes because the mirror names the feature without establishing it is dense retrieval [mojeek] Adoption: `SOURCE-ONLY` — PARTIAL: source exists, never compiled: /api/build it. adoption SOURCE-ONLY	●	◐	◐	●	○
Own independent crawler Measured: `nx_https_fetch_follow` exists in `runtime/nx_https_fetch_follow.nx`, verified at emit. Sovereign TLS crawler; all run their own crawl [mojeek] Adoption: `LIB-WIRED importers=232 nonval=211` — fully adopted (top of its ladder).	●	●	●	●	●
Web-graph authority (PageRank) Measured: `nx_pagerank` exists in `runtime/nx_pagerank.nx`, verified at emit. Integer power-iteration; Google invented it [brin1998] Adoption: `LIB-WIRED importers=2 nonval=1` — fully adopted (top of its ladder).	●	●	●	◉	◐
HTML content extraction Measured: `nx_html_to_text` exists in `runtime/nx_html_to_text.nx`, verified at emit. Tag-stripping renderer; the density-fit text the crawler now indexes is the webscraping board's bd_fit_text contract Adoption: `LIB-WIRED importers=56 nonval=41` — fully adopted (top of its ladder).	●	●	●	●	●
Common Crawl ingest (index-only) Measured: `nx_cc_ingest` exists in `runtime/_hdl_build/nx_cc_ingest.nx`, verified at emit. CDX + range-fetch, Memex-style; majors use own crawl Adoption: `REGISTERED-UNAUTHORISED` — PARTIAL: registered, no cap ever minted; no execution surface runs it either (clock, cron, daemon, roster, actlog and surfaced forks checked). adoption REGISTERED-UNAUTHORISED	●	○	○	◐	◐
Deep-research answer loop Measured: `nx_qabench` exists in `runtime/_hdl_build/nx_qabench.nx`, verified at emit. Fetch to read QA; Google AI Overviews, Brave answers; Mojeek 0 to 3 corrected 2026-08-25 -- our OWN pinned mirror records Mojeek launching Search Summary in 2024 while this cell still read No [mojeek] Adoption: `SOURCE-ONLY` — PARTIAL: source exists, never compiled: /api/build it. adoption SOURCE-ONLY	●	◐	◐	◉	○
Neural rerank / trained encoder Open — watching `runtime/nx_neural_reranker.nx` : `nr_rerank`, re-measured on every compare beat. Ship that symbol and this mark flips itself; the `comparewatch-` plane row flips with it. Cross-encoder rerank; Nishi's number-one gap [nogueira2019] watching nr_rerank	○	○	◐	◉	○
Web-scale index (billions+) Open — watching `runtime/nx_index_scale.nx` : `is_shard`, re-measured on every compare beat. Ship that symbol and this mark flips itself; the `comparewatch-` plane row flips with it. Mojeek passed 9 billion pages in 2025 per its own milestone timeline; Google trillions. CITATION CORRECTED 2026-08-25 -- the Mojeek half of this claim was marked to the BRAVE source, while our pinned Mojeek mirror carrying the actual figure was cited only on the crawler row. The Nishi figure this row used to assert was prose, never a measurement, and is withheld until measured [mojeek] [brave-independence] watching is_shard	○	●	●	◉	●
JS-rendered crawling Open — watching `runtime/nx_headless_scrape.nx` : `hs_render`, re-measured on every compare beat. Ship that symbol and this mark flips itself; the `comparewatch-` plane row flips with it. Headless render; Nishi static-HTML only watching hs_render	○	◐	◐	◉	○
Sovereign -- own language, compiler, TLS, index (zero third-party) Measured exceed: `ss_fold_cp` in `runtime/nx_seg_store.nx`, verified at emit. No Python/C/cloud beneath; Marginalia lean but on JVM/Linux [marginalia-about] Adoption: `LIB-WIRED importers=441 nonval=321` — fully adopted (top of its ladder).	◉	○	○	○	◐
Neutral by construction -- no ads, no profiling, down-rank not delete Measured exceed: `nx_anti_slop` in `runtime/nx_anti_slop.nx`, verified at emit. Quality filter demotes slop, never censors; Google ad-ranked + profiled [google-ranking] Adoption: `LIB-GATE-ONLY importers=1` — PARTIAL: imported only by validation organs (gates, tests, benches): wire it into a shipping program. adoption LIB-GATE-ONLY importers=1	◉	◐	◐	○	◐
Zero-JS, zero-tracker, integer-deterministic SERP Measured exceed: `idf_q10` in `runtime/nx_intlog.nx`, verified at emit. About 5 KB no-JS page, bit-reproducible results Adoption: `LIB-WIRED importers=4 nonval=2` — fully adopted (top of its ladder).	◉	○	○	○	○

On these two registers. Rows are declared in the domain's plan file and carry the debt id, which is the join key back to the sovereign debt plane — that plane, not this page, is the authority on state. Reconciling them automatically (the regen reading the plane and refreshing these rows) is a named, owed rung; until it lands, treat an id here as a pointer to look up, not a status to trust.

Honest verdict. The coverage above is capability presence measured against source — not depth, scale, or polish, where mature rivals may lead. Exceeds are claimed only where a mechanism backs them. Every open gap is a watch contract: it names the organ and symbol that closes it, and this page flips the cell itself when that workstream ships.

Person · product · place — not yet measured for this domain

Every compare carries this layer. Declare knowledge/compare/search.ppp (rows surface|nishi or c1..c4|label|url|connect naming OUR live surface and each rival's front door), run nx_ppp_probe domain search, and this section fills itself on the next beat: the same ruler on both sides — privacy and CX (third-party hosts, tracker classes, cookies, security headers), design and longevity (design hygiene, computed WCAG contrast, render-blocking resources, unsized media, script weight, theme and motion queries), findability (landmarks, skip link, on-site search, breadcrumb, headings, internal links).

References

Beyond a link list. Every reference below resolves twice — the publisher's copy and, where banked, the estate's own non-rottable library mirror with a content pin — and carries its evidence class plus the exact claim on this page it grounds. Keyed marks like [key] in the matrix notes jump here. A dash means honestly absent, never assumed.

[mojeek] Mojeek Ltd. About Mojeek -- building the world's alternative search engine (own crawler-built index, no tracking). Vendor page. publisher · read in our library knowledge/fetched/cmp_search_mojeek.html · pin hed8158b530335cd9b96dc99d4030dc3b1ccaa629487ea33ac4669cbf30347ce5 · accessed 2026-08-18 · vendor-docGrounds: The Mojeek column codes on "Own independent crawler", "Inverted index" and "Neutral by construction -- no ads, no profiling": Mojeek documents its own crawler-built index and no-tracking policy, so it is the closest independent-engine comparator on this page.
[brave-independence] Brave Software. Brave Search removes last remnant of Bing from search results page, achieving 100 percent independence (Brave blog, 2023-04-27). publisher · read in our library knowledge/fetched/cmp_search_brave.html · pin h84d5996dff2ebccbca747802072a4f6886434081742fbcd13f11d7706e6bd5e2 · accessed 2026-08-18 · vendor-docGrounds: The Brave column on "Own independent crawler" and "Web-scale index (billions+)": since April 2023 every web result Brave Search shows is served from Brave's own index with no Bing API calls, which is what the Yes codes in that column assert.
[google-ranking] Google. How Search Works -- Ranking results: meaning, relevance, quality, usability and context signals. Vendor page. publisher · read in our library knowledge/fetched/cmp_search_google.html · pin h11d58cb369e1aac147fe74a03237ed86dd992e621a2cd723c11785e090aece87 · accessed 2026-08-18 · vendor-docGrounds: The Google column: Best on "Web-graph authority (PageRank)" and Yes on "Dense / vector retrieval" -- Google's own description of ranking by meaning (neural language understanding) plus link-based quality signals is the reference those codes are graded against, and its ad-ranked profiled SERP is the foil for the "Neutral by construction" exceed row.
[marginalia-about] Lofgren, V. About Marginalia Search -- an independent open source (AGPL) Internet search engine operating out of Sweden; updated 2025-01-01. publisher · read in our library knowledge/fetched/cmp_search_marginalia_about.html · pin h385ed28b0c08c0fad81247bc7c3df300d35a566300b5c760a486ed51a147cb82 · accessed 2026-08-18 · vendor-docGrounds: The Marginalia column and the "Sovereign -- own language, compiler, TLS, index" note ("Marginalia lean but on JVM/Linux"): the page states the engine is independent, open source under AGPL at git.marginalia.nu, sets no tracking cookies and runs on about 200 dollars a month -- the lean-independent comparator for the zero-third-party claim, and the reason its column reads Part rather than No there.
[robertson2009] Robertson, S., Zaragoza, H. The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval 3(4), 2009. publisher · read in our library knowledge/fetched/cmp_search_robertson2009.pdf · pin h11a9ef825dfd3fe50c8621273d02ea26b6f502b60379db9519d6c5c10de4fd25 · accessed 2026-08-18 · published-paperGrounds: The "BM25 relevance ranking" row: Okapi BM25 (idf weighting, tf saturation k1, length normalisation b) is this framework; nx_intlog idf_q10 implements it in integer arithmetic, so the ranking function every column shares has one published definition to be graded against.
[thakur2021] Thakur, N., Reimers, N., Rueckle, A., Srivastava, A., Gurevych, I. BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models. NeurIPS 2021 Datasets and Benchmarks; arXiv:2104.08663. publisher · read in our library knowledge/fetched/cmp_search_thakur2021.html · pin hac20a9340bbf8531f35ed876a785ef0f3edd1e76fc658c4bc487c00f56a44487 · accessed 2026-08-18 · datasetGrounds: The measured bar behind the "BM25 relevance ranking" and "Neural rerank / trained encoder" rows: BEIR publishes per-dataset BM25 (Anserini) and cross-encoder rerank baselines; the estate's nx_beir_eval reproduces the BM25 arm at 321 permil against the published 320, and any rerank arm is wired only if it EXCEEDS that BM25 arm on the same queries (pre-declared accept rule).
[brin1998] Brin, S., Page, L. The Anatomy of a Large-Scale Hypertextual Web Search Engine. WWW7, Computer Networks and ISDN Systems 30, 1998. publisher · read in our library knowledge/fetched/cmp_search_brin1998.html · pin h5c4894d4604f5b5b475f0aaa6663ca716f351f3594b326f43d15dbbe1f1d0a5c · accessed 2026-08-18 · published-paperGrounds: The "Web-graph authority (PageRank)" row and its note "Google invented it": PageRank as iterated link authority is defined in this paper; nx_pagerank is the integer power-iteration of the same recurrence, and Google's Best code on the row is the inventor's.
[porter1980] Porter, M.F. An algorithm for suffix stripping. Program 14(3), 1980; the official Porter Stemmer page carrying the algorithm and reference implementations. publisher · read in our library knowledge/fetched/cmp_search_porter1980.html · pin h71c856c7b4ce4a5a03d38e9aa718e042b6e9f61a9504574382e84f16b0497b76 · accessed 2026-08-18 · published-paperGrounds: The "Stemming / morphology" row: nx_stem is a Porter-lite of this suffix-stripping algorithm applied at both index and query time; the reference page carries the canonical algorithm and test vocabularies a stemmer is graded against.
[nogueira2019] Nogueira, R., Cho, K. Passage Re-ranking with BERT. arXiv:1901.04085, 2019. publisher · read in our library knowledge/fetched/cmp_search_nogueira2019.html · pin h69d0628dfa9fe889ff88d3594bd9d2831207c9e9f452226ad3db643aa131dd04 · accessed 2026-08-18 · published-paperGrounds: The "Neural rerank / trained encoder" row (Nishi's number-one gap, _ABSENT_): a BERT cross-encoder re-ranking a BM25 candidate list is the method the Best and Part codes describe; the watch contract closes only when a sovereign reranker EXCEEDS the BM25 arm on the same queries.

generated by nx_swcompare_matrix (sovereign NishiLang organ) from knowledge/compare/search.matrix · every Nishi cell verified against organ source at emit time · watch cells re-measured on every compare beat · zero JS, zero trackers