nx_wiki_bm25_search.nx
buildroot/runtime/wiki/nx_wiki_bm25_search.nx
about
nx_wiki_bm25_search.nx -- RANKED (BM25) wiki search that ALSO searches the
no-link-rot archive, unifying the live wiki doc corpus with the
content-addressed archive blobs toward the Nishi Library.
THE UPGRADE (wiki R3): the wiki's prior onsite ranker (nx_search_onsite_engine)
scored by AND-presence + count-of-matched-terms -- it could not tell a
SUBSTANTIVE page from a thin one that merely mentions the term. This module
scores by what each document ACTUALLY CONTAINS via Okapi BM25 (term frequency
SATURATED by k1, LENGTH-NORMALIZED by b, IDF-weighted), so a page that mentions
"sovereign" five times ranks ABOVE one that mentions it once. That is real
relevance ranking, not presence.
REUSE, NOT REINVENT (the Library's BM25 engine):
runtime/nx_bm25.nx -- the canonical sovereign Okapi BM25 (Q16.16 via fx.nx),
genealogy robertson_sparck-jones_bm25. It shares the
SAME FNV-1a tokenizer as nx_search_inverted (the wiki
index), so wiki-doc scoring and index tokenization
agree by construction. nx_bm25_score() takes doc-text
pointers + lengths + query terms and writes a Q16.16
score per doc. We hand it the doc-store bodies and the
archive blob bodies, then sort by descending score.
THE ARCHIVE UNIFICATION (no-link-rot content is searchable too):
The no-rot archive (nx_wiki_archive.nx) stores each page's bytes under
wikiblob:<cid> in the seg_store at prefix knowledge/store/wikiarchive-. The
seg_store ALREADY term-indexes every value (nx_seg_store.nx ss_build_terms),
so ss_term(handle, term) returns the KEYS (wikiblob:<cid>) of archived blobs
whose body contains the term. We fold those archive hits in: for each blob
key found, recover its <cid>, fetch the body via war_get_by_cid, BM25-score
it in the SAME corpus, and merge it into the ranked list. So a term that
lives ONLY in the archive (never in the live doc store) is still found.
SEPARATION: this module owns BM25 ranking + archive folding. It does NOT own
HTTP/render (hub primitives) or the index lifecycle (the site builds the
store). It is a pure scorer/merger over caller-provided corpora.
IMPORTS: de-duped by module identity. nx_search_inverted imports syscalls.nx
which is an alias-stub that splices nx_syscalls.nx (path-dedup, no dup
symbols); nx_bm25 / nx_seg_store / nx_wiki_archive import nx_syscalls.nx
directly -> all resolve to the one canonical syscall shelf. Single import of
dependencies 5 imports · 2 importers
imports: nx_syscalls.nxnx_bm25.nxnx_seg_store.nxnx_wiki_archive.nxnx_wiki_index_builder.nx
imported by: _wiki_search_gate.nxnx_wiki_search_wiring.nx
structs
| 76 | struct NxWikiRanked |
consts
| 51 | const NX_WBS_OK: i64 = 0 |
| 52 | const NX_WBS_BAD_INPUT: i64 = 2700 |
| 53 | const NX_WBS_CORPUS_OVERFLOW: i64 = 2701 |
| 54 | const NX_WBS_TERMS_OVERFLOW: i64 = 2702 |
| 57 | const NX_WBS_MAX_CORPUS: i64 = 4096 // total docs scored per query (wiki + archive) |
| 58 | const NX_WBS_MAX_QTERMS: i64 = 32 // per nx_search query-term cap |
| 59 | const NX_WBS_ARCH_SCAN_CAP: i64 = 256 // seg_store manifest segment-scan cap |
| 60 | const NX_WBS_BLOB_PREFIX_N: i64 = 9 // strlen("wikiblob:") |
| 61 | const NX_WBS_CID_LEN: i64 = 69 // strlen("nxc1-" + 64 hex) |
| 62 | const NX_WBS_ARCH_HITS_CAP: i64 = 1024 // per-term archive key hits buffer |
| 65 | const NX_WBS_SRC_DOC: i64 = 0 // a live wiki doc-store rowid |
| 66 | const NX_WBS_SRC_ARCHIVE: i64 = 1 // an archive blob (body recovered by CID) |
functions
| 87 | func nx_wiki_ranked_init(r: *NxWikiRanked, cap: i64) -> i64 |
| 102 | func nx_wiki_ranked_count(r: *NxWikiRanked) -> i64 |
| 107 | func nx_wiki_ranked_rowid_at(r: *NxWikiRanked, i: i64) -> i64 |
| 114 | func nx_wiki_ranked_score_at(r: *NxWikiRanked, i: i64) -> i64 |
| 121 | func nx_wiki_ranked_kind_at(r: *NxWikiRanked, i: i64) -> i64 |
| 129 | func nx_wbs_bytes_eq(a: *u8, b: *u8, n: i64) -> i64 called by 1: nx_wbs_gather_archive |
| 155 | func nx_wbs_gather_archive(prefix: *u8, |
| 248 | func nx_wiki_bm25_rank(store: *NxWikiDocStore, archive_prefix: *u8, |