code wiki / wiki / nx_wiki_bm25_search.nx

nx_wiki_bm25_search.nx

buildroot/runtime/wiki/nx_wiki_bm25_search.nx

15568 B351 linesdepth 7pulls 12 transitivereach 7 importersview sourcekind librarytopic wiki
docsdependenciesstructsconstsfunctions

about

nx_wiki_bm25_search.nx -- RANKED (BM25) wiki search that ALSO searches the no-link-rot archive, unifying the live wiki doc corpus with the content-addressed archive blobs toward the Nishi Library. THE UPGRADE (wiki R3): the wiki's prior onsite ranker (nx_search_onsite_engine) scored by AND-presence + count-of-matched-terms -- it could not tell a SUBSTANTIVE page from a thin one that merely mentions the term. This module scores by what each document ACTUALLY CONTAINS via Okapi BM25 (term frequency SATURATED by k1, LENGTH-NORMALIZED by b, IDF-weighted), so a page that mentions "sovereign" five times ranks ABOVE one that mentions it once. That is real relevance ranking, not presence. REUSE, NOT REINVENT (the Library's BM25 engine): runtime/nx_bm25.nx -- the canonical sovereign Okapi BM25 (Q16.16 via fx.nx), genealogy robertson_sparck-jones_bm25. It shares the SAME FNV-1a tokenizer as nx_search_inverted (the wiki index), so wiki-doc scoring and index tokenization agree by construction. nx_bm25_score() takes doc-text pointers + lengths + query terms and writes a Q16.16 score per doc. We hand it the doc-store bodies and the archive blob bodies, then sort by descending score. THE ARCHIVE UNIFICATION (no-link-rot content is searchable too): The no-rot archive (nx_wiki_archive.nx) stores each page's bytes under wikiblob:<cid> in the seg_store at prefix knowledge/store/wikiarchive-. The seg_store ALREADY term-indexes every value (nx_seg_store.nx ss_build_terms), so ss_term(handle, term) returns the KEYS (wikiblob:<cid>) of archived blobs whose body contains the term. We fold those archive hits in: for each blob key found, recover its <cid>, fetch the body via war_get_by_cid, BM25-score it in the SAME corpus, and merge it into the ranked list. So a term that lives ONLY in the archive (never in the live doc store) is still found. SEPARATION: this module owns BM25 ranking + archive folding. It does NOT own HTTP/render (hub primitives) or the index lifecycle (the site builds the store). It is a pure scorer/merger over caller-provided corpora. IMPORTS: de-duped by module identity. nx_search_inverted imports syscalls.nx which is an alias-stub that splices nx_syscalls.nx (path-dedup, no dup symbols); nx_bm25 / nx_seg_store / nx_wiki_archive import nx_syscalls.nx directly -> all resolve to the one canonical syscall shelf. Single import of

dependencies 5 imports · 2 importers

nx_syscalls.nx nx_bm25.nx nx_seg_store.nx nx_wiki_archive.nx nx_wiki_index_builder.nx nx_wiki_bm25_search.nx _wiki_search_gate.nx nx_wiki_search_wiring.nx

imports: nx_syscalls.nxnx_bm25.nxnx_seg_store.nxnx_wiki_archive.nxnx_wiki_index_builder.nx

imported by: _wiki_search_gate.nxnx_wiki_search_wiring.nx

structs

76struct NxWikiRanked

consts

51const NX_WBS_OK: i64 = 0
52const NX_WBS_BAD_INPUT: i64 = 2700
53const NX_WBS_CORPUS_OVERFLOW: i64 = 2701
54const NX_WBS_TERMS_OVERFLOW: i64 = 2702
57const NX_WBS_MAX_CORPUS: i64 = 4096 // total docs scored per query (wiki + archive)
58const NX_WBS_MAX_QTERMS: i64 = 32 // per nx_search query-term cap
59const NX_WBS_ARCH_SCAN_CAP: i64 = 256 // seg_store manifest segment-scan cap
60const NX_WBS_BLOB_PREFIX_N: i64 = 9 // strlen("wikiblob:")
61const NX_WBS_CID_LEN: i64 = 69 // strlen("nxc1-" + 64 hex)
62const NX_WBS_ARCH_HITS_CAP: i64 = 1024 // per-term archive key hits buffer
65const NX_WBS_SRC_DOC: i64 = 0 // a live wiki doc-store rowid
66const NX_WBS_SRC_ARCHIVE: i64 = 1 // an archive blob (body recovered by CID)

functions

87func nx_wiki_ranked_init(r: *NxWikiRanked, cap: i64) -> i64
102func nx_wiki_ranked_count(r: *NxWikiRanked) -> i64
107func nx_wiki_ranked_rowid_at(r: *NxWikiRanked, i: i64) -> i64
114func nx_wiki_ranked_score_at(r: *NxWikiRanked, i: i64) -> i64
121func nx_wiki_ranked_kind_at(r: *NxWikiRanked, i: i64) -> i64
129func nx_wbs_bytes_eq(a: *u8, b: *u8, n: i64) -> i64
155func nx_wbs_gather_archive(prefix: *u8,
248func nx_wiki_bm25_rank(store: *NxWikiDocStore, archive_prefix: *u8,