code wiki / (root) / nx_bm25_lib.nx

nx_bm25_lib.nx

buildroot/runtime/nx_bm25_lib.nx

8461 B232 linesdepth 8pulls 13 transitivereach 1 importersview sourcekind librarytopic bm25
docsdependenciesstructsconstsfunctions

about

nx_bm25_lib.nx -- BM25 lexical retrieval + RECIPROCAL RANK FUSION, in pure integer fixed point. WHY THIS, WHY NOW (July 2026 SOTA, researched): every credible legal-retrieval stack this year is HYBRID -- a dense vector retriever AND BM25, fused with Reciprocal Rank Fusion, then reranked by a cross-encoder. Neither leg alone wins: vectors catch paraphrase, BM25 catches the exact defined term ("Force Majeure Event", a party name, a section cross-reference) that an embedder blurs away. In contracts the exact term is very often the whole question, which is why the lexical leg never gets dropped even in fully neural stacks. This builds the leg that needs NO MODEL, plus the fusion that any future embedder plugs straight into. The banked house order is embedder -> cross-encoder -> grep-tools -> graph LAST; BM25+RRF is the skeleton those slot into, and it is useful on its own from day one. u2605NO FLOAT. BM25 needs a logarithm and a saturating ratio, both normally float. Everything here is integer fixed point in MILLI units (1000 = 1.0), including a bit-position natural log with linear interpolation of the mantissa. Ranking only needs ORDER preservation, and integer fixed point preserves order exactly while staying reproducible bit-for-bit across machines -- a float BM25 can reorder two near-tied clauses depending on FMA and compile flags, which makes a retrieval regression impossible to reproduce. Determinism is worth more here than the third decimal place. u2605IDF IS THE HALF PEOPLE DROP. Scoring on raw term frequency ranks a clause highly for containing "the" fifty times. IDF is what makes a rare defined term outweigh filler, and BM25's length normalisation (b) is what stops a long clause winning purely by being long. Both are load-bearing. Parameters are DATA (rule 11): k1 and b as named constants in milli, the standard 1.2 / 0.75. license_tier: ORIGINAL LIB.

dependencies 1 imports · 1 importers

nx_clause_lib.nx nx_bm25_lib.nx nx_bm25_gate.nx

imports: nx_clause_lib.nx

imported by: nx_bm25_gate.nx

structs

none

consts

30const BM_K1_MILLI: i64 = 1200
31const BM_B_MILLI: i64 = 750
32const BM_RRF_K: i64 = 60
33const BM_NO_DOC: i64 = 0 - 1
34const BM_LOG2_1000: i64 = 9953

functions

39func bm_log2_floor(x: i64) -> i64
called by 1: bm_log2_milli
51func bm_log2_milli(x: i64) -> i64
65func bm_ln_milli(x: i64) -> i64
called by 1: main calls 1: bm_log2_milli
75func bm_ln_of_milli(x_milli: i64) -> i64
called by 1: bm_idf_milli calls 1: bm_log2_milli
84func bm_doclen(doc: *u8) -> i64
89func bm_tf(doc: *u8, term: *u8) -> i64
called by 2: mainbm_term_score_milli calls 1: cl_word_at
103func bm_df(lib: *i64, n: i64, term: *u8) -> i64
called by 2: mainbm_term_score_milli calls 1: cl_has_word
113func bm_avgdl(lib: *i64, n: i64) -> i64
called by 2: mainbm_score_milli calls 1: bm_doclen
126func bm_idf_milli(ndocs: i64, df: i64) -> i64
139func bm_term_score_milli(doc: *u8, term: *u8, lib: *i64, n: i64, avgdl: i64) -> i64
155func bm_score_milli(doc: *u8, query: *u8, lib: *i64, n: i64) -> i64
170func bm_best(query: *u8, lib: *i64, n: i64) -> i64
called by 1: main calls 1: bm_score_milli
187func bm_rank_of(query: *u8, lib: *i64, n: i64, idx: i64) -> i64
called by 1: main calls 1: bm_score_milli
210func bm_rrf_milli(rank_a: i64, rank_b: i64) -> i64
called by 2: mainbm_rrf_best
218func bm_rrf_best(ranks_a: *i64, ranks_b: *i64, n: i64) -> i64
called by 1: main calls 1: bm_rrf_milli