nx_bm25_lib.nx
buildroot/runtime/nx_bm25_lib.nx
about
nx_bm25_lib.nx -- BM25 lexical retrieval + RECIPROCAL RANK FUSION, in pure integer fixed point.
WHY THIS, WHY NOW (July 2026 SOTA, researched): every credible legal-retrieval stack this year is
HYBRID -- a dense vector retriever AND BM25, fused with Reciprocal Rank Fusion, then reranked by a
cross-encoder. Neither leg alone wins: vectors catch paraphrase, BM25 catches the exact defined term
("Force Majeure Event", a party name, a section cross-reference) that an embedder blurs away. In
contracts the exact term is very often the whole question, which is why the lexical leg never gets
dropped even in fully neural stacks.
This builds the leg that needs NO MODEL, plus the fusion that any future embedder plugs straight into.
The banked house order is embedder -> cross-encoder -> grep-tools -> graph LAST; BM25+RRF is the
skeleton those slot into, and it is useful on its own from day one.
u2605NO FLOAT. BM25 needs a logarithm and a saturating ratio, both normally float. Everything here is
integer fixed point in MILLI units (1000 = 1.0), including a bit-position natural log with linear
interpolation of the mantissa. Ranking only needs ORDER preservation, and integer fixed point
preserves order exactly while staying reproducible bit-for-bit across machines -- a float BM25 can
reorder two near-tied clauses depending on FMA and compile flags, which makes a retrieval regression
impossible to reproduce. Determinism is worth more here than the third decimal place.
u2605IDF IS THE HALF PEOPLE DROP. Scoring on raw term frequency ranks a clause highly for containing
"the" fifty times. IDF is what makes a rare defined term outweigh filler, and BM25's length
normalisation (b) is what stops a long clause winning purely by being long. Both are load-bearing.
Parameters are DATA (rule 11): k1 and b as named constants in milli, the standard 1.2 / 0.75.
license_tier: ORIGINAL LIB.
dependencies 1 imports · 1 importers
imports: nx_clause_lib.nx
imported by: nx_bm25_gate.nx
structs
| none |
consts
| 30 | const BM_K1_MILLI: i64 = 1200 |
| 31 | const BM_B_MILLI: i64 = 750 |
| 32 | const BM_RRF_K: i64 = 60 |
| 33 | const BM_NO_DOC: i64 = 0 - 1 |
| 34 | const BM_LOG2_1000: i64 = 9953 |
functions
| 39 | func bm_log2_floor(x: i64) -> i64 called by 1: bm_log2_milli |
| 51 | func bm_log2_milli(x: i64) -> i64 |
| 65 | func bm_ln_milli(x: i64) -> i64 |
| 75 | func bm_ln_of_milli(x_milli: i64) -> i64 |
| 84 | func bm_doclen(doc: *u8) -> i64 |
| 89 | func bm_tf(doc: *u8, term: *u8) -> i64 |
| 103 | func bm_df(lib: *i64, n: i64, term: *u8) -> i64 |
| 113 | func bm_avgdl(lib: *i64, n: i64) -> i64 |
| 126 | func bm_idf_milli(ndocs: i64, df: i64) -> i64 |
| 139 | func bm_term_score_milli(doc: *u8, term: *u8, lib: *i64, n: i64, avgdl: i64) -> i64 |
| 155 | func bm_score_milli(doc: *u8, query: *u8, lib: *i64, n: i64) -> i64 |
| 170 | func bm_best(query: *u8, lib: *i64, n: i64) -> i64 |
| 187 | func bm_rank_of(query: *u8, lib: *i64, n: i64, idx: i64) -> i64 |
| 210 | func bm_rrf_milli(rank_a: i64, rank_b: i64) -> i64 |
| 218 | func bm_rrf_best(ranks_a: *i64, ranks_b: *i64, n: i64) -> i64 |