nx_bm25.nx
buildroot/runtime/nx_bm25.nx
about
nx_bm25.nx -- production-grade ranked retrieval: Okapi BM25 (Robertson & Sparck Jones), integer-only.
This is the HONEST upgrade over the team's first TF-IDF ranker, which lacked LENGTH NORMALIZATION:
a long document that mentions a term once would out-rank a short focused document with the same
per-length relevance. BM25 fixes that with (1) tf SATURATION (k1) -- the 10th occurrence adds less
than the 2nd -- and (2) LENGTH NORMALIZATION (b) -- penalize long docs by dl/avgdl. The point of
building the real standard (not a weak baseline) is to TRIANGULATE: nx_bm25_test asserts the team's
integer ranking reproduces a float reference BM25's ranking, so the "exceed" is measured against the
production formula, not cherry-picked. license_tier: ORIGINAL
BM25(q,d) = sum_t idf(t) * tf(t,d)*(k1+1) / ( tf(t,d) + k1*(1 - b + b*dl/avgdl) )
idf(t) = ln( 1 + (N - df + 0.5)/(df + 0.5) ) = ln( (2N+2)/(2df+1) ) [always positive]
dependencies 2 imports · 28 importers
diagram shows first 10 each side; +0 more imports, +18 more importers in the complete lists below.
imports: nx_research_extract.nxnx_syscalls.nx
imported by: nx_bm25_test.nxnx_bm25f.nxnx_book_search.nxnx_book_search_lib.nxnx_cms_search_gate.nxnx_crawl_bfs.nxnx_crawl_https.nxnx_crawl_polite.nxnx_crawl_run.nxnx_diora_live_reach.nxnx_diora_live_reach_guarded.nxnx_diora_live_serp.nxnx_dms_search.nxnx_docstore_test.nxnx_entity_discover.nxnx_gallery_serve.nxnx_library_search.nxnx_live_crawl.nxnx_live_https.nxnx_onsite_search.nxnx_ppmi_svd.nxnx_rank_fused.nxnx_research_digest.nxnx_research_digest_gate.nxnx_research_web.nxnx_search_cli.nxnx_search_neutrality_gate.nxnx_wiki_bm25_search.nx
structs
| none |
consts
| 17 | const BM_K1_X1000: i64 = 1200 // k1 = 1.2 (term-frequency saturation) |
| 18 | const BM_B_X1000: i64 = 750 // b = 0.75 (length-normalization strength) |
| 19 | const BM_MICRO: i64 = 1000000 // fixed-point scale for ln / idf |
| 20 | const BM_MILLI: i64 = 1000 // fixed-point scale for the saturation factor |
functions
| 23 | func bm_token_count(text: *u8, n: i64) -> i64 |
| 36 | func bm_ln_micro(num: i64, den: i64) -> i64 |
| 53 | func bm_idf_micro(N: i64, df: i64) -> i64 { return bm_ln_micro(2 * N + 2, 2 * df + 1) } |
| 57 | func bm_sat_milli(tf: i64, dl: i64, avgdl: i64) -> i64 |
| 67 | func bm_df(ptrs: *i64, lens: *i64, N: i64, term: *u8) -> i64 |
| 74 | func bm_score(ptrs: *i64, lens: *i64, dls: *i64, N: i64, k: i64, avgdl: i64, qterms: *i64, nq: i64) -> i64 |
| 90 | func bm_best(ptrs: *i64, lens: *i64, dls: *i64, N: i64, avgdl: i64, qterms: *i64, nq: i64) -> i64 |
| 117 | func nx_bm25_score(docs: **u8, dlens: *i64, n: i64, qterms: **u8, qlens: *i64, nq: i64, out: *i64) -> i64 |
| 138 | func bm_naive_tfidf(ptrs: *i64, lens: *i64, N: i64, k: i64, qterms: *i64, nq: i64) -> i64 |
| 149 | func bm_naive_best(ptrs: *i64, lens: *i64, N: i64, qterms: *i64, nq: i64) -> i64 |