code wiki / (root) / nx_bm25.nx

nx_bm25.nx

buildroot/runtime/nx_bm25.nx

7806 B157 linesdepth 3pulls 3 transitivereach 53 importersview sourcekind librarytopic bm25
docsdependenciesstructsconstsfunctions

about

nx_bm25.nx -- production-grade ranked retrieval: Okapi BM25 (Robertson & Sparck Jones), integer-only. This is the HONEST upgrade over the team's first TF-IDF ranker, which lacked LENGTH NORMALIZATION: a long document that mentions a term once would out-rank a short focused document with the same per-length relevance. BM25 fixes that with (1) tf SATURATION (k1) -- the 10th occurrence adds less than the 2nd -- and (2) LENGTH NORMALIZATION (b) -- penalize long docs by dl/avgdl. The point of building the real standard (not a weak baseline) is to TRIANGULATE: nx_bm25_test asserts the team's integer ranking reproduces a float reference BM25's ranking, so the "exceed" is measured against the production formula, not cherry-picked. license_tier: ORIGINAL BM25(q,d) = sum_t idf(t) * tf(t,d)*(k1+1) / ( tf(t,d) + k1*(1 - b + b*dl/avgdl) ) idf(t) = ln( 1 + (N - df + 0.5)/(df + 0.5) ) = ln( (2N+2)/(2df+1) ) [always positive]

dependencies 2 imports · 28 importers

nx_research_extract.nx nx_syscalls.nx nx_bm25.nx nx_bm25_test.nx nx_bm25f.nx nx_book_search.nx nx_book_search_lib.nx nx_cms_search_gate.nx nx_crawl_bfs.nx nx_crawl_https.nx nx_crawl_polite.nx nx_crawl_run.nx nx_diora_live_reach.nx

diagram shows first 10 each side; +0 more imports, +18 more importers in the complete lists below.

imports: nx_research_extract.nxnx_syscalls.nx

imported by: nx_bm25_test.nxnx_bm25f.nxnx_book_search.nxnx_book_search_lib.nxnx_cms_search_gate.nxnx_crawl_bfs.nxnx_crawl_https.nxnx_crawl_polite.nxnx_crawl_run.nxnx_diora_live_reach.nxnx_diora_live_reach_guarded.nxnx_diora_live_serp.nxnx_dms_search.nxnx_docstore_test.nxnx_entity_discover.nxnx_gallery_serve.nxnx_library_search.nxnx_live_crawl.nxnx_live_https.nxnx_onsite_search.nxnx_ppmi_svd.nxnx_rank_fused.nxnx_research_digest.nxnx_research_digest_gate.nxnx_research_web.nxnx_search_cli.nxnx_search_neutrality_gate.nxnx_wiki_bm25_search.nx

structs

none

consts

17const BM_K1_X1000: i64 = 1200 // k1 = 1.2 (term-frequency saturation)
18const BM_B_X1000: i64 = 750 // b = 0.75 (length-normalization strength)
19const BM_MICRO: i64 = 1000000 // fixed-point scale for ln / idf
20const BM_MILLI: i64 = 1000 // fixed-point scale for the saturation factor

functions

23func bm_token_count(text: *u8, n: i64) -> i64
36func bm_ln_micro(num: i64, den: i64) -> i64
53func bm_idf_micro(N: i64, df: i64) -> i64 { return bm_ln_micro(2 * N + 2, 2 * df + 1) }
57func bm_sat_milli(tf: i64, dl: i64, avgdl: i64) -> i64
67func bm_df(ptrs: *i64, lens: *i64, N: i64, term: *u8) -> i64
74func bm_score(ptrs: *i64, lens: *i64, dls: *i64, N: i64, k: i64, avgdl: i64, qterms: *i64, nq: i64) -> i64
90func bm_best(ptrs: *i64, lens: *i64, dls: *i64, N: i64, avgdl: i64, qterms: *i64, nq: i64) -> i64
called by 3: mainmainls_best_bm25 calls 1: bm_score
117func nx_bm25_score(docs: **u8, dlens: *i64, n: i64, qterms: **u8, qlens: *i64, nq: i64, out: *i64) -> i64
138func bm_naive_tfidf(ptrs: *i64, lens: *i64, N: i64, k: i64, qterms: *i64, nq: i64) -> i64
149func bm_naive_best(ptrs: *i64, lens: *i64, N: i64, qterms: *i64, nq: i64) -> i64
called by 1: main calls 1: bm_naive_tfidf