code wiki / _hdl_build / nx_bm25_naive.nx

nx_bm25_naive.nx

buildroot/runtime/_hdl_build/nx_bm25_naive.nx

5673 B120 linesdepth 3pulls 3 transitivereach 0 importersview sourcekind orphan librarytopic bm25
docsdependenciesstructsconstsfunctions

about

nx_bm25.nx -- production-grade ranked retrieval: Okapi BM25 (Robertson & Sparck Jones), integer-only. This is the HONEST upgrade over the team's first TF-IDF ranker, which lacked LENGTH NORMALIZATION: a long document that mentions a term once would out-rank a short focused document with the same per-length relevance. BM25 fixes that with (1) tf SATURATION (k1) -- the 10th occurrence adds less than the 2nd -- and (2) LENGTH NORMALIZATION (b) -- penalize long docs by dl/avgdl. The point of building the real standard (not a weak baseline) is to TRIANGULATE: nx_bm25_test asserts the team's integer ranking reproduces a float reference BM25's ranking, so the "exceed" is measured against the production formula, not cherry-picked. license_tier: ORIGINAL BM25(q,d) = sum_t idf(t) * tf(t,d)*(k1+1) / ( tf(t,d) + k1*(1 - b + b*dl/avgdl) ) idf(t) = ln( 1 + (N - df + 0.5)/(df + 0.5) ) = ln( (2N+2)/(2df+1) ) [always positive]

dependencies 2 imports · 0 importers

nx_research_extract.nx nx_syscalls.nx nx_bm25_naive.nx

imports: nx_research_extract.nxnx_syscalls.nx

imported by: nobody (leaf or entry point)

structs

none

consts

17const BM_K1_X1000: i64 = 1200 // k1 = 1.2 (term-frequency saturation)
18const BM_B_X1000: i64 = 750 // b = 0.75 (length-normalization strength)
19const BM_MICRO: i64 = 1000000 // fixed-point scale for ln / idf
20const BM_MILLI: i64 = 1000 // fixed-point scale for the saturation factor

functions

23func bm_token_count(text: *u8, n: i64) -> i64
36func bm_ln_micro(num: i64, den: i64) -> i64
called by 1: bm_idf_micro
53func bm_idf_micro(N: i64, df: i64) -> i64 { return bm_ln_micro(2 * N + 2, 2 * df + 1) }
called by 2: bm_scorebm_naive_tfidf calls 1: bm_ln_micro
57func bm_sat_milli(tf: i64, dl: i64, avgdl: i64) -> i64
called by 1: bm_score
67func bm_df(ptrs: *i64, lens: *i64, N: i64, term: *u8) -> i64
called by 2: bm_scorebm_naive_tfidf calls 1: re_has
74func bm_score(ptrs: *i64, lens: *i64, dls: *i64, N: i64, k: i64, avgdl: i64, qterms: *i64, nq: i64) -> i64
90func bm_best(ptrs: *i64, lens: *i64, dls: *i64, N: i64, avgdl: i64, qterms: *i64, nq: i64) -> i64
calls 1: bm_score
101func bm_naive_tfidf(ptrs: *i64, lens: *i64, N: i64, k: i64, qterms: *i64, nq: i64) -> i64
112func bm_naive_best(ptrs: *i64, lens: *i64, N: i64, qterms: *i64, nq: i64) -> i64
calls 1: bm_naive_tfidf