code wiki / _hdl_build / nx_bm25_naive.nx
nx_bm25_naive.nx
buildroot/runtime/_hdl_build/nx_bm25_naive.nx
about
nx_bm25.nx -- production-grade ranked retrieval: Okapi BM25 (Robertson & Sparck Jones), integer-only.
This is the HONEST upgrade over the team's first TF-IDF ranker, which lacked LENGTH NORMALIZATION:
a long document that mentions a term once would out-rank a short focused document with the same
per-length relevance. BM25 fixes that with (1) tf SATURATION (k1) -- the 10th occurrence adds less
than the 2nd -- and (2) LENGTH NORMALIZATION (b) -- penalize long docs by dl/avgdl. The point of
building the real standard (not a weak baseline) is to TRIANGULATE: nx_bm25_test asserts the team's
integer ranking reproduces a float reference BM25's ranking, so the "exceed" is measured against the
production formula, not cherry-picked. license_tier: ORIGINAL
BM25(q,d) = sum_t idf(t) * tf(t,d)*(k1+1) / ( tf(t,d) + k1*(1 - b + b*dl/avgdl) )
idf(t) = ln( 1 + (N - df + 0.5)/(df + 0.5) ) = ln( (2N+2)/(2df+1) ) [always positive]
dependencies 2 imports · 0 importers
imports: nx_research_extract.nxnx_syscalls.nx
imported by: nobody (leaf or entry point)
structs
| none |
consts
| 17 | const BM_K1_X1000: i64 = 1200 // k1 = 1.2 (term-frequency saturation) |
| 18 | const BM_B_X1000: i64 = 750 // b = 0.75 (length-normalization strength) |
| 19 | const BM_MICRO: i64 = 1000000 // fixed-point scale for ln / idf |
| 20 | const BM_MILLI: i64 = 1000 // fixed-point scale for the saturation factor |
functions
| 23 | func bm_token_count(text: *u8, n: i64) -> i64 |
| 36 | func bm_ln_micro(num: i64, den: i64) -> i64 called by 1: bm_idf_micro |
| 53 | func bm_idf_micro(N: i64, df: i64) -> i64 { return bm_ln_micro(2 * N + 2, 2 * df + 1) } |
| 57 | func bm_sat_milli(tf: i64, dl: i64, avgdl: i64) -> i64 called by 1: bm_score |
| 67 | func bm_df(ptrs: *i64, lens: *i64, N: i64, term: *u8) -> i64 |
| 74 | func bm_score(ptrs: *i64, lens: *i64, dls: *i64, N: i64, k: i64, avgdl: i64, qterms: *i64, nq: i64) -> i64 |
| 90 | func bm_best(ptrs: *i64, lens: *i64, dls: *i64, N: i64, avgdl: i64, qterms: *i64, nq: i64) -> i64 calls 1: bm_score |
| 101 | func bm_naive_tfidf(ptrs: *i64, lens: *i64, N: i64, k: i64, qterms: *i64, nq: i64) -> i64 |
| 112 | func bm_naive_best(ptrs: *i64, lens: *i64, N: i64, qterms: *i64, nq: i64) -> i64 calls 1: bm_naive_tfidf |