nx_memfind.nx

nx_memfind.nx -- BM25 ranked retrieval over the laptop-local memory corpus. CONVERTED TO nx_memplane_lib 2026-08-06. It now owns only the ranking: term counting, document length, the integer log, the stopword list and query derivation. The walk, name table, read, join and dump list come from the lib. WHY BM25, AND WHAT IT REPLACED (measured -- three rankers, one test) ------------------------------------------------------------------- Test: reconstruct the ORIGINAL description of a memory that duplicates an existing one (the store law, written twice by two sessions three weeks apart) and ask whether the ranker surfaces the file it duplicates. rarity-weighted presence -> TARGET NOT FOUND conjunction (fraction present) -> TARGET RANK 22 of 2173 BM25 -> TARGET RANK 1 of 2173 Both hand-rolled attempts failed for the SAME reason, solved in retrieval decades ago: * RARITY fails because this is a SINGLE-PROJECT corpus -- the words that identify a doctrine are the words the whole project uses (sovereign 1190/2176, data 1138, files 808, store 697, tsv 442). Dropping "common" terms deletes the topic and keeps the incidental. * CONJUNCTION fails because the biggest file contains every word; a 255 KB roadmap won a query about flat-file policy purely by being long. ★★★★★★ BOTH ARE PRESENCE MEASURES, AND PRESENCE IN A LONG DOCUMENT IS NEARLY FREE. BM25 fixes exactly that pair: term-frequency SATURATION (k1) and LENGTH NORMALISATION (b). ★ WHEN A HAND-ROLLED RANKER KEEPS FAILING, THE FIELD'S BASELINE IS THE KNOWN GOOD YOU WERE SUPPOSED TO MATCH FIRST. Integer notes: idf uses log2 rather than ln, which scales every term identically and so leaves the RANKING unchanged; the BM25 idf ratio simplifies to (2N+2)/(2df+1). Fixed point 1/256 for logs. TWO ENTRY POINTS, ONE RANKER: nx_memfind <term> ... -- a human picks the query (interactive retrieval) nx_memfind --file <path> -- the query is DERIVED from a file's own name+description ⚠ per-term df is printed deliberately: a rewrite once dropped it silently and the gate caught it. ★ A REWRITE THAT PRESERVES THE OUTPUT BUT DROPS THE EVIDENCE FOR IT IS A REGRESSION. usage: nx_memfind [--dir D] [--n K] [--quiet] (--file <path> | <term> [<term> ...]) Sovereign: imports nx_memplane_lib. license_tier: ORIGINAL