nx_textnorm_lib.nx

nx_textnorm_lib.nx -- ONE COPY OF THE PRE-SHINGLING NORMALISATION DECISION. module: nishi-core.search.textnorm capability: CORE_COMPUTE (pure; reads no conf, touches no store, writes no file) license_tier: ORIGINAL No hw writes (Rule 26). WHY THIS EXISTS, MEASURED 2026-08-25 BY THE GATE THAT MOTIVATED IT: nx_sitededup_gate fingerprints a REAL capture pair -- one document stored twice, the second copy carrying an HTTP capture header and a bumped revision line -- and measured their Hamming distance at SIX bits against the published k=3 bar. The collapse pass therefore did NOT collapse the very duplicate class it was built for. A fingerprint taken over un-normalised bytes cannot: the header and the revision id are TOKENS, and tokens move bits. THE ESTABLISHED FIX, and it is not ours: normalise the text BEFORE shingling, and in particular normalise DIGITS TO A CONSTANT -- the step that collapses counter, date, revision and price near-duplicates, which is exactly the class our store holds. Charikar (STOC 2002) defines the fingerprint and Manku/Jain/Das Sarma (WWW 2007) the k=3 bar for 64-bit fingerprints; NEITHER says anything about what text you hand it, and that omission is where our duplicates were surviving. The normalisation is the missing half of the published recipe, not an invention of ours. COMPOSES, NEVER RE-IMPLEMENTS: nx_simhash for the fingerprint (there must stay exactly ONE fingerprint kernel in the estate) and nx_html_to_text for markup (there must stay exactly ONE markup stripper). This file adds only what neither has: the capture-header strip and the digit fold. WHAT IT DELIBERATELY DOES NOT DO: it does not re-implement block-density boilerplate extraction. bd_fit_text (nx_block_density) is the ruler for that and belongs to the ingest lane, which already applies it FORWARD. The residue actually present in already-stored rows is the HTTP capture header, which is what tn_body_offset removes -- precisely, by parsing the header grammar, not by a heuristic that could eat the first paragraph of a document. FAILS TOWARD DOING NOTHING IS NOT AVAILABLE HERE, SO SAY SO PLAINLY: every step below can only make two texts MORE similar, so the direction of any error is a FALSE POSITIVE -- a document wrongly judged a duplicate and hidden from search. That is why the consumer carries a neg-control of genuinely different documents and measures its false-positive rate against REAL corpus documents, never against fixtures written by the same hand that chose the rule.