nx_sitededup.nx

nx_sitededup.nx -- COLLAPSE DUPLICATE DOCUMENTS IN A SITE SEARCH SHARD. THE DEFECT, measured 2026-08-25 on a live scope=site SERP for "nishi search": of 30 results, SIX were the SAME document under two different cids -- e.g. "VORTEX RISC-V GPGPU -- COMPLETE PUBLICATION BIBLIOGRAPHY" as 7477184972746352550 AND 2839761502039440748, with IDENTICAL scores. ROOT CAUSE: a cid is a content hash of the RAW STORED BYTES, and TWO DIFFERENT HASH FUNCTIONS write the same doc:<cid> keyspace -- dp_cid (FNV-1a, nx_docportal_lib) and ci_hash (base-131 polynomial, nx_corpus_ingest). Two captures of one page that differ only in a Date: header or a revision line are therefore two different documents, and cross-path dedup is impossible by construction. Ingest has since been fixed FORWARD (dp_ingest_policy strips capture headers and runs bd_fit_text before hashing). That does nothing for rows already stored. This organ is the backward half. ---- WHY THE FIRST VERSION OF THIS ORGAN COULD NOT HAVE WORKED, AND HOW WE KNOW ---------------- The first cut fingerprinted the RAW stored bytes. Its own gate then measured the real capture pair -- one document, stored twice, the second copy carrying an HTTP header and a bumped revision line -- at SIX bits of Hamming distance against the published k=3 bar. It would have collapsed NOTHING. The published recipe has a step the fingerprint papers do not cover: NORMALISE THE TEXT BEFORE SHINGLING, and specifically normalise DIGITS TO A CONSTANT, which is what kills counter, date, revision and price near-duplicates. That is precisely our duplicate class. nx_textnorm_lib is that step, it is a SHARED lib so the gate proves the same normalisation the actuator ships, and this organ now reports the raw-vs-normalised comparison ON THE WHOLE POPULATION so the claim "normalisation is what made this work" is a measurement rather than an assertion. WHY SUPPRESSION AND NOT DELETION: rule 13, additive-only. Nothing is removed. A duplicate is marked by appending a pol:<cid> row with the public-search consent bit cleared -- the SAME row the search path ALREADY reads per candidate (dss_search, DSS_POL_SEARCH), so this needs no change to any serve organ and takes effect on the next query. It is reversible by appending a row with the bit set. Every other owner flag on that row is preserved (sdd_clear_search_bit). WHY DURABLE AND NOT AT RANK TIME: the alternative is fingerprinting every candidate on every query. Fingerprinting is a full pass over the document TEXT, and the serve path deliberately does not read doc bodies during candidate filtering (the site: filter's own comment: "Cheap: ss_hget per candidate, no doc reads"). Paying a text scan per candidate per query to rediscover a fact that does not change between queries is the wrong side of the trade. This pass pays it once. COMPOSES, NEVER RE-IMPLEMENTS: nx_simhash for the fingerprint (Charikar 2002 / Manku 2007), nx_textnorm_lib for the ONE copy of the normalisation decision, nx_sitededup_lib for the ONE copy of the collapse decision, dss_mkpolkey for the policy row, and ccz_cat_num as the integer emitter. THE FINGERPRINT PLANE IS DELIBERATELY A NEW KEYSPACE. Persisted fingerprints go to fpn:<cid>