nx_sitededup_lib.nx

nx_sitededup_lib.nx -- THE ONE COPY OF THE NEAR-DUPLICATE COLLAPSE DECISION. module: nishi-core.search.sitededup capability: CORE_COMPUTE (pure; reads a conf, touches no store, writes no file) license_tier: ORIGINAL No hw writes (Rule 26). WHY A LIB AND NOT A FUNCTION INSIDE THE ACTUATOR: the actuator suppresses rows in a LIVE production shard and the gate proves the decision is correct. If each held its own copy of the rule they could disagree, and the one that ships is the one nobody proved. There is exactly one clusterer here, and both import it -- disagreement is impossible by construction. COMPOSES nx_simhash (Charikar STOC 2002 fingerprint, Manku WWW 2007 threshold). It does NOT contain a second fingerprinting ruler and must never grow one. The TEXT handed to that fingerprint is normalised by nx_textnorm_lib, which is likewise a single shared copy: the clusterer never sees raw bytes and never decides what normalisation means. ---- THE TRANSITIVITY HAZARD, HANDLED RATHER THAN ASSUMED ------------------------------- Near-duplication is NOT transitive: a~b and b~c does not give a~c. So a connected component of the near-dup graph can be a CHAIN whose endpoints are genuinely different documents, and collapsing it onto one survivor would suppress a document that is not a near-duplicate of anything left searchable -- i.e. it would DELETE INFORMATION, which is the one outcome this organ exists to avoid. The fix is not to hope chains are rare. sdd_mark_chained MEASURES every component's diameter (its widest internal pair) and any component wider than the threshold is marked UNSAFE and left ENTIRELY ALONE -- every member stays searchable. The organ abstains on exactly the cases it cannot justify, and reports how many. Abstain, never acquit; and here, abstain rather than destroy. The survivor of a safe component is therefore always within `k` bits of EVERY doc it replaces, which is the property that makes suppression defensible at all. DETERMINISM: union-by-lowest-cid means a component's root is its globally minimum cid no matter what order the unions happen in. Two runs over the same shard pick the same survivor, so the pass is idempotent and re-runnable -- it can never flip-flop and eat the corpus.