nx_sitededup_gate.nx

nx_sitededup_gate.nx -- proves the near-duplicate collapse decision COLLAPSES WHAT IT SHOULD AND NOTHING ELSE, and that DIGIT NORMALISATION is what makes the first one possible. A dedup pass that collapses everything scores a perfect duplicate-removal rate and destroys the corpus. So the load-bearing tooth here is not "did the duplicate collapse" -- it is the NEG-CONTROL: genuinely different documents that must SURVIVE. Without it this gate would go green on an implementation that suppressed every document but one. And because normalisation can only ever make two texts MORE similar, the neg-control matters MORE after normalisation than before it, not less. ---- WHAT THIS GATE MEASURED THAT CHANGED THE DESIGN ------------------------------------------- The first version of this gate ran its fixtures through the RAW fingerprint and reported dup(A,A2) = 6 against a bar of k = 3 i.e. the collapse pass DID NOT COLLAPSE the exact production duplicate class it was written for. That is why nx_textnorm_lib exists. The teeth below pin BOTH sides of that finding: the raw ruler must MISS the pair (or the normalisation tooth proves nothing) and the normalised ruler must CATCH it. A single tooth asserting only the second would pass just as happily on a ruler that collapses the entire corpus. ---- AND WHAT THE SECOND RUN MEASURED, WHICH CHANGED THE FIXTURES ------------------------------- Two of the first fixtures were VACUOUS and this gate said so rather than being quietly believed: * the capture-header pair was a single string literal with NO NEWLINES, so the header block had no line structure to parse, the strip correctly refused, and the pair measured raw=6 vs normalised=7 -- normalisation made it WORSE. A capture header in the real store has line breaks; a fixture without them tests a document shape that cannot occur. * the digit-only pair was 607 bytes of prose carrying nine numbers, and its RAW distance was already 2 -- INSIDE the k=3 bar. It could never have shown that normalisation was necessary, because the un-normalised ruler already caught it. Both are fixed below by BUILDING the fixtures at runtime, and the prose pair is kept -- with an honest tooth of its own asserting the opposite property (normalisation must not LOSE a duplicate the raw ruler already had). A fixture that cannot fail is not a test; a fixture that fails for a reason other than the one under test is worse, because its red looks like evidence. The fixtures are REAL TEXT run through the REAL hashers and the REAL shared clusterer (nx_sitededup_lib) -- not hand-picked fingerprints. The one place fingerprints ARE constructed by bit-flipping is the CHAIN fixture, where the property under test is about exact distances and must be exercised at exact distances. THE DIGIT FIXTURES ARE VERIFIED MECHANICALLY, NOT ASSERTED IN A COMMENT. sg_differs_only_in_digits walks both texts byte by byte and requires every differing position to be a digit in BOTH and at least one difference to exist. A fixture that merely CLAIMS to differ only in digits, and in fact