nx_crawlaudit.nx

nx_crawlaudit.nx -- INDEPENDENT validator for the crawl outcome log. The anti-navel-gazing organ. WHY IT EXISTS. nx_crawlmine reads OUR OWN log, so it can only ever confirm what the crawler already believes. If the classifier is wrong, the miner reports the wrong thing CONFIDENTLY -- a closed loop that measures itself and calls the result evidence. Operator 2026-08-06: "a secondary system to validate we arent navel gazing like we have in all kinds of other systems." INDEPENDENCE IS THE ENTIRE POINT, so this organ deliberately shares NO decision logic with the crawler. The crawler classifies a bot-wall by PHRASE MATCH ("Just a moment", "Attention Required"). Re-using those phrases here would be circular -- it would agree with itself by construction and prove nothing. Instead this judges by TEXT YIELD: strip the HTML and measure how much prose comes out. A challenge interstitial yields almost nothing; a real page yields thousands of characters. Different mechanism, different failure modes, so agreement is INFORMATIVE. SEPARATING FLAKINESS FROM MISCLASSIFICATION. A re-fetch happens at a different time, so disagreement alone proves nothing -- MEASURED on this very estate: the same v2ph url ingested 2,176 chars one night and returned a 403 challenge the next morning. So each sampled url is fetched TWICE in this run: if the two fetches disagree WITH EACH OTHER the HOST is unstable (not our bug); if they agree with each other but disagree with what we RECORDED, that is a classifier candidate worth reading. u2605A DISAGREEMENT IS A FLAG, NEVER A VERDICT -- and a validator that forgets that just relocates the overconfidence it was built to catch. nx_crawlaudit [log-path] [max-samples] license_tier: ORIGINAL expect_exit: 0