nx_crawl_sufficiency.nx

nx_crawl_sufficiency.nx -- ADAPTIVE STOP: "do we know enough to stop fetching?" -- the /compare/webscraping R5 contract (symbol cs_confidence). Concept source: crawl4ai's AdaptiveCrawler (docs/core/adaptive-crawling.md, StatisticalStrategy in adaptive_crawler.py): confidence = coverage + consistency + saturation, stop at a bar. Sovereign form, integer permil, no floats, no LLM, no embeddings: COVERAGE for each query term t: doc_cov = df_t * 1000 / docs; freq = ilog2(1+tf_t) / ilog2(1+max_tf); term = doc_cov * (1000 + freq_boost * freq / 1000) / 1000, capped at 1000; coverage = mean over terms. CONSISTENCY permil of counted docs that are ON TOPIC: hold at least relevant_permil of the query terms (min 1). (The source uses mean pairwise Jaccard of term sets, which REWARDS redundancy -- five copies of one page score as perfectly consistent. Ours cannot be inflated by copies: see the dedup below.) SATURATION 1000 - new_terms(last window) * 1000 / new_terms(first window): diminishing discovery of NEW vocabulary. UNOBSERVABLE (not 0, not 1000) until two windows exist or when the term set is full. confidence = (w_cov * coverage + w_con * consistency + w_sat * saturation) / (w_cov + w_con + w_sat) EXCEED over the source: a doc whose simhash lands within simhash_ham of an already-counted doc is a NEAR-DUPLICATE and is not counted anywhere (not df, not tf, not the new-term history) -- it is tallied as a dup so the partition docs_added == counted + dups always sums. A crawl that keeps fetching copies therefore cannot talk itself into confidence, which is exactly the failure the source's Jaccard consistency has. THIRD STATE: cs_should_stop returns CONTINUE / CONFIDENT / SATURATED, and saturation carries its own sat_observed flag: when the term set fills, saturation is UNOBSERVABLE and counts as 0 in confidence (it can only make the verdict more conservative, never acquit) and the report names it. EVERY NUMBER IS A CONF ROW: knowledge/crawl_sufficiency.conf (nx_lane_conf semantics), bootstrap defaults below cite their source. Consumers: nx_web_crawl_step (query-scoped crawl), the research fetch, the CLI. license_tier: ORIGINAL module: nishi-core.search.sufficiency No hw writes (Rule 26).