nx_crawlmine.nx

nx_crawlmine.nx -- MINE the crawl outcome log so the crawler improves against MEASURED loss. WHY THIS EXISTS. Until R11 the crawler dropped pages silently: a Cloudflare interstitial, a 403, a TLS failure and a genuinely empty host all ended the same way -- no line, no row, nothing to count. The host-health streak then retired the host, which turns OUR capability gap into PERMANENT coverage loss that looks like the web's fault. Operator 2026-08-06: "make sure success isnt accidental and that we have an error log we are mining and improving our crawler on." This is the mining half. It answers the only questions that can drive the next rung: - what fraction of fetches actually succeed (the rate, not the anecdote) - which failure CLASS dominates (botwall vs http vs transport) = what to build next - which HOSTS we are losing, and to what = whether escalation is working per-host A SUCCESS ROW IS AS LOAD-BEARING AS A FAILURE ROW: without `ok` counts there is no denominator, and a failure count with no denominator is not a rate -- the same trap that once reported a fixed crash-loop as live. That is why the crawler logs ok too. 2026-08-13 WHOLE-FILE + WINDOW. v1 read the FIRST 4MiB and reported that head as the population -- a silent cap: once the log outgrew it every run returned the same frozen numbers (rows=50427 forever) and the post-R12 window was unobservable by construction. Now the 4MiB buffer is a CHUNK, the file streams to EOF, and coverage (bytes, rows, epoch span) is printed so a partial read can never pass as a population again. [since-epoch] restricts TALLIES to rows at/after that epoch = the windowed referee for before/after claims (ok_per_day, hostcap paid-fetch vs free-skip). nx_crawlmine [log-path] [since-epoch] license_tier: ORIGINAL expect_exit: 0