nx_arcmine.nx

nx_arcmine.nx -- INDEX-FIRST domain media harvester. The rung that makes a sovereign archive EXCEED the Wayback Machine on media completeness (the 2026-06-29 goal: page3.com / megastar.co.uk day-by-day WITH their media). module: nishi-core.archive.arcmine depends: nx_medrec.nx (admission), nx_cdx_parse.nx (rows/fields), nx_https_fetch_follow.nx usage: nx_arcmine <domain> [page] [min_bytes] [max_fetch] [mime] capability: LIVE FETCH, read-only against the archive. Additive + IDEMPOTENT (Rule 10). *WHY INDEX-FIRST, AND WHY IT BEATS THE PAGE-WALK -- the whole point of this organ: nx_archive_site_viewer discovers media by RENDERING a captured page and reading its <img> tags. That ceiling is structural: it can only ever find media referenced by the pages it happens to render, at the one timestamp it renders them. Every image on a page never rendered, on a page never captured, or referenced only from a since-changed layout, is INVISIBLE to it -- and its absence is invisible too. The CDX index knows every image the crawler EVER saw across the whole host and all time. Enumerating from the index is exhaustive BY CONSTRUCTION rather than by luck of which page we happened to open. That is the difference between browsing an archive and MINING one. *DEDUPE BY DIGEST, MEASURED 2026-08-06: captures of one URL are byte-identical. Real sample -- page3.com/3zine_pages/html_preview/01/ held 21 status-200 capture rows that collapse to just 6 unique digests (01.jpg alone: 6 captures, ONE digest HSTRJJRQ.., sizes 27236..27245 = 9 bytes of WARC record framing, not pixels). Fetching per-capture would spend 3.5x the bandwidth for byte-identical output. collapse=digest is therefore applied at the QUERY, so the redundancy never crosses the network at all. *AND THE COROLLARY THAT KILLED A PLANNED CHANGE: because captures are byte-identical, choosing the LARGEST capture over the EARLIEST buys exactly nothing on this corpus. That tweak was measured, found worthless, and NOT shipped. *TOMBSTONES ARE REJECTED FROM THE INDEX (nx_medrec), before a single byte is fetched -- a host's "image not found" placeholder is a structurally valid JPEG that magic-byte validation cannot fail. *IDEMPOTENT BY CONSTRUCTION: the output filename IS the content digest, so a re-run rewrites identical bytes and can never duplicate or corrupt. Safe to run twice, or to resume a partial page. *PACING: max_fetch bounds one invocation deliberately. The archive is a shared resource and we have rate-limited ourselves off hosts before by sweeping too fast; the operator advances pages at their own cadence. OWED: wire pace_before(host) from nx_crawl_pace so politeness is structural, not procedural. license_tier: ORIGINAL