nx_cc_warc_ingest.nx

nx_cc_warc_ingest.nx -- BULK Common Crawl ingester: a LOCAL .warc.gz file -> the dp-web-pub- web shard. SCALE rung over nx_cc_ingest (one network round-trip per record): here ONE pre-downloaded WARC file (~1GB, ~30-40k pages; each CC record = its own gzip member) is stream-walked member-by-member -> 'response' records -> HTTP 200 + text/html -> html->text -> quality filters -> ss_add doc:/url:/loc: rows + out: edges (same node identity as the crawler/serve, so every page feeds PageRank). BATCH+RESUME BY CONSTRUCTION: the deflate path allocates interior mappings it cannot free, so one process walks at most <maxmembers> members starting at the byte offset checkpointed in <localfile>.off, commits its segments, rewrites the checkpoint, exits. A driver loops invocations until DONE prints. (Also crash-safe: a killed batch never advances the checkpoint past committed work.) usage: nx_cc_warc_ingest <localfile> <cc-warc-path> [maxmembers] <cc-warc-path> = the crawl-data/... path the file came from; stored in loc: rows for the rot-fallback, byte-offset-exact, so a served result can re-range-fetch its record just like nx_cc_ingest's rows. exit 0 = batch clean (prints MORE nextoff=N or DONE, + verdict=PASS) ; nonzero = failure, checkpoint NOT advanced past the bad byte (surface corruption, never silently skip). license_tier: ORIGINAL