code wiki / _hdl_build / nx_cc_warc_fetch.nx
nx_cc_warc_fetch.nx
buildroot/runtime/_hdl_build/nx_cc_warc_fetch.nx
about
nx_cc_warc_fetch.nx -- sovereign Common Crawl bulk downloader (feeds nx_cc_warc_ingest).
Two verbs:
nx_cc_warc_fetch paths <crawl-id> [n] -> fetch crawl-data/<id>/warc.paths.gz, print first n paths
nx_cc_warc_fetch get <cc-warc-path> <outfile> -> chunked RESUMABLE download of one ~1GB .warc.gz
get: resumes from the existing outfile size (append-only, additive rule 13); sequential single-stream
range fetches with backoff on non-206 (polite to data.commoncrawl.org, which throttles bursts with 503);
EOF = a short chunk. Own TLS-1.3 + Mozilla trust store; zero curl. license_tier: ORIGINAL
dependencies 4 imports · 0 importers
imports: nx_x509_trust_store.nxnx_trust_store_load_from_certdata.nxnx_https_fetch_follow.nxnx_gzip_wrap.nx
imported by: nobody (leaf or entry point)
call flow from main pre-order; caps 40 nodes / depth 6 declared; ↻ = already shown
structs
| none |
consts
| 12 | const CWF_MAGIC_4194304: i64 = 4194304 |
| 13 | const CWF_MAGIC_1024: i64 = 1024 |
| 14 | const CWF_MAGIC_2048: i64 = 2048 |
| 15 | const CWF_MAGIC_131072: i64 = 131072 |
| 17 | const CWF_CHUNK: i64 = 4194304 // 4MiB range per request: big enough to amortize TLS, small enough to retry cheaply |
| 18 | const CWF_MAXRETRY: i64 = 6 // attempts per chunk before giving up |
| 19 | const CWF_BACKOFF_MS: i64 = 3000 // linear backoff step between retries (3s, 6s, 9s...) rides the 503 throttle |
| 20 | const CWF_PATHCAP: i64 = 4194304 // compressed warc.paths.gz cap |
| 21 | const CWF_PATHINFL: i64 = 33554432 // inflated warc.paths cap (~80k lines x ~100B) |
| 22 | const CWF_PROGRESS: i64 = 16 // print progress every N chunks (= 64MiB) |
functions
| 24 | func cf_puts(s: *u8) -> i64 { var n: i64 = 0; while s[n] != (0 as u8) { n = n + 1 } sys_write(1, s, n); return 0 } called by 1: main |
| 25 | func cf_num(v: i64) -> i64 called by 1: main |
| 34 | func cf_len(s: *u8) -> i64 { var n: i64 = 0; while s[n] != (0 as u8) { n = n + 1 } return n } |
| 35 | func cf_cat(d: *u8, o: i64, s: *u8) -> i64 { var i: i64 = 0; while s[i] != (0 as u8) { d[o+i] = s[i]; i = i + 1 } return o + i } called by 1: main |
| 36 | func cf_atoi(s: *u8) -> i64 called by 1: main |
| 41 | func cf_eq(a: *u8, b: *u8) -> i64 called by 1: main |
| 48 | func main(argc: i64, argv: *i64) -> i64 |