nx_warc_reader.nx
buildroot/runtime/nx_warc_reader.nx
about
nx_warc_reader.nx -- LIB: sovereign WARC (ISO 28500 / Common Crawl) record reader. A .warc file is a sequence of
records: "WARC/1.0" CRLF, then "Name: Value" header lines, a blank line, a Content-Length-delimited content block,
then trailing CRLFs. A 'response' record wraps a full HTTP response (status line + headers + body). This parses the
plain-WARC record loop from an in-memory buffer; multi-member-gzip streaming (.warc.gz = one gzip member per record)
composes the existing nx_deflate inflate + bytes-consumed in a follow-on (R2b). Zero-dependency, byte-exact.
warc_next_record(buf,n,pos,fields) fills fields[6] for the record at pos and returns the START OF THE NEXT record
(>=0), or 0-1 when there is no record at pos. fields: [0]=type_off [1]=type_len [2]=uri_off [3]=uri_len
[4]=content_off [5]=content_len. license_tier: ORIGINAL
dependencies 1 imports · 2 importers
imports: nx_syscalls.nx
imported by: nx_cc_warc_ingest.nxnx_warc_reader_gate.nx
structs
| none |
consts
| none |
functions
| 11 | func wr_starts(buf: *u8, off: i64, n: i64, lit: *u8, litlen: i64) -> i64 |
| 17 | func wr_lc(c: i64) -> i64 { if c >= 0x41 { if c <= 0x5a { return c + 0x20 } } return c } called by 1: wr_starts_ci |
| 18 | func wr_starts_ci(buf: *u8, off: i64, n: i64, lit: *u8, litlen: i64) -> i64 |
| 24 | func wr_uint(buf: *u8, off: i64, n: i64, endp: *i64) -> i64 called by 1: warc_next_record |
| 37 | func wr_find_blankline(buf: *u8, off: i64, n: i64) -> i64 |
| 45 | func wr_contains(buf: *u8, off: i64, len: i64, lit: *u8, litlen: i64) -> i64 calls 1: wr_starts |
| 55 | func wr_header_val(buf: *u8, hstart: i64, hend: i64, n: i64, name: *u8, namelen: i64, out2: *i64) -> i64 |
| 79 | func warc_next_record(buf: *u8, n: i64, pos: i64, fields: *i64) -> i64 |