nx_warc_index.nx
buildroot/runtime/nx_warc_index.nx
about
nx_warc_index.nx -- C3: the WARC -> INVERTED-INDEX ingest pipeline = the heart of "ingest Common Crawl into our
index". Walks WARC 'response' records (Common Crawl / ISO 28500), extracts each record's content, and two-pass
indexes it into nx_search_inverted -> queryable. This is what runs at NAS scale over CC WET/WARC segments (composing
gz-inflate + posting-compression + sharding + write-read-decoupling, all gated separately). The WARC record walk is
inlined here (byte-identical framing to the canonical nx_warc_reader) to keep a single clean syscalls lineage --
nx_warc_reader carries its own main() + the nx_syscalls/syscalls.nx alias, and importing it alongside
nx_search_inverted trips the large-graph double-import/double-main limitation (filed). Built + gated NOW on an
in-memory WARC fixture; the live CC fetch + gz-inflate run on the NAS. license_tier: ORIGINAL
dependencies 1 imports · 1 importers
imports: nx_search_inverted.nx
imported by: nx_warc_index_gate.nx
structs
| none |
consts
| none |
functions
| 11 | func wi_lc(c: i64) -> i64 { if c >= 0x41 { if c <= 0x5a { return c + 0x20 } } return c } |
| 12 | func wi_starts(buf: *u8, off: i64, n: i64, lit: *u8, litlen: i64) -> i64 called by 1: wi_next_record |
| 17 | func wi_starts_ci(buf: *u8, off: i64, n: i64, lit: *u8, litlen: i64) -> i64 |
| 22 | func wi_blankline(buf: *u8, off: i64, n: i64) -> i64 called by 1: wi_next_record |
| 27 | func wi_uint(buf: *u8, off: i64, n: i64) -> i64 called by 1: wi_next_record |
| 33 | func wi_header_val(buf: *u8, hstart: i64, hend: i64, n: i64, name: *u8, namelen: i64, out2: *i64) -> i64 |
| 49 | func wi_next_record(buf: *u8, n: i64, pos: i64, fields: *i64) -> i64 |
| 70 | func wi_type_is(buf: *u8, off: i64, len: i64, lit: *u8, litlen: i64) -> i64 |
| 76 | func wi_build_index(warcbuf: *u8, n: i64, idx: *NxInvIndex, coff: *i64, clen: *i64, maxdoc: i64) -> i64 called by 1: main calls 5: wi_next_recordwi_type_isnx_inv_index_rownx_inv_finalize_offsetsnx_inv_emit_row |