code wiki / _hdl_build / nx_web_ingest.nx

nx_web_ingest.nx

buildroot/runtime/_hdl_build/nx_web_ingest.nx

11769 B235 linesdepth 21pulls 157 transitivereach 0 importersview sourcekind tooltopic web
docsdependenciesstructsconstsfunctions

about

nx_web_ingest.nx -- OPEN-WEB front of nishi search: BFS-crawl the REAL web (sovereign TLS-1.3 + Mozilla CA, polite, simhash near-dup killed) and ingest each kept page STRAIGHT into the shared "web" seg_store shard (dp-web-pub-): doc:<cid> = the page's extracted text, url:<cid> = its ABSOLUTE url -> the /search?scope=web SERP links results to the real pages. Composes the PROVEN pieces only: nx_crawl_web's fetch/link spine + nx_corpus_ingest's writer/key/hash mechanics (ci_hash / ci_mkurlkey / fresh-segid batching, gate 7/7). usage: nx_web_ingest <seed-url> <max_pages> Bounded by max_pages AND the known per-process certloop budget (~19 fetches) -- crawl-at-scale = many small runs, each committing its own segments (append-only, idempotent by content cid). license_tier: ORIGINAL

dependencies 5 imports · 0 importers

nx_corpus_ingest.nx nx_x509_trust_store.nx nx_trust_store_load_from_certdata. nx_https_fetch_follow.nx nx_simhash.nx nx_web_ingest.nx

imports: nx_corpus_ingest.nxnx_x509_trust_store.nxnx_trust_store_load_from_certdata.nxnx_https_fetch_follow.nxnx_simhash.nx

imported by: nobody (leaf or entry point)

call flow from main pre-order; caps 40 nodes / depth 6 declared; ↻ = already shown

main wi_puts wi_atoi nx_trust_store_load_from_c sys_mmap sys_read_file sys_openat_rd sys_lseek sys_mmap ↻ sys_read sys_close nx_nss_certdata_parse sys_mmap ↻ _pat_class_cert _pat_value_octal _pat_end _find_newline _starts_with _parse_octal_line sys_mmap ↻ _is_space _decode_octal_escape _is_octal trust_store_alloc sys_mmap ↻ nx_x509_trust_store_load sys_mmap ↻ x509_parse sys_mmap ↻ asn1_cursor_init asn1_expect_tag sys_mmap ↻ asn1_read_tlv_header asn1_read_tag asn1_read_length asn1_read_length ↻ x509_read_tlv sys_mmap ↻ asn1_expect_tag ↻ x509_read_alg_id

structs

none

consts

14const K_MAGIC_8192: i64 = 8192
15const K_MAGIC_4194304: i64 = 4194304
16const K_MAGIC_8388608: i64 = 8388608
17const K_MAGIC_1048576: i64 = 1048576
18const K_MAGIC_2048: i64 = 2048

functions

20func wi_puts(s: *u8) -> i64 { var n: i64 = 0; while s[n] != (0 as u8) { n = n + 1 } sys_write(1, s, n); return 0 }
called by 1: main
21func wi_num(v: i64) -> i64
called by 1: main
30func wi_len(s: *u8) -> i64 { var n: i64 = 0; while s[n] != (0 as u8) { n = n + 1 } return n }
called by 2: wi_basemain
31func wi_atoi(s: *u8) -> i64 { var v: i64 = 0; var i: i64 = 0; while s[i] != (0 as u8) { let c: i64 = s[i] as i64; if c >= 48 { if c <= 57 { v = v * 10 + (c - 48) } } i = i + 1 } return v }
called by 1: main
34func wi_base(url: *u8, out: *u8) -> i64
called by 1: main calls 1: wi_len
53func wi_eqn(a: *u8, alen: i64, b: *u8) -> i64
called by 1: wi_enqueue
58func wi_enqueue(url: *u8, ulen: i64, seen: **u8, nseen: *i64, queue: **u8, qt: *i64, cap: i64) -> i64
called by 2: wi_linksmain calls 1: wi_eqn
65func wi_pre(h: *u8, hlen: i64, off: i64, lit: *u8, litlen: i64) -> i64
called by 1: wi_links
69func wi_lc(c: i64) -> i64 { if c >= 0x41 { if c <= 0x5a { return c + 0x20 } } return c }
called by 1: wi_href_at
70func wi_href_at(h: *u8, hlen: i64, off: i64) -> i64
called by 1: wi_links calls 1: wi_lc
123func main(argc: i64, argv: *i64) -> i64