code wiki / _hdl_build / nx_crawl_web.nx

nx_crawl_web.nx

buildroot/runtime/_hdl_build/nx_crawl_web.nx

11681 B198 linesdepth 21pulls 151 transitivereach 0 importersview sourcekind tooltopic crawl
docsdependenciesstructsconstsfunctions

about

nx_crawl_web.nx -- REAL-WEB BFS crawler: "crawl whatever is out there", sovereign + polite. Combines what the ecosystem already had in pieces: nx_crawl_https (validated TLS fetch, but single-seed) + nx_crawl_frontier (link extraction, absolute-only) + adds RELATIVE-URL resolution (the v2 the frontier flagged) so the spider actually WALKS a site's links, not just the seed. Pipeline per page: dequeue -> polite sleep -> sovereign nx_https_fetch_follow (TLS-1.3 + Mozilla CA, the proven fetcher) -> nx_html_to_text (now OOM-safe) -> simhash dedup -> record -> extract+resolve outbound links -> enqueue. Bounded (max_pages + the per-process certloop leak ~19 fetches = the NEXT OOM-class fix for crawl-at-scale). Seeded on a cost-intelligence page so the crawl grows the corpus. license_tier: ORIGINAL

dependencies 6 imports · 0 importers

nx_syscalls.nx nx_x509_trust_store.nx nx_trust_store_load_from_certdata. nx_https_fetch_follow.nx nx_html_to_text.nx nx_simhash.nx nx_crawl_web.nx

imports: nx_syscalls.nxnx_x509_trust_store.nxnx_trust_store_load_from_certdata.nxnx_https_fetch_follow.nxnx_html_to_text.nxnx_simhash.nx

imported by: nobody (leaf or entry point)

call flow from main pre-order; caps 40 nodes / depth 6 declared; ↻ = already shown

main nx_trust_store_load_from_c sys_mmap sys_read_file sys_openat_rd sys_lseek sys_mmap ↻ sys_read sys_close nx_nss_certdata_parse sys_mmap ↻ _pat_class_cert _pat_value_octal _pat_end _find_newline _starts_with _parse_octal_line sys_mmap ↻ _is_space _decode_octal_escape _is_octal trust_store_alloc sys_mmap ↻ nx_x509_trust_store_load sys_mmap ↻ x509_parse sys_mmap ↻ asn1_cursor_init asn1_expect_tag sys_mmap ↻ asn1_read_tlv_header asn1_read_tag asn1_read_length asn1_read_length ↻ x509_read_tlv sys_mmap ↻ asn1_expect_tag ↻ x509_read_alg_id sys_mmap ↻ asn1_expect_tag ↻

structs

none

consts

15const K_MAGIC_8192: i64 = 8192
16const K_MAGIC_4194304: i64 = 4194304
17const K_MAGIC_8388608: i64 = 8388608
18const K_MAGIC_1048576: i64 = 1048576
19const K_MAGIC_2048: i64 = 2048

functions

21func gp(s: *u8) -> i64 { var n: i64=0; while s[n]!=(0 as u8){n=n+1} sys_write(1,s,n); return 0 }
called by 1: main calls 1: sys_write
22func gn(v0: i64) -> i64 { var v: i64=v0; if v<0{sys_write(1,"-" as *u8,1);v=0-v} let b: *u8=sys_mmap(24); var k: i64=0; if v==0{b[0]=48 as u8;k=1} while v>0{b[k]=(48+(v%10)) as u8;v=v/10;k=k+1} let o: *u8=sys_mmap(24); var j: i64=0; while j<k{o[j]=b[k-1-j];j=j+1} sys_write(1,o,k); return 0 }
called by 1: main calls 2: sys_writesys_mmap
23func slen(s: *u8) -> i64 { var n: i64=0; while s[n]!=(0 as u8){n=n+1} return n }
called by 1: cw_base
24func streq(a: *u8, b: *u8) -> i64 { var i: i64=0; while a[i]!=(0 as u8){ if a[i]!=b[i]{return 0} i=i+1 } if b[i]!=(0 as u8){return 0} return 1 }
25func cw_lc(c: i64) -> i64 { if c>=0x41 { if c<=0x5a { return c+0x20 } } return c }
called by 1: cw_href_at
27func cw_pre(h: *u8, hlen: i64, off: i64, lit: *u8, litlen: i64) -> i64
called by 1: cw_links
32func cw_href_at(h: *u8, hlen: i64, off: i64) -> i64
called by 1: cw_links calls 1: cw_lc
42func eq_nl(a: *u8, alen: i64, b: *u8) -> i64
called by 1: cw_enqueue
47func cw_enqueue(url: *u8, ulen: i64, seen: **u8, nseen: *i64, queue: **u8, qt: *i64, cap: i64) -> i64
called by 1: cw_links calls 2: eq_nlsys_mmap
100func cw_base(url: *u8, out: *u8) -> i64
119func cw_atoi(s: *u8) -> i64 { var v: i64=0; var i: i64=0; while s[i]!=(0 as u8){ let c: i64=s[i] as i64; if c>=48 { if c<=57 { v=v*10+(c-48) } } i=i+1 } return v }
calls 6: nx_trust_store_load_from_certdatagpsys_exitgntrust_store_countsys_mmap
121func main(argc: i64, argv: *i64) -> i64