code wiki / (root) / nx_web_crawl_breadth.nx

nx_web_crawl_breadth.nx

buildroot/runtime/nx_web_crawl_breadth.nx

9388 B188 linesdepth 21pulls 128 transitivereach 0 importersview sourcekind tooltopic web
docsdependenciesstructsconstsfunctions

about

nx_web_crawl_breadth.nx -- BREADTH crawl: start from ONE seed page, EXTRACT its links, and fetch the linked pages too (bounded) -> the external engine reaches the web by FOLLOWING links, not just a fixed seedlist. Sovereign TLS-1.3 fetch (no browser). Politeness/safety BY CONSTRUCTION: same-domain article links only (href="/wiki/<Title>", namespaced titles with ':' skipped), de-duplicated, hard MAX_PAGES cap (no runaway). Emits one `id<TAB>text` line per page to knowledge/index/web_breadth_corpus.tsv for nx_onsite_index. expect_exit: 0 license_tier: ORIGINAL (fetch machinery from nx_web_crawl_index; + link extractor + frontier)

dependencies 10 imports · 0 importers

nx_syscalls.nx nx_x509_trust_store.nx nx_trust_store_load_from_certdata. nx_tls13_client_validate_certifica nx_tls13_client_session_run.nx nx_https_url_for_fetch.nx nx_https_url_connect.nx nx_https_get.nx nx_https_get_complete.nx nx_http_response_parse.nx nx_web_crawl_breadth.nx

imports: nx_syscalls.nxnx_x509_trust_store.nxnx_trust_store_load_from_certdata.nxnx_tls13_client_validate_certificate.nxnx_tls13_client_session_run.nxnx_https_url_for_fetch.nxnx_https_url_connect.nxnx_https_get.nxnx_https_get_complete.nxnx_http_response_parse.nx

imported by: nobody (leaf or entry point)

call flow from main pre-order; caps 40 nodes / depth 6 declared; ↻ = already shown

main cb_puts sys_write nx_trust_store_load_from_c sys_mmap sys_read_file sys_openat_rd sys_lseek sys_mmap ↻ sys_read sys_close nx_nss_certdata_parse sys_mmap ↻ _pat_class_cert _pat_value_octal _pat_end _find_newline _starts_with _parse_octal_line sys_mmap ↻ _is_space _decode_octal_escape _is_octal trust_store_alloc sys_mmap ↻ nx_x509_trust_store_load sys_mmap ↻ x509_parse sys_mmap ↻ asn1_cursor_init asn1_expect_tag sys_mmap ↻ asn1_read_tlv_header asn1_read_tag asn1_read_length asn1_read_length ↻ x509_read_tlv sys_mmap ↻ asn1_expect_tag ↻ x509_read_alg_id

structs

none

consts

17const CB_MAGIC_4194304: i64 = 4194304
18const CB_MAGIC_65536: i64 = 65536
20const CB_TEXTCAP: i64 = 65536
21const CB_MAX_PAGES: i64 = 6 // hard cap (seed + up to 5 discovered) -- no runaway
22const CB_HOST: *u8 = "en.wikipedia.org" as *u8
23const CB_PREFIX: *u8 = "https://en.wikipedia.org/wiki/" as *u8 // 30 chars
24const CB_SEED: *u8 = "https://en.wikipedia.org/wiki/Information_retrieval\x00"
25const CB_SEED_ID: *u8 = "Information_retrieval" as *u8
26const CB_OUT: *u8 = "knowledge/index/web_breadth_corpus.tsv" as *u8

functions

28func cb_puts(s: *u8) -> i64 { var n: i64=0; while s[n]!=(0 as u8){n=n+1} sys_write(1,s,n); return 0 }
called by 1: main calls 2: sys_mmapsys_write
29func cb_num(v: i64) -> i64 { let bb: *u8=sys_mmap(28); var m: i64=v; if m<0{m=0-m;sys_write(1,"-" as *u8,1)}; let t: *u8=sys_mmap(28); var k: i64=0; if m==0{t[0]=(48 as u8);k=1}; while m>0{t[k]=((48+(m%10)) as u8);m=m/10;k=k+1}; var i: i64=0; while i<k{bb[i]=t[k-1-i];i=i+1}; sys_write(1,bb,k); return 0 }
called by 3: cb_fetchcb_emitcb_links
30func cb_strlen(s: *u8) -> i64 { var n: i64=0; while s[n]!=(0 as u8){n=n+1} return n }
32func cb_path_off(url: *u8, n: i64) -> i64
44func cb_fetch(store: *TrustStore, full_url: *u8, out_buf: *i64, out_len: *i64) -> i64
80func cb_emit(ofd: i64, id: *u8, buf: *u8, n: i64, tbuf: *u8) -> i64
133func main() -> i64