code wiki / _hdl_build / nx_crawl_callee_probe.nx

nx_crawl_callee_probe.nx

buildroot/runtime/_hdl_build/nx_crawl_callee_probe.nx

9566 B203 linesdepth 21pulls 163 transitivereach 0 importersview sourcekind probetopic crawl
docsdependenciesstructsconstsfunctions

about

nx_crawl_callee_probe.nx -- PER-CALLEE VmSize PROBE for the crawler's per-page pipeline (debt 1787082132). The +282 MB/min ingest growth survived the seg-store release-list fix; the named per-page mappings (wc_harvest 8 KiB + olh_scan ~9.5 KiB) explain only ~20 KB/page, so the dominant allocator is in the per-page CALLEES. This organ MEASURES each callee instead of reading it (the debt row's own law: "do NOT hoist by reading"): run the SAME page bytes through each stage N times and diff /proc/self/status VmSize around the loop. Per stage it prints BOTH numbers that matter: warm_kb = VmSize cost of the FIRST call (one-time lazy tables/policy loads -- not a leak) steady per-call = (delta over N calls after warm) / N -- the number that scales with pages/min Stages: H2T (nx_html_to_text), FEED (nx_feed_discover + nx_feed_extract), OLH (olh_scan), FETCH (nx_https_fetch_follow -- only when /tmp/mozilla_certdata.txt exists AND the arg is a url; otherwise that stage prints UNOBSERVABLE and abstains rather than acquitting). argv: nx_crawl_callee_probe <page-file-or-https-url> [iters] (file mode = no network, deterministic) Output: one CALLEE line per stage; the LAST line names the worst steady grower (positional anchor). license_tier: ORIGINAL expect_exit: 0

dependencies 5 imports · 0 importers

nx_x509_trust_store.nx nx_trust_store_load_from_certdata. nx_https_fetch_follow.nx nx_outlink_harvest.nx nx_feed_extract.nx nx_crawl_callee_probe.nx

imports: nx_x509_trust_store.nxnx_trust_store_load_from_certdata.nxnx_https_fetch_follow.nxnx_outlink_harvest.nxnx_feed_extract.nx

imported by: nobody (leaf or entry point)

call flow from main pre-order; caps 40 nodes / depth 6 declared; ↻ = already shown

main cp_w nx_trust_store_load_from_c sys_mmap nxa_die sys_write sys_exit nxa_lock_take nxa_lock_addr sys_write ↻ nxa_lock_give nxa_lock_addr ↻ nxa_report_overrun sys_write ↻ nxa_dump_printable sys_write ↻ nxa_dump_sizes sys_write ↻ sys_read_file sys_openat_rd sys_lseek sys_mmap ↻ sys_read sys_munmap sys_close nx_nss_certdata_parse sys_mmap ↻ _pat_class_cert _pat_value_octal _pat_end _find_newline _starts_with _parse_octal_line sys_mmap ↻ _is_space _decode_octal_escape _is_octal cp_wn cp_w ↻

structs

none

consts

21const CP_STATUSB: i64 = 8192 // /proc/self/status read buffer
22const CP_PAGECAP: i64 = 8388608 // page fixture cap == the crawler's CI_RAWCAP (largest raw file it reads)
23const CP_TXTCAP: i64 = 1048576 // html_to_text output cap (crawler docs are <= ~900 KB)
24const CP_FEEDCAP: i64 = 65536 // feed extractor output cap (== its own K_MAGIC_65536)
25const CP_URLCAP: i64 = 2048 // discovered feed url
26const CP_EDGES: i64 = 1024 // olh_scan edge slots (== OLH_MAXEDGE)
27const CP_ITERS: i64 = 24 // default steady-state iterations per stage
28const CP_CERTDATA: *u8 = "/tmp/mozilla_certdata.txt"
29const CP_FETCH_HOPS: i64 = 5

functions

31func cp_w(s: *u8) -> i64 { var n: i64 = 0; while s[n] != (0 as u8) { n = n + 1 } sys_write(1, s, n); return 0 }
called by 3: cp_wncp_reportmain
33func cp_wn(v: i64) -> i64
called by 2: cp_reportmain calls 1: cp_w
49func cp_vmsize() -> i64
84func cp_slen(s: *u8) -> i64 { var n: i64 = 0; while s[n] != (0 as u8) { n = n + 1 } return n }
87func cp_report(name: *u8, warm_kb: i64, steady_kb: i64, iters: i64) -> i64
calls 2: cp_wcp_wn
97func main(argc: i64, argv: *i64) -> i64