code wiki / _hdl_build / nx_crawl_callee_probe.nx
nx_crawl_callee_probe.nx
buildroot/runtime/_hdl_build/nx_crawl_callee_probe.nx
about
nx_crawl_callee_probe.nx -- PER-CALLEE VmSize PROBE for the crawler's per-page pipeline (debt 1787082132).
The +282 MB/min ingest growth survived the seg-store release-list fix; the named per-page mappings
(wc_harvest 8 KiB + olh_scan ~9.5 KiB) explain only ~20 KB/page, so the dominant allocator is in the
per-page CALLEES. This organ MEASURES each callee instead of reading it (the debt row's own law:
"do NOT hoist by reading"): run the SAME page bytes through each stage N times and diff
/proc/self/status VmSize around the loop. Per stage it prints BOTH numbers that matter:
warm_kb = VmSize cost of the FIRST call (one-time lazy tables/policy loads -- not a leak)
steady per-call = (delta over N calls after warm) / N -- the number that scales with pages/min
Stages: H2T (nx_html_to_text), FEED (nx_feed_discover + nx_feed_extract), OLH (olh_scan),
FETCH (nx_https_fetch_follow -- only when /tmp/mozilla_certdata.txt exists AND the arg is a url;
otherwise that stage prints UNOBSERVABLE and abstains rather than acquitting).
argv: nx_crawl_callee_probe <page-file-or-https-url> [iters] (file mode = no network, deterministic)
Output: one CALLEE line per stage; the LAST line names the worst steady grower (positional anchor).
license_tier: ORIGINAL expect_exit: 0
dependencies 5 imports · 0 importers
imports: nx_x509_trust_store.nxnx_trust_store_load_from_certdata.nxnx_https_fetch_follow.nxnx_outlink_harvest.nxnx_feed_extract.nx
imported by: nobody (leaf or entry point)
call flow from main pre-order; caps 40 nodes / depth 6 declared; ↻ = already shown
structs
| none |
consts
| 21 | const CP_STATUSB: i64 = 8192 // /proc/self/status read buffer |
| 22 | const CP_PAGECAP: i64 = 8388608 // page fixture cap == the crawler's CI_RAWCAP (largest raw file it reads) |
| 23 | const CP_TXTCAP: i64 = 1048576 // html_to_text output cap (crawler docs are <= ~900 KB) |
| 24 | const CP_FEEDCAP: i64 = 65536 // feed extractor output cap (== its own K_MAGIC_65536) |
| 25 | const CP_URLCAP: i64 = 2048 // discovered feed url |
| 26 | const CP_EDGES: i64 = 1024 // olh_scan edge slots (== OLH_MAXEDGE) |
| 27 | const CP_ITERS: i64 = 24 // default steady-state iterations per stage |
| 28 | const CP_CERTDATA: *u8 = "/tmp/mozilla_certdata.txt" |
| 29 | const CP_FETCH_HOPS: i64 = 5 |
functions
| 31 | func cp_w(s: *u8) -> i64 { var n: i64 = 0; while s[n] != (0 as u8) { n = n + 1 } sys_write(1, s, n); return 0 } |
| 33 | func cp_wn(v: i64) -> i64 |
| 49 | func cp_vmsize() -> i64 |
| 84 | func cp_slen(s: *u8) -> i64 { var n: i64 = 0; while s[n] != (0 as u8) { n = n + 1 } return n } |
| 87 | func cp_report(name: *u8, warm_kb: i64, steady_kb: i64, iters: i64) -> i64 |
| 97 | func main(argc: i64, argv: *i64) -> i64 |