code wiki / _hdl_build / nx_page_ingest.nx

nx_page_ingest.nx

buildroot/runtime/_hdl_build/nx_page_ingest.nx

12633 B239 linesdepth 21pulls 155 transitivereach 0 importersview sourcekind tooltopic page
docsdependenciesstructsconstsfunctions

about

nx_page_ingest.nx -- ENTITY-DEPTH live ingester: ONE operator-chosen seed page + up to N of its SAME-HOST outlinks -> the dp-web-pub- web shard. The targeted complement to the bulk CC pipeline (breadth) and the frontier crawler (which stalls on stale frontiers -- debt seq628): every URL here is FRESH (seed + its own live links), fetches are sequential + politely paced, the job is bounded. usage: nx_page_ingest <seed-url> [maxfollow] Seed is ingested UNCONDITIONALLY (operator chose it; only a minimal length floor applies); followed links pass the normal thin filter. Writes doc:/url: + out: edges (same node identity as serve/pagerank). No loc: row (live pages have no CC locator; absent loc is a supported state). exit 0 + verdict=PASS when the seed ingested (or was already present). license_tier: ORIGINAL

dependencies 6 imports · 0 importers

nx_corpus_ingest.nx nx_x509_trust_store.nx nx_trust_store_load_from_certdata. nx_https_fetch_follow.nx nx_outlink_harvest.nx nx_gzip_wrap.nx nx_page_ingest.nx

imports: nx_corpus_ingest.nxnx_x509_trust_store.nxnx_trust_store_load_from_certdata.nxnx_https_fetch_follow.nxnx_outlink_harvest.nxnx_gzip_wrap.nx

imported by: nobody (leaf or entry point)

call flow from main pre-order; caps 40 nodes / depth 6 declared; ↻ = already shown

main pi_puts pi_atoi nx_trust_store_load_from_c sys_mmap sys_read_file sys_openat_rd sys_lseek sys_mmap ↻ sys_read sys_close nx_nss_certdata_parse sys_mmap ↻ _pat_class_cert _pat_value_octal _pat_end _find_newline _starts_with _parse_octal_line sys_mmap ↻ _is_space _decode_octal_escape _is_octal trust_store_alloc sys_mmap ↻ nx_x509_trust_store_load sys_mmap ↻ x509_parse sys_mmap ↻ asn1_cursor_init asn1_expect_tag sys_mmap ↻ asn1_read_tlv_header asn1_read_tag asn1_read_length asn1_read_length ↻ x509_read_tlv sys_mmap ↻ asn1_expect_tag ↻ x509_read_alg_id

structs

none

consts

16const PI_MAGIC_1048576: i64 = 1048576
17const PI_MAGIC_4194304: i64 = 4194304
18const PI_MAGIC_8192: i64 = 8192
19const PI_MAGIC_2048: i64 = 2048
20const PI_MAGIC_1500: i64 = 1500
22const PI_MAXFOLLOW: i64 = 60 // hard cap on followed links per run (bounded job by design)
23const PI_FETCHCAP: i64 = 4194304 // page byte cap
24const PI_MINSEED: i64 = 100 // seed min extracted chars (operator chose it -- floor only)
25const PI_MINFOLLOW: i64 = 300 // followed-page min extracted chars (thin filter)
26const PI_PACE_MS: i64 = 500 // politeness delay between same-host fetches
27const PI_SEGCAP: i64 = 16777216 // writer segment size
33const PI_SEGID_SANE: i64 = 100000000

functions

35func pi_puts(s: *u8) -> i64 { var n: i64 = 0; while s[n] != (0 as u8) { n = n + 1 } sys_write(1, s, n); return 0 }
called by 2: pi_onemain
36func pi_num(v: i64) -> i64
called by 2: pi_onemain
45func pi_len(s: *u8) -> i64 { var n: i64 = 0; while s[n] != (0 as u8) { n = n + 1 } return n }
called by 2: pi_onemain
46func pi_atoi(s: *u8) -> i64
called by 1: main
52func pi_host(u: *u8, ul: i64, ob: *i64) -> i64
called by 1: pi_samehost
67func pi_samehost(a: *u8, al: i64, b: *u8, bl: i64) -> i64
called by 1: main calls 1: pi_host
79func pi_one(url: *u8, store: *TrustStore, h: *i64, w: *i64, prefix: *u8, segbox: *i64,
called by 1: main calls 11: pi_putspi_numnx_gzip_inflatenx_html_to_textci_hashdss_mkkey+5
127func main(argc: i64, argv: *i64) -> i64
calls 13: pi_putspi_atoinx_trust_store_load_from_certdatapi_numdss_prefixss_manifest_cap+7