code wiki / _hdl_build / nx_outlink_harvest.nx
nx_outlink_harvest.nx
buildroot/runtime/_hdl_build/nx_outlink_harvest.nx
about
nx_outlink_harvest.nx — shared outlink -> link-graph-edge harvester (P1 PageRank feed). LIB (no main).
Extracted (by copy) from nx_web_crawl_step.wc_harvest so BOTH the live crawler AND nx_cc_ingest emit the same
row shape: out:<ci_hash(page_url)> = packed i64 ci_hash(target_url). Node identity = ci_hash(url) ==
dss_urlcid(url) at serve time, so an outlink resolves to a graph node with zero extra index; serve-time
authority = content_cid -> url:<cid> -> ci_hash(url) -> pr:<cid>. ABSOLUTE cross-site links (the authority-
bearing ones) produce byte-identical cids to the crawler.
Three intentional improvements over the crawler's inline copy (all to backport when the crawler adopts this
lib): (1) root-relative "/path" resolves HOST-based (scheme://host + path) not base+path — the crawler's
concatenation makes ".../Trust_law" + "/about" a phantom node; (2) edge capture is DECOUPLED from the frontier
cap (the crawler stops recording edges once discbox hits WC_MAXDISC — a graph must not depend on frontier
state); (3) one REUSED url buffer instead of an mmap per outlink (bounds address-space growth for CC-scale
ingests of thousands of pages). license_tier: ORIGINAL
dependencies 1 imports · 5 importers
imports: nx_corpus_ingest.nx
imported by: nx_cc_ingest.nxnx_cc_warc_ingest.nxnx_outlink_harvest_gate.nxnx_page_ingest.nxnx_web_crawl_step.nx
structs
| none |
consts
| 15 | const OLH_MAGIC_8192: i64 = 8192 |
| 17 | const OLH_MAXURL: i64 = 1500 // == the crawler's WC_MAXURL (measured junk/tracking-monster threshold) so |
| 19 | const OLH_MAXEDGE: i64 = 1024 |
functions
| 21 | func olh_len(s: *u8) -> i64 { var n: i64 = 0; while s[n] != (0 as u8) { n = n + 1 } return n } |
| 24 | func olh_hassub(u: *u8, ul: i64, lit: *u8) -> i64 |
| 38 | func olh_endswith(u: *u8, ul: i64, lit: *u8) -> i64 calls 1: olh_len |
| 53 | func olh_urlok_q(u: *u8, ul: i64, allowq: i64) -> i64 |
| 111 | func olh_urlok(u: *u8, ul: i64) -> i64 { return olh_urlok_q(u, ul, 0) } |
| 113 | func olh_outkey(cid: i64, out: *u8) -> i64 |
| 132 | func olh_resolve_root(base: *u8, hp: *u8, clen: i64, scratch: *u8) -> i64 |
| 158 | func olh_scan(hh: *u8, hlen: i64, base: *u8, blen: i64, edgebuf: *i64, maxedge: i64) -> i64 |