code wiki / _hdl_build / nx_outlink_harvest.nx

nx_outlink_harvest.nx

buildroot/runtime/_hdl_build/nx_outlink_harvest.nx

11769 B228 linesdepth 5pulls 10 transitivereach 6 importersview sourcekind library
docsdependenciesstructsconstsfunctions

about

nx_outlink_harvest.nx — shared outlink -> link-graph-edge harvester (P1 PageRank feed). LIB (no main). Extracted (by copy) from nx_web_crawl_step.wc_harvest so BOTH the live crawler AND nx_cc_ingest emit the same row shape: out:<ci_hash(page_url)> = packed i64 ci_hash(target_url). Node identity = ci_hash(url) == dss_urlcid(url) at serve time, so an outlink resolves to a graph node with zero extra index; serve-time authority = content_cid -> url:<cid> -> ci_hash(url) -> pr:<cid>. ABSOLUTE cross-site links (the authority- bearing ones) produce byte-identical cids to the crawler. Three intentional improvements over the crawler's inline copy (all to backport when the crawler adopts this lib): (1) root-relative "/path" resolves HOST-based (scheme://host + path) not base+path — the crawler's concatenation makes ".../Trust_law" + "/about" a phantom node; (2) edge capture is DECOUPLED from the frontier cap (the crawler stops recording edges once discbox hits WC_MAXDISC — a graph must not depend on frontier state); (3) one REUSED url buffer instead of an mmap per outlink (bounds address-space growth for CC-scale ingests of thousands of pages). license_tier: ORIGINAL

dependencies 1 imports · 5 importers

nx_corpus_ingest.nx nx_outlink_harvest.nx nx_cc_ingest.nx nx_cc_warc_ingest.nx nx_outlink_harvest_gate.nx nx_page_ingest.nx nx_web_crawl_step.nx

imports: nx_corpus_ingest.nx

imported by: nx_cc_ingest.nxnx_cc_warc_ingest.nxnx_outlink_harvest_gate.nxnx_page_ingest.nxnx_web_crawl_step.nx

structs

none

consts

15const OLH_MAGIC_8192: i64 = 8192
17const OLH_MAXURL: i64 = 1500 // == the crawler's WC_MAXURL (measured junk/tracking-monster threshold) so
19const OLH_MAXEDGE: i64 = 1024

functions

21func olh_len(s: *u8) -> i64 { var n: i64 = 0; while s[n] != (0 as u8) { n = n + 1 } return n }
24func olh_hassub(u: *u8, ul: i64, lit: *u8) -> i64
called by 1: olh_urlok_q calls 1: olh_len
38func olh_endswith(u: *u8, ul: i64, lit: *u8) -> i64
calls 1: olh_len
53func olh_urlok_q(u: *u8, ul: i64, allowq: i64) -> i64
called by 2: wc_urlokmain calls 1: olh_hassub
111func olh_urlok(u: *u8, ul: i64) -> i64 { return olh_urlok_q(u, ul, 0) }
113func olh_outkey(cid: i64, out: *u8) -> i64
132func olh_resolve_root(base: *u8, hp: *u8, clen: i64, scratch: *u8) -> i64
158func olh_scan(hh: *u8, hlen: i64, base: *u8, blen: i64, edgebuf: *i64, maxedge: i64) -> i64