code wiki / _hdl_build / nx_domain_map.nx

nx_domain_map.nx

buildroot/runtime/_hdl_build/nx_domain_map.nx

34893 B671 linesdepth 21pulls 153 transitivereach 1 importersview sourcekind librarytopic domain
docsdependenciesstructsconstsfunctions

about

nx_domain_map.nx -- DOMAIN MAP: every URL under a domain BEFORE crawling it -- the /compare/webscraping R8 contracts dm_scan (the composed census) and dm_soft404 (the fingerprint that keeps a 200-that-is-really-404 out of it). Concept source: crawl4ai DomainMapper (eight discovery sources + soft-404 fingerprint). Sovereign form composes the estate's own parsers, never a second one: robots nx_sitemap_extract.sm_robots_sitemaps (Sitemap: lines) + dm_robots_paths (Allow/Disallow paths a site ACKNOWLEDGES exist) ............................................................ source=robots sitemap sm_extract_locs over every sitemap, RECURSING sitemapindex (sm_is_subsitemap) and inflating .gz bodies (nx_gzip_wrap) ........................................................... source=sitemap feed nx_feed_extract on the homepage (feed href -> items) ............................ source=feed wayback the Internet Archive CDX API, original urls, 200s only ............................ source=wayback crt crt.sh certificate-transparency JSON -> subdomains ............................... source=crt (kind=host) probe knowledge/domain_map_probe.conf paths (robots-checked) classified against the SOFT-404 FINGERPRINT taken first from a path that cannot exist ............................................. source=probe EVERY ROW IS APPENDED AS IT IS DECIDED (a run killed at 95 pct keeps 95 pct; the estate's long-sweep law), to knowledge/status/domain_map/<domain>.tsv: ts TAB source TAB kind(url|host|path) TAB verdict TAB value. The final line prints the partition (rows per source) and it SUMS to rows written; fetches used, refused (robots), oversize (a <loc> past the url slot) and truncated (a body that filled the cap) are counted, never silent. PACED: every fetch goes through nx_crawl_pace (per host, Retry-After, robots Crawl-delay); probes are refused when robots forbids the path, and a refusal is a ROW, not a silence. usage: nx_domain_map <domain> [max_fetches] [out.tsv] (rows and caps: knowledge/domain_map.conf) exit: 0 rows written | 2 usage | 3 trust store | 4 nothing reachable (0 fetches succeeded) license_tier: ORIGINAL No hw writes (Rule 26).

dependencies 11 imports · 1 importers

nx_syscalls.nx nx_lane_conf.nx nx_x509_trust_store.nx nx_trust_store_load_from_certdata. nx_https_fetch_follow.nx nx_gzip_wrap.nx nx_sitemap_extract.nx nx_feed_extract.nx nx_robots.nx nx_crawl_pace.nx nx_domain_map.nx nx_domain_map_gate.nx

diagram shows first 10 each side; +1 more imports, +0 more importers in the complete lists below.

imports: nx_syscalls.nxnx_lane_conf.nxnx_x509_trust_store.nxnx_trust_store_load_from_certdata.nxnx_https_fetch_follow.nxnx_gzip_wrap.nxnx_sitemap_extract.nxnx_feed_extract.nxnx_robots.nxnx_crawl_pace.nxnx_itoa_lib.nx

imported by: nx_domain_map_gate.nx

structs

none

consts

35const DM_CONF: *u8 = "knowledge/domain_map.conf"
36const DM_PROBES: *u8 = "knowledge/domain_map_probe.conf"
37const DM_OUTDIR: *u8 = "knowledge/status/domain_map"
38const DM_CERTDATA: *u8 = "data/mozilla_certdata.txt"
40const DM_DEF_MAX_FETCHES: i64 = 64 // one map is a reconnaissance, not a crawl: 64 fetches covers robots + a sitemap tree + probes
41const DM_DEF_WAYBACK_LIMIT: i64 = 500 // CDX rows requested; the API accepts a limit and 500 keeps the body under the fetch cap
42const DM_DEF_MAX_SITEMAPS: i64 = 32 // sitemap files followed through a sitemapindex before the map says so and stops
43const DM_DEF_MAX_LOCS: i64 = 50000 // sitemaps.org's own per-file ceiling; a file past it is out of spec
44const DM_DEF_MAX_HOSTS: i64 = 256 // distinct subdomains kept from crt.sh; the rest are counted as oversize
45const DM_DEF_FETCH_CAP: i64 = 8388608 // 8 MiB body cap per fetch (announced when filled); crt.sh answers can be MBs
46const DM_DEF_SOFT404_TOL_PERMIL: i64 = 100 // crawl4ai Soft404Fingerprint compares status, title and content length; +-10 pct length
47const DM_DEF_MAX_HOPS: i64 = 6 // redirect hops, the crawler's own number
48const DM_URL_SLOT: i64 = 2048 // one url slot; sitemaps.org allows a 2048-char <loc>, longer is out of spec
49const DM_HOST_SLOT: i64 = 256 // one host slot (a DNS name is at most 253 bytes)
50const DM_PATH_SLOT: i64 = 512 // one probe or robots path slot
51const DM_TS_BUF: i64 = 32
52const DM_LINE_CAP: i64 = 4096
53const DM_SMALL: i64 = 1024
54const DM_TRUST_ANCHORS: i64 = 512 // the crawler's own trust-store sizing
55const DM_TRUST_PARSE_CAP: i64 = 4194304
56const DM_MODE_0644: i64 = 420
57const DM_MODE_0755: i64 = 493
58const DM_GZ_ID1: i64 = 31
59const DM_GZ_ID2: i64 = 139
60const DM_UALEN: i64 = 8 // len("nishibot"), the crawler's UA token
61const DM_TAB: i64 = 9
62const DM_NL: i64 = 10
63const DM_CR: i64 = 13
64const DM_HASH: i64 = 35
65const DM_QUOTE: i64 = 34
66const DM_BACKSLASH: i64 = 92
67const DM_SLASH: i64 = 47
68const DM_COLON: i64 = 58
69const DM_LT: i64 = 60
70const DM_GT: i64 = 62
71const DM_DIGIT0: i64 = 48
72const DM_DIGIT9: i64 = 57
73const DM_HEX_BASE: i64 = 16
74const DM_HASH_SEED: i64 = 5381
75const DM_HASH_MASK: i64 = 0x7fffffffffffffff
76const DM_PROBE_DEFAULTS: i64 = 15
78const DM_V_DECLARED: *u8 = "DECLARED"
79const DM_V_LISTED: *u8 = "LISTED"
80const DM_V_ARCHIVED: *u8 = "ARCHIVED"
81const DM_V_CERT: *u8 = "CERT"
82const DM_V_LIVE: *u8 = "LIVE"
83const DM_V_SOFT404: *u8 = "SOFT404"
84const DM_V_MISSING: *u8 = "MISSING"
85const DM_V_BLOCKED: *u8 = "BLOCKED"
86const DM_V_UNREACHABLE: *u8 = "UNREACHABLE"
87const DM_V_ROBOTS_REFUSED: *u8 = "ROBOTS-REFUSED"
88const DM_V_FINGERPRINT: *u8 = "FINGERPRINT"
89const DM_HTTP_OK: i64 = 200
90const DM_HTTP_NOTFOUND: i64 = 404
91const DM_HTTP_ERR_FLOOR: i64 = 400
92const DM_STATUS_BUDGET: i64 = 0 - 1 // status-box value meaning: this fetch was refused by OUR budget, the host was never asked
93const DM_V_BUDGET: *u8 = "BUDGET-REFUSED"

functions

123func dm_len(s: *u8) -> i64 { var n: i64 = 0; while s[n] != (0 as u8) { n = n + 1 } return n }
called by 2: dm_putsdm_crt_hosts
124func dm_puts(s: *u8) -> i64 { sys_write(1, s, dm_len(s)); return 0 }
calls 2: sys_writedm_len
125func dm_num(v: i64) -> i64 { nxi_out(v); return 0 }
calls 1: nxi_out
126func dm_cat(d: *u8, o: i64, s: *u8) -> i64 { var i: i64 = 0; var p: i64 = o; while s[i] != (0 as u8) { d[p] = s[i]; p = p + 1; i = i + 1 } d[p] = 0 as u8; return p }
called by 3: dm_rowdm_urldm_source_crt
127func dm_catn(d: *u8, o: i64, s: *u8, n: i64) -> i64 { var i: i64 = 0; var p: i64 = o; while i < n { d[p] = s[i]; p = p + 1; i = i + 1 } d[p] = 0 as u8; return p }
called by 1: dm_row
128func dm_streq(a: *u8, b: *u8) -> i64 { var i: i64 = 0; while a[i] == b[i] { if a[i] == (0 as u8) { return 1 } i = i + 1 } return 0 }
called by 3: dm_robots_pathsdm_crt_hostsmain
129func dm_lc(c: i64) -> i64 { if c >= 65 { if c <= 90 { return c + 32 } } return c }
called by 2: dm_title_hashdm_crt_hosts
130func dm_conf_one(key: *u8, dflt: i64) -> i64
called by 1: dm_load_conf calls 1: lc_geti
136func dm_load_conf() -> i64
called by 1: main calls 1: dm_conf_one
149func dm_hash(b: *u8, n: i64) -> i64
called by 1: dm_title_hash
157func dm_title_hash(html: *u8, n: i64) -> i64
called by 1: main calls 2: dm_lcdm_hash
176func dm_soft404_classify(fp_status: i64, fp_len: i64, fp_title: i64, status: i64, len: i64, title: i64, tol_permil: i64) -> i64
called by 2: dm_soft404main
187func dm_soft404(status: i64, len: i64, title: i64) -> i64
called by 1: main calls 1: dm_soft404_classify
192func dm_robots_paths(rob: *u8, n: i64, out: *u8, slot: i64, max: i64) -> i64
calls 3: sm_ci_atsm_copy_urldm_streq
229func dm_crt_hosts(js: *u8, n: i64, domain: *u8, out: *u8, slot: i64, max: i64, oversize: *i64) -> i64
called by 1: main calls 3: dm_lendm_lcdm_streq
284func dm_is_gzip(b: *u8, n: i64) -> i64
called by 1: main
291func dm_row(source: *u8, kind: *u8, verdict: *u8, value: *u8, vlen: i64) -> i64
called by 1: dm_url calls 7: sys_mmapccz_cat_numsys_now_realtime_secdm_catdm_catnsys_write+1
311func dm_fetch(url: *u8, store: *TrustStore, out: *u8, cap: i64, stbox: *i64) -> i64
called by 2: dm_urlmain calls 4: sys_mmappace_beforepace_aftersys_munmap
336func dm_url(dst: *u8, domain: *u8, path: *u8) -> i64
calls 4: dm_catccz_cat_numdm_fetchdm_row
343func dm_nonce_path(dst: *u8, domain: *u8) -> i64
353func dm_source_sitemaps(domain: *u8, store: *TrustStore, robots: *u8, rn: i64, body: *u8, cap: i64, stbox: *i64) -> i64
402func dm_source_feed(domain: *u8, store: *TrustStore, body: *u8, cap: i64, stbox: *i64, feedxml: *u8) -> i64
429func dm_source_wayback(domain: *u8, store: *TrustStore, body: *u8, cap: i64, stbox: *i64) -> i64
451func dm_source_crt(domain: *u8, store: *TrustStore, body: *u8, cap: i64, stbox: *i64) -> i64
475func dm_probe_paths_load(out: *u8, slot: i64, max: i64) -> i64
532func dm_source_probe(domain: *u8, store: *TrustStore, robots: *u8, rn: i64, body: *u8, cap: i64, stbox: *i64) -> i64
593func dm_scan(domain: *u8, store: *TrustStore, body: *u8, feedxml: *u8, robots: *u8, cap: i64, stbox: *i64) -> i64
618func main(argc: i64, argv: *i64) -> i64
671func wc_atoi_dm(s: *u8) -> i64 { var v: i64 = 0; var i: i64 = 0; while s[i] != (0 as u8) { let c: i64 = s[i] as i64; if c >= DM_DIGIT0 { if c <= DM_DIGIT9 { v = v * 10 + (c - DM_DIGIT0) } } i = i + 1 } return v }