nx_web_crawl_index.nx
buildroot/runtime/nx_web_crawl_index.nx
about
nx_web_crawl_index.nx -- LIVE external crawl: fetch FRESH URLs on demand over the sovereign TLS-1.3 stack
(no browser/curl), normalize each page to one `id<TAB>text` line, and emit the offsite corpus the reusable
engine indexes. DATA-DRIVEN: the URLs come from a seedlist (knowledge/index/web_seedlist.tsv, lines
"id<TAB>full_url") -- point it at any URLs and it crawls + builds the searchable corpus. This is the LIVE
half of external search (vs nx_external_corpus which re-uses already-fetched .raw). Pipeline after this:
nx_onsite_index web_seed_corpus.tsv https://en.wikipedia.org/wiki/ webcrawl.idx webcrawl.manifest
expect_exit: 0 license_tier: ORIGINAL (fetch machinery cloned from nx_onsite_research_fetch)
dependencies 11 imports · 0 importers
diagram shows first 10 each side; +1 more imports, +0 more importers in the complete lists below.
imports: nx_syscalls.nxnx_itoa_lib.nxnx_x509_trust_store.nxnx_trust_store_load_from_certdata.nxnx_tls13_client_validate_certificate.nxnx_tls13_client_session_run.nxnx_https_url_for_fetch.nxnx_https_url_connect.nxnx_https_get.nxnx_https_get_complete.nxnx_http_response_parse.nx
imported by: nobody (leaf or entry point)
call flow from main pre-order; caps 40 nodes / depth 6 declared; ↻ = already shown
structs
| none |
consts
| 19 | const WC_MAGIC_4194304: i64 = 4194304 |
| 21 | const WC_TEXTCAP: i64 = 65536 |
| 22 | const WC_SEED: *u8 = "knowledge/index/web_seedlist.tsv" as *u8 |
| 23 | const WC_OUT: *u8 = "knowledge/index/web_seed_corpus.tsv" as *u8 |
functions
| 25 | func wc_puts(s: *u8) -> i64 { var n: i64=0; while s[n]!=(0 as u8){n=n+1} sys_write(1,s,n); return 0 } |
| 30 | func wc_num(v: i64) -> i64 { nxi_out(v); return 0 } |
| 31 | func wc_strlen(s: *u8) -> i64 { var n: i64=0; while s[n]!=(0 as u8){n=n+1} return n } called by 1: wc_fetch_emit |
| 34 | func wc_path_off(url: *u8, n: i64) -> i64 called by 1: wc_fetch_emit |
| 46 | func wc_fetch_emit(store: *TrustStore, full_url: *u8, id: *u8, ofd: i64, tbuf: *u8) -> i64 called by 1: main calls 14: wc_strlenwc_path_offsys_mmapnx_url_newnx_https_url_for_fetchnx_https_url_connect+8 |
| 94 | func main() -> i64 |