code wiki / (root) / nx_fetch.nx

nx_fetch.nx

buildroot/runtime/nx_fetch.nx

10643 B221 linesdepth 21pulls 146 transitivereach 0 importersview sourcekind tooltopic fetch
docsdependenciesstructsconstsfunctions

about

nx_fetch.nx -- THE sovereign manifest-driven research fetcher. MERGES the N copy-pasted one-time nx_*_research_fetch organs into ONE data-driven tool (rule 11 data-driven + rule 15 DRY): new research is a MANIFEST ROW, never a new organ. Reads a TSV manifest (each line: url<TAB>name), and for each row: 1. ensures the raw archive knowledge/fetched/<name>.raw (fetch over sovereign TLS-1.3 if absent; idempotent), 2. strips it to clean plain text -> knowledge/library/<name>.txt (the dir the librarian nx_library_harvest_v2 walks for .txt) so it BANKS into the durable index, queryable via nx_search_cli. Plaintext strip is purpose-fit for the index (drop tags, suppress <script>/<style>, collapse whitespace) -- NOT the browser's nx_html_to_text (which emits GUI link-markers wrong for a corpus). 100% sovereign. usage: nx_fetch <manifest.tsv> expect_exit: 0 license_tier: ORIGINAL

dependencies 4 imports · 0 importers

nx_syscalls.nx nx_x509_trust_store.nx nx_trust_store_load_from_certdata. nx_https_fetch_follow.nx nx_fetch.nx

imports: nx_syscalls.nxnx_x509_trust_store.nxnx_trust_store_load_from_certdata.nxnx_https_fetch_follow.nx

imported by: nobody (leaf or entry point)

call flow from main pre-order; caps 40 nodes / depth 6 declared; ↻ = already shown

main puts sys_write nx_trust_store_load_from_c sys_mmap sys_read_file sys_openat_rd sys_lseek sys_mmap ↻ sys_read sys_close nx_nss_certdata_parse sys_mmap ↻ _pat_class_cert _pat_value_octal _pat_end _find_newline _starts_with _parse_octal_line sys_mmap ↻ _is_space _decode_octal_escape _is_octal trust_store_alloc sys_mmap ↻ nx_x509_trust_store_load sys_mmap ↻ x509_parse sys_mmap ↻ asn1_cursor_init asn1_expect_tag sys_mmap ↻ asn1_read_tlv_header asn1_read_tag asn1_read_length asn1_read_length ↻ x509_read_tlv sys_mmap ↻ asn1_expect_tag ↻ x509_read_alg_id

structs

none

consts

14const K_MAGIC_4096: i64 = 4096
15const K_MAGIC_4194304: i64 = 4194304
16const K_MAGIC_8388608: i64 = 8388608
17const K_MAGIC_1024: i64 = 1024

functions

19func puts(s: *u8) -> i64 { var n: i64 = 0; while s[n] != (0 as u8) { n = n + 1 } sys_write(1, s, n); return 0 }
called by 2: fetch_to_libmain calls 1: sys_write
20func putn(v: i64) -> i64
called by 2: fetch_to_libmain calls 2: sys_writesys_mmap
30func have_file(path: *u8) -> i64 { let fd: i64 = sys_openat_rd(path); if fd < 0 { return 0 } sys_close(fd); return 1 }
called by 1: fetch_to_lib calls 2: sys_openat_rdsys_close
31func scat(dst: *u8, o: i64, s: *u8) -> i64 { var i: i64 = 0; while s[i] != (0 as u8) { dst[o + i] = s[i]; i = i + 1 } return o + i }
called by 1: bpath
32func bpath(dst: *u8, dir: *u8, name: *u8, suf: *u8) -> i64
called by 1: fetch_to_lib calls 1: scat
35func save_file(path: *u8, buf: *u8, n: i64) -> i64
called by 1: fetch_to_lib calls 3: sys_openat_wrsys_writesys_close
44func tag_eq_ci(src: *u8, off: i64, n: i64, lit: *u8, litlen: i64) -> i64
called by 2: tag_isstrip_html
57func tag_is(src: *u8, off: i64, n: i64, lit: *u8, litlen: i64) -> i64
called by 1: is_block_tag calls 1: tag_eq_ci
74func is_block_tag(src: *u8, off: i64, n: i64) -> i64
called by 1: strip_html calls 1: tag_is
103func strip_html(src: *u8, n: i64, out: *u8, cap: i64) -> i64
called by 1: fetch_to_lib calls 2: tag_eq_ciis_block_tag
155func fetch_to_lib(url: *u8, name: *u8, store: *TrustStore, out: *u8, cap: i64) -> i64
called by 1: main calls 8: sys_mmapbpathhave_filesys_read_fileputsputn+2
179func main(argc: i64, argv: *i64) -> i64
calls 8: putsnx_trust_store_load_from_certdatasys_mkdirputntrust_store_countsys_mmap+2