code wiki / _hdl_build / nx_corpus_ingest.nx

nx_corpus_ingest.nx

buildroot/runtime/_hdl_build/nx_corpus_ingest.nx

11715 B220 linesdepth 4pulls 9 transitivereach 17 importersview sourcekind tooltopic corpus
docsdependenciesstructsconstsfunctions

about

nx_corpus_ingest.nx -- SOVEREIGN corpus ingestion: a DIRECTORY of documents -> a domain's PUBLIC seg_store shard, searchable at once (ss_write_seg builds .terms per segment; the store IS the index -- no tsv, no derived artifact). This is the front door for ALL THREE nishi-search fronts (operator 2026-07-03): library: nx_corpus_ingest knowledge/library nishifamily.com (.txt/.md -> /doc text views) site pages: nx_corpus_ingest <pagesdir> andelinwest.com / (.html -> text + url:<cid> row, so the SERP result links to the REAL page instead of /doc) web: the crawler drops fetched pages in a dir -> same ingest with absolute-URL prefix rows Mechanics: getdents64 walk (flat dir) -> .txt/.md raw, .html/.htm via nx_html_to_text (script/style suppressed, entities decoded) -> cid = the same polynomial hash nx_dp_tsv_migrate uses (content-addressed, idempotent) -> skip-if-already-present (re-runs are clean) -> RAW ss_add doc:<cid> (+ url:<cid>) batched into 1MB writer segments, each committed under a FRESH segid (manifest max + 1 -- never clobbers existing segments). license_tier: ORIGINAL

dependencies 2 imports · 13 importers

nx_docportal_search_seg.nx nx_html_to_text.nx nx_corpus_ingest.nx nx_bulk_index.nx nx_cc_ingest.nx nx_cc_warc_ingest.nx nx_corpus_ingest_gate.nx nx_outlink_harvest.nx nx_page_ingest.nx nx_pagerank_build.nx nx_trusted_flag.nx nx_url_index.nx nx_web_crawl_step.nx

diagram shows first 10 each side; +0 more imports, +3 more importers in the complete lists below.

imports: nx_docportal_search_seg.nxnx_html_to_text.nx

imported by: nx_bulk_index.nxnx_cc_ingest.nxnx_cc_warc_ingest.nxnx_corpus_ingest_gate.nxnx_outlink_harvest.nxnx_page_ingest.nxnx_pagerank_build.nxnx_trusted_flag.nxnx_url_index.nxnx_web_crawl_step.nxnx_web_ingest.nxnx_web_purge_slop.nxnx_web_search_probe.nx

call flow from main pre-order; caps 40 nodes / depth 6 declared; ↻ = already shown

main ci_puts ci_run dss_prefix ss_manifest_file sys_mmap ss_cat ss_readall sys_openat_rd sys_lseek sys_mmap ↻ sys_read sys_close ss_open ss_open2 sys_mmap ↻ ss_manifest_dyn ss_manifest_file_dyn sys_mmap ↻ ss_cat ↻ ss_readall ↻ sys_munmap ss_cat ↻ ss_loadfile sys_map_file sys_openat_rd ↻ sys_lseek ↻ sys_close ↻ ss_readall ↻ ss_load_aux2 sys_mmap ↻ ss_cat ↻ ss_loadfile ↻ ss_r32 sys_munmap ↻ ssl_total_keys ss_r32 ↻ ssl_pow2 ssl_build ss_r32 ↻

structs

none

consts

15const CI_MAGIC_1125899906842597: i64 = 1125899906842597
16const CI_MAGIC_131072: i64 = 131072
17const CI_MAGIC_1024: i64 = 1024
18const CI_MAGIC_65536: i64 = 65536
20const CI_DOCCAP: i64 = 900000 // per-doc byte cap (writer segments are 1MB; a doc must fit with headroom)
21const CI_MINDOC: i64 = 20 // skip empty-ish extractions
22const CI_RAWCAP: i64 = 8388608 // largest raw file we read

functions

24func ci_puts(s: *u8) -> i64 { var n: i64 = 0; while s[n] != (0 as u8) { n = n + 1 } sys_write(1, s, n); return 0 }
called by 2: ci_runmain
25func ci_num(v: i64) -> i64
called by 2: ci_runmain
34func ci_len(s: *u8) -> i64 { var n: i64 = 0; while s[n] != (0 as u8) { n = n + 1 } return n }
called by 3: ci_endsci_containsci_run
35func ci_cat(d: *u8, o: i64, s: *u8) -> i64 { var i: i64 = 0; while s[i] != (0 as u8) { d[o + i] = s[i]; i = i + 1 } return o + i }
called by 1: ci_run
38func ci_hash(s: *u8, n: i64) -> i64
called by 11: mainmainmainci_runpi_onemain+5
46func ci_mkurlkey(cid: i64, out: *u8) -> i64
called by 12: mainmainmainci_runmainpi_one+6
60func ci_ends(name: *u8, suf: *u8) -> i64
called by 1: ci_kind calls 1: ci_len
68func ci_contains(name: *u8, ndl: *u8) -> i64
called by 1: ci_kind calls 1: ci_len
83func ci_kind(name: *u8) -> i64
called by 1: ci_run calls 2: ci_containsci_ends
95func ci_run(dir: *u8, domain: *u8, urlprefix: *u8, counts: *i64) -> i64
called by 2: mainmain calls 21: dss_prefixss_manifest_filess_openci_putsss_beginsys_getdents64+15
203func main(argc: i64, argv: *i64) -> i64
calls 3: ci_putsci_runci_num