code wiki / _hdl_build / nx_corpus_ingest.nx

nx_corpus_ingest.nx

buildroot/runtime/_hdl_build/nx_corpus_ingest.nx

36150 B681 linesdepth 5pulls 12 transitivereach 22 importersview sourcekind tooltopic corpus
docsdependenciesstructsconstsfunctions

about

nx_corpus_ingest.nx -- SOVEREIGN corpus ingestion: a DIRECTORY of documents -> a domain's PUBLIC seg_store shard, searchable at once (ss_write_seg builds .terms per segment; the store IS the index -- no tsv, no derived artifact). This is the front door for ALL THREE nishi-search fronts (operator 2026-07-03): library: nx_corpus_ingest knowledge/library nishifamily.com (.txt/.md -> /doc text views) site pages: nx_corpus_ingest <pagesdir> andelinwest.com / (.html -> text + url:<cid> row, so the SERP result links to the REAL page instead of /doc) web: the crawler drops fetched pages in a dir -> same ingest with absolute-URL prefix rows estate: nx_corpus_ingest tree <root> estate estate://<root>/ (E1, 2026-09-14: every source file under a tree, recursively, one document per file, its relative path as the url, its identifiers anchored, and a changed file REPLACING its previous document through a path row + tombstones) Mechanics: getdents64 walk (flat dir, or the recursive tree walk) -> .txt/.md raw, .html/.htm via nx_html_to_text (script/style suppressed, entities decoded) -> cid = the same polynomial hash nx_dp_tsv_migrate uses (content-addressed, idempotent) -> skip-if-already-present (re-runs are clean) -> RAW ss_add doc:<cid> (+ url:<cid>) batched into 1MB writer segments, each committed under a FRESH segid (the seg store's ss_next_segid -- never clobbers existing segments). ONE ingest body (ci_ingest_file) and ONE writer put (ci_put) serve both walks: two copies would agree the day they were written and drift on the next edit. license_tier: ORIGINAL

dependencies 3 imports · 15 importers

nx_docportal_search_seg.nx nx_html_to_text.nx nx_funchead_lib.nx nx_corpus_ingest.nx nx_bulk_index.nx nx_cc_ingest.nx nx_cc_warc_ingest.nx nx_corpus_ingest_gate.nx nx_fp_backfill.nx nx_outlink_harvest.nx nx_page_ingest.nx nx_pagerank_build.nx nx_sitededup.nx nx_trusted_flag.nx

diagram shows first 10 each side; +0 more imports, +5 more importers in the complete lists below.

imports: nx_docportal_search_seg.nxnx_html_to_text.nxnx_funchead_lib.nx

imported by: nx_bulk_index.nxnx_cc_ingest.nxnx_cc_warc_ingest.nxnx_corpus_ingest_gate.nxnx_fp_backfill.nxnx_outlink_harvest.nxnx_page_ingest.nxnx_pagerank_build.nxnx_sitededup.nxnx_trusted_flag.nxnx_url_index.nxnx_web_crawl_step.nxnx_web_ingest.nxnx_web_purge_slop.nxnx_web_search_probe.nx

call flow from main pre-order; caps 40 nodes / depth 6 declared; ↻ = already shown

main ci_puts ci_len ci_contains ci_len ↻ ci_ingest_source_tree dss_prefix ci_state_new ss_begin ss_begin_cap sys_mmap nxa_die nxa_lock_take nxa_lock_give nxa_report_overrun ci_next_segid ss_next_segid ss_max_segid sys_mmap ↻ ss_cat ss_readall ss_segid_ok ss_open ss_open2 ss_open3 sys_now_us sys_mmap ↻ ss_manifest_dyn ss3_admit ss3_load_range ssl_total_keys_mask ss3_ann ss_close_seg ss3_move ss3_remap_table sys_madvise ss_tr ss3_insert_new sys_munmap ss3_tomb_count

structs

none

consts

21const CI_MAGIC_1125899906842597: i64 = 1125899906842597
22const CI_MAGIC_131072: i64 = 131072
23const CI_MAGIC_1024: i64 = 1024
24const CI_MAGIC_65536: i64 = 65536
26const CI_DOCCAP: i64 = 900000 // per-doc byte cap (writer segments are 1MB; a doc must fit with headroom)
27const CI_MINDOC: i64 = 20 // skip empty-ish extractions
28const CI_RAWCAP: i64 = 8388608 // largest raw file we read
29const CI_DT_DIR: i64 = 4 // dirent d_type of a directory
30const CI_WORD: i64 = 8
31const CI_BOX: i64 = 16
32const CI_KEYBUF: i64 = 64
33const CI_PREFIXBUF: i64 = 512
34const CI_DIGIT0: i64 = 48
35const CI_DIGIT9: i64 = 57
36const CI_DECBUF: i64 = 24
37const CI_LF: i64 = 10
38const CI_KIND_LIVE: i64 = 1 // seg-store entry kinds: a live row, and a tombstone that retires a key
39const CI_KIND_TOMB: i64 = 2
40const CI_DOC_JOURNAL: i64 = 3 // ci_kind_source: an append-only journal (.jrnl .log), ingested TAIL-anchored (E3)
41const CI_DOC_BOARD: i64 = 4 // ci_kind_source: a compare board (.plan .matrix), ingested whole AND row by row (E3b)
42const CI_ROW_SEP: *u8 = "#" // a row document's url is the board url, this separator, the row index
43const CI_ROW_RUNG: *u8 = "rung|" // a rung row carries the definition token of its id, the way a function head does
44const CI_CONF_PIPE: i64 = 124 // estate_corpora.conf field separator
45const CI_CONF_HASH: i64 = 35 // estate_corpora.conf comment lead byte
46const CI_CR: i64 = 13 // a CRLF conf leaves this as the last byte of a row; stripped, never parsed as data
47const CI_STATE_WORDS: i64 = 12
48const CI_ANCHOR_PAD: i64 = 64 // the anchor tail is never longer than the text it came from; this is the newline and slack
49const CI_TREE_MAXDEPTH: i64 = 12 // a source tree deeper than this is not a source tree; announced, never silent
250const CI_DEF_MINHEAD: i64 = 7 // the shortest head: func, a space, a one-byte name, the paren

functions

51func ci_puts(s: *u8) -> i64 { var n: i64 = 0; while s[n] != (0 as u8) { n = n + 1 } sys_write(1, s, n); return 0 }
called by 7: ci_state_finishci_putci_runci_walk_treeci_ingest_source_treeci_ingest_boards+1
52func ci_len(s: *u8) -> i64 { var n: i64 = 0; while s[n] != (0 as u8) { n = n + 1 } return n }
called by 7: ci_endsci_containsci_name_eqci_def_anchorsci_ingest_docci_ingest_board_rows+1
53func ci_cat(d: *u8, o: i64, s: *u8) -> i64 { var i: i64 = 0; while s[i] != (0 as u8) { d[o + i] = s[i]; i = i + 1 } return o + i }
called by 6: ci_mkkey_pfxci_ingest_docci_ingest_board_rowsci_runci_walk_treemain
56func ci_dec(v: i64, out: *u8) -> i64
called by 4: ci_numci_mkkey_pfxci_ingest_docci_ingest_board_rows
66func ci_num(v: i64) -> i64
called by 3: ci_putci_ingest_boardsmain calls 1: ci_dec
75func ci_atoi(p: *u8, n: i64) -> i64
called by 1: ci_ingest_doc
83func ci_hash(s: *u8, n: i64) -> i64
called by 7: mainmainmainci_ingest_docpi_onemain+1
92func ci_mkkey_pfx(pfx: *u8, id: i64, out: *u8) -> i64
called by 4: ci_mkurlkeyci_mkfpkeyci_mkfekeyci_mkpthkey calls 2: ci_catci_dec
99func ci_mkurlkey(cid: i64, out: *u8) -> i64 { return ci_mkkey_pfx("url:" as *u8, cid, out) }
called by 9: mainmainmainci_ingest_docmainpi_one+3 calls 1: ci_mkkey_pfx
111func ci_mkfpkey(cid: i64, out: *u8) -> i64 { return ci_mkkey_pfx("fp:" as *u8, cid, out) }
called by 1: main calls 1: ci_mkkey_pfx
116func ci_mkfekey(cid: i64, out: *u8) -> i64 { return ci_mkkey_pfx("fe:" as *u8, cid, out) }
calls 1: ci_mkkey_pfx
120func ci_mkpthkey(pathhash: i64, out: *u8) -> i64 { return ci_mkkey_pfx("pth:" as *u8, pathhash, out) }
called by 1: ci_ingest_doc calls 1: ci_mkkey_pfx
122func ci_ends(name: *u8, suf: *u8) -> i64
called by 2: ci_kindci_kind_source calls 1: ci_len
130func ci_contains(name: *u8, ndl: *u8) -> i64
called by 4: ci_kindci_kind_sourceci_name_eqmain calls 1: ci_len
145func ci_kind(name: *u8) -> i64
called by 1: ci_run calls 2: ci_containsci_ends
156func ci_kind_source(name: *u8) -> i64
called by 1: ci_walk_tree calls 2: ci_containsci_ends
181func ci_name_eq(name: *u8, lit: *u8) -> i64
called by 1: ci_tree_skipdir calls 2: ci_lenci_contains
185func ci_tree_skipdir(name: *u8) -> i64
called by 1: ci_walk_tree calls 1: ci_name_eq
198func ci_next_segid(prefix: *u8) -> i64
called by 1: ci_state_new calls 1: ss_next_segid
206func ci_state_new(prefix: *u8, counts: *i64) -> *i64
called by 2: ci_runci_ingest_source_tree calls 3: ss_beginci_next_segidss_open
224func ci_state_finish(st: *i64) -> i64
called by 2: ci_runci_ingest_source_tree calls 2: ss_commitci_puts
234func ci_put(st: *i64, kind: i64, key: *u8, val: *u8, vlen: i64) -> i64
called by 1: ci_ingest_doc calls 5: ss_addss_commitci_putsci_numss_begin
251func ci_def_anchors(src: *u8, n: i64, out: *u8, cap: i64) -> i64
called by 1: ci_ingest_file calls 3: fh_scanci_lenss_def_token
282func ci_ingest_file(st: *i64, path: *u8, kind: i64, urlprefix: *u8, urlname: *u8) -> i64
called by 2: ci_runci_walk_tree calls 7: ss_readallnx_html_to_textss_freeallss_ident_anchorsci_def_anchorsci_ingest_doc+1
341func ci_ingest_doc(st: *i64, doc: *u8, dn: i64, urlprefix: *u8, urlname: *u8) -> i64
called by 2: ci_ingest_fileci_ingest_board_rows calls 10: ci_hashdss_mkkeyss_hgetci_lenci_mkpthkeyci_atoi+4
400func ci_ingest_board_rows(st: *i64, txt: *u8, tn: i64, urlprefix: *u8, urlname: *u8) -> i64
called by 1: ci_ingest_file calls 6: ci_lenss_ident_anchorsss_def_tokenci_catci_decci_ingest_doc
458func ci_run(dir: *u8, domain: *u8, urlprefix: *u8, counts: *i64) -> i64
called by 2: mainmain calls 7: dss_prefixci_state_newci_putsci_kindci_catci_ingest_file+1
496func ci_walk_tree(st: *i64, dir: *u8, rel: *u8, depth: i64, urlprefix: *u8) -> i64
called by 2: ci_walk_treeci_ingest_source_tree calls 6: ci_putsci_catci_tree_skipdirci_walk_treeci_kind_sourceci_ingest_file
544func ci_ingest_source_tree(root: *u8, domain: *u8, urlprefix: *u8, counts: *i64) -> i64
called by 2: ci_ingest_boardsmain calls 5: dss_prefixci_state_newci_putsci_walk_treeci_state_finish
561func ci_ingest_boards(confpath: *u8, counts: *i64) -> i64
called by 1: main calls 5: ss_readallci_putsci_ingest_source_treeci_numss_freeall
620func main(argc: i64, argv: *i64) -> i64
calls 7: ci_putsci_lenci_containsci_ingest_source_treeci_numci_ingest_boards+1