code wiki / _hdl_build / _galx_bulk_ingest.nx

_galx_bulk_ingest.nx

buildroot/runtime/_hdl_build/_galx_bulk_ingest.nx

7340 B133 linesdepth 7pulls 10 transitivereach 0 importersview sourcekind tooltopic galx
docsdependenciesstructsconstsfunctions

about

_galx_bulk_ingest.nx -- HIGH-THROUGHPUT bulk corpus ingest for the FULL ~230k NAS image library. The per-image path (_galx_prod_ingest -> nx_store_ingest_ingest_cid) commits ONE segment per image AND does an O(store) cap-aware dedup scan per image -- fine for thousands, fatal for 230k (the scan re-reads every seg file each time over slow drvfs). This driver instead: (1) COMPUTES the CID via the SHARED nx_store_ingest_compute_cid (byte-identical to the per-image path -- same GENREC -> same CID), so the bulk store is consistent with the existing one; (2) BATCHES commits -- BATCH images accumulate in ONE ss_begin writer, committed as ONE segment (230k images -> ~460 segments, not 230k), so browse + future dedup stay bounded; (3) SKIPS the per-image dedup -- this is a fresh distinct set; any accidental duplicate (e.g. the 4436 already ingested) is COLLAPSED losslessly by the final ss_compact_cap (latest-per-key). segid = SEGBASE + start/BATCH + (batch# in this slice) -- globally unique iff start & count are multiples of BATCH (the runner nx_galx_fill enforces this). One process per slice (the runner forks it) bounds RSS = slice*~filesize, reclaimed on exit (no munmap). Appends <CID>\t<path> to the sidecar. Usage: _galx_bulk_ingest <start> <count> [corpus_path]. license_tier: ORIGINAL

dependencies 4 imports · 0 importers

nx_store_ingest.nx nx_seg_store.nx nx_canon_cid.nx nx_syscalls.nx _galx_bulk_ingest.nx

imports: nx_store_ingest.nxnx_seg_store.nxnx_canon_cid.nxnx_syscalls.nx

imported by: nobody (leaf or entry point)

call flow from main pre-order; caps 40 nodes / depth 6 declared; ↻ = already shown

main bp sys_write sys_exit b_atoi sys_mmap sys_read_file sys_openat_rd sys_lseek sys_mmap ↻ sys_read sys_close sys_openat_append ss_begin ss_begin_cap sys_mmap ↻ nx_store_ingest_compute_ci sys_mmap ↻ nx_store_ingest_harvest sys_mmap ↻ _pe_pngchunk_rdbe _pe_pngtext_locate nx_store_ingest_keq canon_encode sys_mmap ↻ cc_cmp cc_w32 cc_len sys_munmap cid_of sys_mmap ↻ sha256_digest sys_mmap ↻ sha256_init sys_mmap ↻ sha256_k sha256_update sha256_compress_ni_blocks blk_set_byte sha256_compress

structs

none

consts

20const PREFIX: *u8 = "knowledge/store/galx-prod-"
21const SEGBASE: i64 = 5000000
22const BATCH: i64 = 500

functions

24func bp(s: *u8) -> i64 { var n: i64=0; while s[n]!=(0 as u8){n=n+1} sys_write(1,s,n); return 0 }
called by 1: main calls 1: sys_write
25func bn(v: i64) -> i64 { let b: *u8=sys_mmap(28); var m: i64=v; if m<0{m=0-m}; let t: *u8=sys_mmap(28); var k: i64=0; if m==0{t[0]=48;k=1}; while m>0{t[k]=(48+(m%10)) as u8;m=m/10;k=k+1}; var i: i64=0; while i<k{b[i]=t[k-1-i];i=i+1}; sys_write(1,b,k); return 0 }
called by 1: main calls 2: sys_mmapsys_write
26func b_atoi(s: *u8) -> i64 { var v: i64=0; var i: i64=0; while s[i]!=(0 as u8){ if s[i]>=(48 as u8){ if s[i]<=(57 as u8){ v=v*10+((s[i]-(48 as u8)) as i64) } } i=i+1 } return v }
called by 1: main
28func main(argc: i64, argv: *i64) -> i64
calls 14: bpsys_exitb_atoisys_mmapsys_read_filesys_openat_append+8