code wiki / _hdl_build / nx_store_compact.nx

nx_store_compact.nx

buildroot/runtime/_hdl_build/nx_store_compact.nx

18445 B380 linesdepth 5pulls 6 transitivereach 0 importersview sourcekind tooltopic store
docsdependenciesstructsconstsfunctions

about

nx_store_compact.nx -- collapse a quadratic seg-store plane back to O(rows). RK013 sev9 / F829. THE DEFECT (measured 2026-07-20, root-caused from source, not theorised): sts_seed commits the WHOLE plane as a NEW segment on EVERY write (nx_store_seed_lib:45/65), sts_load does one ss_get PER ROW (:70), and ss_scan_seglist walks ALL segments chronologically for each key (nx_seg_store:1136). So a load costs O(rows x segments). Live numbers: debt- had 464 segments / 42MB of .docs for ~200KB of actual rows (~210x amplification), ~124k segment-key scans per load, one nx_debt add took >15min then FAILED, nx_debt page was OOM-KILLED, and the store reached 15,024 files. The NAS went down under that load the same afternoon. THE INSIGHT THAT MAKES THE FIX SAFE AND CHEAP: because every segment is ALREADY a complete plane snapshot, and ss_scan_seglist resolves each key to its LAST (newest) version, the NEWEST SEGMENT ALONE IS A SEMANTICALLY COMPLETE PLANE -- including tombstones. So compaction does not merge, copy or rewrite any record: it rewrites ONLY THE MANIFEST to reference the newest segment. - The superseded .docs/.idx files are LEFT ON DISK, untouched (rule 13 additive-only). Nothing is deleted; the manifest is the only mutation, and it is backed up first. - Therefore ROLLBACK IS ONE FILE COPY: restore <prefix>manifest.txt.bak-precompact. PROOF, NOT ASSERTION: apply loads the plane BEFORE, rewrites the manifest, loads AFTER, and compares the two buffers BYTE-FOR-BYTE. Any difference -> the manifest is RESTORED and the organ REFUSES (exit 3). A caller can never be left with a silently-lossy plane. nx_store_compact <prefix> check -> segment count + newest segid (CHEAP: manifest only, no load) nx_store_compact <prefix> apply -> the verified rewrite above (sts planes ONLY: every segment is a whole-plane snapshot there) nx_store_compact <prefix> fold -> GENERIC merge compaction via the store's own ss_compact_cap (latest entry per key, tombstones kept, O(data) single pass). REQUIRED for reg planes (nx_registry: one record + index per segment -- newest segment is NOT a complete plane, so apply's manifest re-point would lose every other key). Verified by a pre-fold logical snapshot re-read AFTER through production ss_get; mismatch -> manifest restored, REFUSE (exit 3). FAIL-CLOSED: missing/empty manifest, a load that fills the buffer (cannot verify a truncated view), or a byte-mismatch all refuse without leaving the plane changed. license_tier: ORIGINAL No hw writes (Rule 26). expect_exit: 0

dependencies 3 imports · 0 importers

nx_store_seed_lib.nx nx_seg_store.nx nx_syscalls.nx nx_store_compact.nx

imports: nx_store_seed_lib.nxnx_seg_store.nxnx_syscalls.nx

imported by: nobody (leaf or entry point)

call flow from main pre-order; caps 40 nodes / depth 6 declared; ↻ = already shown

main sc_werr sys_write sc_slen sys_exit sys_mmap sc_path ss_cat sc_read sys_openat_rd sys_read sys_close sc_scan_manifest sc_eqs sc_puts sys_write ↻ sc_slen ↻ sc_putn sys_mmap ↻ ss_catn sys_mmap ↻ sys_write ↻ sys_openat_append sys_flock sc_snapshot sys_mmap ↻ ss_cat ↻ ss_readall sys_openat_rd ↻ sys_lseek sys_mmap ↻ sys_read ↻ sys_close ↻ ss_r32 ss_kcmp sc_write sys_openat_wr sys_write ↻ sys_fsync sys_close ↻

structs

none

consts

39const SC_LOADCAP: i64 = 8388608
40const SC_KEYCAP: i64 = 1024
41const SC_MFCAP: i64 = 262144
42const SC_PATHCAP: i64 = 1024
43const SC_MSGCAP: i64 = 1024
44const SC_NL: i64 = 10
45const SC_DASH: i64 = 45
46const SC_ZERO: i64 = 48
47const SC_NINE: i64 = 57
48const SC_MODE: i64 = 420
49const SC_LOCK_EX: i64 = 2
50const SC_STDERR: i64 = 2
51const SC_EXIT_USAGE: i64 = 2
52const SC_EXIT_REFUSED: i64 = 3
53const SC_EXIT_IO: i64 = 1

functions

55func sc_slen(s: *u8) -> i64 { var n: i64 = 0; while s[n] != (0 as u8) { n = n + 1 } return n }
called by 2: sc_putssc_werr
56func sc_puts(s: *u8) -> i64 { sys_write(1, s, sc_slen(s)); return 0 }
called by 1: main calls 2: sys_writesc_slen
57func sc_werr(s: *u8) -> i64 { sys_write(SC_STDERR, s, sc_slen(s)); return 0 }
called by 1: main calls 2: sys_writesc_slen
58func sc_putn(v: i64) -> i64 { let b: *u8 = sys_mmap(32); let e: i64 = ss_catn(b, 0, v); sys_write(1, b, e); return 0 }
called by 1: main calls 3: sys_mmapss_catnsys_write
59func sc_eqs(a: *u8, b: *u8) -> i64
called by 1: main
65func sc_path(prefix: *u8, tail: *u8, out: *u8) -> i64
called by 1: main calls 1: ss_cat
71func sc_read(path: *u8, b: *u8, cap: i64) -> i64
called by 1: main calls 3: sys_openat_rdsys_readsys_close
84func sc_write(path: *u8, b: *u8, n: i64) -> i64
93func sc_scan_manifest(b: *u8, n: i64, segmax: *i64) -> i64
called by 1: main
126func sc_snapshot(prefix: *u8, mfb: *u8, mfn: i64, tkp: *i64, tkl: *i64, tkind: *i64, tvp: *i64, tvl: *i64, maxk: i64) -> i64
180func main(argc: i64, argv: *i64) -> i64