code wiki / _hdl_build / nx_store_compact.nx
nx_store_compact.nx
buildroot/runtime/_hdl_build/nx_store_compact.nx
about
nx_store_compact.nx -- collapse a quadratic seg-store plane back to O(rows). RK013 sev9 / F829.
THE DEFECT (measured 2026-07-20, root-caused from source, not theorised):
sts_seed commits the WHOLE plane as a NEW segment on EVERY write (nx_store_seed_lib:45/65),
sts_load does one ss_get PER ROW (:70), and ss_scan_seglist walks ALL segments chronologically
for each key (nx_seg_store:1136). So a load costs O(rows x segments). Live numbers: debt- had
464 segments / 42MB of .docs for ~200KB of actual rows (~210x amplification), ~124k segment-key
scans per load, one nx_debt add took >15min then FAILED, nx_debt page was OOM-KILLED, and the
store reached 15,024 files. The NAS went down under that load the same afternoon.
THE INSIGHT THAT MAKES THE FIX SAFE AND CHEAP: because every segment is ALREADY a complete plane
snapshot, and ss_scan_seglist resolves each key to its LAST (newest) version, the NEWEST SEGMENT
ALONE IS A SEMANTICALLY COMPLETE PLANE -- including tombstones. So compaction does not merge, copy
or rewrite any record: it rewrites ONLY THE MANIFEST to reference the newest segment.
- The superseded .docs/.idx files are LEFT ON DISK, untouched (rule 13 additive-only). Nothing is
deleted; the manifest is the only mutation, and it is backed up first.
- Therefore ROLLBACK IS ONE FILE COPY: restore <prefix>manifest.txt.bak-precompact.
PROOF, NOT ASSERTION: apply loads the plane BEFORE, rewrites the manifest, loads AFTER, and compares
the two buffers BYTE-FOR-BYTE. Any difference -> the manifest is RESTORED and the organ REFUSES
(exit 3). A caller can never be left with a silently-lossy plane.
nx_store_compact <prefix> check -> segment count + newest segid (CHEAP: manifest only, no load)
nx_store_compact <prefix> apply -> the verified rewrite above (sts planes ONLY: every segment
is a whole-plane snapshot there)
nx_store_compact <prefix> fold -> GENERIC merge compaction via the store's own ss_compact_cap
(latest entry per key, tombstones kept, O(data) single pass).
REQUIRED for reg planes (nx_registry: one record + index per
segment -- newest segment is NOT a complete plane, so apply's
manifest re-point would lose every other key). Verified by a
pre-fold logical snapshot re-read AFTER through production
ss_get; mismatch -> manifest restored, REFUSE (exit 3).
FAIL-CLOSED: missing/empty manifest, a load that fills the buffer (cannot verify a truncated view),
or a byte-mismatch all refuse without leaving the plane changed.
license_tier: ORIGINAL No hw writes (Rule 26). expect_exit: 0
dependencies 3 imports · 0 importers
imports: nx_store_seed_lib.nxnx_seg_store.nxnx_syscalls.nx
imported by: nobody (leaf or entry point)
call flow from main pre-order; caps 40 nodes / depth 6 declared; ↻ = already shown
structs
| none |
consts
| 39 | const SC_LOADCAP: i64 = 8388608 |
| 40 | const SC_KEYCAP: i64 = 1024 |
| 41 | const SC_MFCAP: i64 = 262144 |
| 42 | const SC_PATHCAP: i64 = 1024 |
| 43 | const SC_MSGCAP: i64 = 1024 |
| 44 | const SC_NL: i64 = 10 |
| 45 | const SC_DASH: i64 = 45 |
| 46 | const SC_ZERO: i64 = 48 |
| 47 | const SC_NINE: i64 = 57 |
| 48 | const SC_MODE: i64 = 420 |
| 49 | const SC_LOCK_EX: i64 = 2 |
| 50 | const SC_STDERR: i64 = 2 |
| 51 | const SC_EXIT_USAGE: i64 = 2 |
| 52 | const SC_EXIT_REFUSED: i64 = 3 |
| 53 | const SC_EXIT_IO: i64 = 1 |
functions
| 55 | func sc_slen(s: *u8) -> i64 { var n: i64 = 0; while s[n] != (0 as u8) { n = n + 1 } return n } |
| 56 | func sc_puts(s: *u8) -> i64 { sys_write(1, s, sc_slen(s)); return 0 } |
| 57 | func sc_werr(s: *u8) -> i64 { sys_write(SC_STDERR, s, sc_slen(s)); return 0 } |
| 58 | func sc_putn(v: i64) -> i64 { let b: *u8 = sys_mmap(32); let e: i64 = ss_catn(b, 0, v); sys_write(1, b, e); return 0 } |
| 59 | func sc_eqs(a: *u8, b: *u8) -> i64 called by 1: main |
| 65 | func sc_path(prefix: *u8, tail: *u8, out: *u8) -> i64 |
| 71 | func sc_read(path: *u8, b: *u8, cap: i64) -> i64 |
| 84 | func sc_write(path: *u8, b: *u8, n: i64) -> i64 |
| 93 | func sc_scan_manifest(b: *u8, n: i64, segmax: *i64) -> i64 called by 1: main |
| 126 | func sc_snapshot(prefix: *u8, mfb: *u8, mfn: i64, tkp: *i64, tkl: *i64, tkind: *i64, tvp: *i64, tvl: *i64, maxk: i64) -> i64 |
| 180 | func main(argc: i64, argv: *i64) -> i64 |