nx_web_shard_compact.nx

nx_web_shard_compact.nx -- SCALE compaction for big multi-segment shards (the web shard's rung). ss_compact/ss_compact_cap have EXACT semantics (last entry per key wins, tombstones kept, retired segments archived then an atomic manifest swap) but their key dedup is a LINEAR table scan = O(entries^2): fine at 4k keys, unusable at the web shard's ~1.4M entries (bulk docs + pr: priors). This organ is the same merge with an open-addressing HASH dedup = O(entries), plus a post-swap verification pass. Semantics kept byte-equal to ss_compact: chronological walk, last wins, ALL kinds carried (incl. kind-2 tombstones), archive-before-swap, every failure path returns BEFORE the swap. BYTE-BUDGETED SUFFIX FOLD (2026-07-29): an unbounded whole-shard merge materializes the merged segment in anonymous RAM (the writer buffer, plus aux blobs and key tables that all scale with merged bytes) -- the OOM killer reaped it twice at bytes-in=1883885124 on 2026-07-25, mid-merge, before the swap (never-brick held; shard untouched). The merge therefore folds a contiguous run whose .docs bytes fit a budget derived from the box itself (MemAvailable/WSC_BUDGET_SHARE, clamped to WSC_BUDGET_CEIL; argv[2] in MB overrides, same clamp). Repeated runs converge the shard LSM-style with bounded memory; a streaming (file-backed) writer that lifts the budget entirely is the named next rung. RANGE FOLD (2026-08-10, supersedes suffix-only selection). The suffix rule stalls forever at the first FAT segment: once a merged product nears the budget it heads every suffix walk (runbytes starts at the tail), so each later fold must carry it and the hundreds of small segments BEHIND it can never be reached -- measured on this shard: ~836 pre-08-08 segments stranded behind a ~240MB merged tail while the live count climbed 878 -> 1205 and query latency followed. A fold of ANY contiguous run, replaced IN PLACE in the manifest, preserves last-wins semantics by the same argument the suffix fold's header makes: shadowing depends only on RELATIVE manifest order, and the merged segment sits exactly where the run sat -- newer kept segments still shadow it, and it still shadows older ones, precisely as the unmerged run did. Kind-2 tombstones are carried as before. FAT SKIP: a segment >= budget/WSC_FAT_DIV is an already-merged product; re-folding it buys almost no count reduction per byte, so runs are chosen from non-fat segments only, newest first. usage: nx_web_shard_compact <domain> [budget-MB] license_tier: ORIGINAL