nx_kvcache.nx
buildroot/runtime/nx_kvcache.nx
about
nx_kvcache.nx -- sovereign PAGED KV-CACHE (census OPPORTUNITY axis
"KV-cache / paged attention", momentum 3; vLLM-class, Kwon 2023).
The contiguous NxF32KVCache allocates max_seq_len rows per sequence up
front: fragmentation, no sharing, one sequence per buffer. Paging fixes
all three: the cache is a POOL of fixed-size BLOCKS (BS tokens); a
sequence is a PAGE TABLE of block ids; blocks allocate on demand and
refcount for sharing.
KEY STRUCTURAL FACT exploited here: decode is APPEND-ONLY -- past rows
are never mutated -- so full blocks are IMMUTABLE and can be shared
forever; only the PARTIAL TAIL block ever needs copy-on-append. fork()
is therefore O(page-table) + refcounts: prefill a prompt ONCE, fork N
sequences that share the prompt blocks (the self-consistency / best-of-N
serving pattern), each fork copies at most ONE partial block on its
first append.
Pool layout (f32 bits in i64 slots, engine convention):
K_pool/V_pool: [n_blocks][n_layers][BS][kv_dim]
row t of seq -> block pt[t / BS], offset t % BS.
Fail-fast: pool exhaustion returns an error id (-1) -- NO silent
fallback to contiguous.
genealogy_id: kwon_2023_pagedattention
lineage_id: substrate_paged_kv_v1
dependencies 2 imports · 12 importers
diagram shows first 10 each side; +0 more imports, +2 more importers in the complete lists below.
imports: nx_syscalls.nxnx_tier.nx
imported by: nx_batched_gate.nxnx_batched_tput.nxnx_f32_attn_paged.nxnx_f32_llama_v4b.nxnx_f32_llama_v4p.nxnx_llm_batch_serve.nxnx_llm_sched.nxnx_llm_sched_gate.nxnx_paged_fwd_gate.nxnx_paged_kv_gate.nxnx_reason_paged_probe.nxnx_reasoning_paged.nx
structs
| 40 | struct NxPagedPool |
| 107 | struct NxPagedSeq |
consts
| 30 | const NX_PKV_BS: nx_int = 16 // tokens per block |
| 32 | const NX_PKV_OK: nx_int = 0 |
| 33 | const NX_PKV_ERR_NULL: nx_int = 1 |
| 34 | const NX_PKV_ERR_BAD_DIM: nx_int = 2 |
| 35 | const NX_PKV_ERR_EXHAUST: nx_int = 3 |
| 36 | const NX_PKV_ERR_BOUNDS: nx_int = 4 |
| 51 | const NX_PKV_POOL_BYTES: nx_int = 64 |
| 115 | const NX_PKV_SEQ_BYTES: nx_int = 48 |
functions
| 53 | func nx_pkv_pool_new(n_blocks: nx_int, n_layers: nx_int, |
| 77 | func nx_pkv_alloc_block(p: *NxPagedPool) -> i64 |
| 85 | func nx_pkv_ref_block(p: *NxPagedPool, b: i64) -> i64 called by 1: nx_pkv_seq_fork |
| 90 | func nx_pkv_unref_block(p: *NxPagedPool, b: i64) -> i64 |
| 101 | func _pkv_base(p: *NxPagedPool, b: i64, L: nx_int, r: nx_int) -> i64 |
| 117 | func nx_pkv_seq_new(p: *NxPagedPool, max_tokens: nx_int) -> *NxPagedSeq |
| 130 | func nx_pkv_seq_fork(src: *NxPagedSeq) -> *NxPagedSeq |
| 148 | func nx_pkv_seq_free(s: *NxPagedSeq) -> i64 called by 7: mainbt_measurenx_sched_releasesg2_solomainnx_reason_selfconsist_paged+1 calls 1: nx_pkv_unref_block |
| 161 | func nx_pkv_seq_truncate(s: *NxPagedSeq, new_len: nx_int) -> nx_int |
| 178 | func nx_pkv_ensure_append(s: *NxPagedSeq, n_new: nx_int) -> nx_int |
| 227 | func nx_pkv_append_layer(s: *NxPagedSeq, L: nx_int, |
| 248 | func nx_pkv_advance(s: *NxPagedSeq, n_new: nx_int) -> nx_int |
| 255 | func nx_pkv_gather_head(s: *NxPagedSeq, L: nx_int, kv_head: nx_int, |