code wiki / (root) / nx_kvcache.nx

nx_kvcache.nx

buildroot/runtime/nx_kvcache.nx

9562 B273 linesdepth 2pulls 2 transitivereach 12 importersview sourcekind library
docsdependenciesstructsconstsfunctions

about

nx_kvcache.nx -- sovereign PAGED KV-CACHE (census OPPORTUNITY axis "KV-cache / paged attention", momentum 3; vLLM-class, Kwon 2023). The contiguous NxF32KVCache allocates max_seq_len rows per sequence up front: fragmentation, no sharing, one sequence per buffer. Paging fixes all three: the cache is a POOL of fixed-size BLOCKS (BS tokens); a sequence is a PAGE TABLE of block ids; blocks allocate on demand and refcount for sharing. KEY STRUCTURAL FACT exploited here: decode is APPEND-ONLY -- past rows are never mutated -- so full blocks are IMMUTABLE and can be shared forever; only the PARTIAL TAIL block ever needs copy-on-append. fork() is therefore O(page-table) + refcounts: prefill a prompt ONCE, fork N sequences that share the prompt blocks (the self-consistency / best-of-N serving pattern), each fork copies at most ONE partial block on its first append. Pool layout (f32 bits in i64 slots, engine convention): K_pool/V_pool: [n_blocks][n_layers][BS][kv_dim] row t of seq -> block pt[t / BS], offset t % BS. Fail-fast: pool exhaustion returns an error id (-1) -- NO silent fallback to contiguous. genealogy_id: kwon_2023_pagedattention lineage_id: substrate_paged_kv_v1

dependencies 2 imports · 12 importers

nx_syscalls.nx nx_tier.nx nx_kvcache.nx nx_batched_gate.nx nx_batched_tput.nx nx_f32_attn_paged.nx nx_f32_llama_v4b.nx nx_f32_llama_v4p.nx nx_llm_batch_serve.nx nx_llm_sched.nx nx_llm_sched_gate.nx nx_paged_fwd_gate.nx nx_paged_kv_gate.nx

diagram shows first 10 each side; +0 more imports, +2 more importers in the complete lists below.

imports: nx_syscalls.nxnx_tier.nx

imported by: nx_batched_gate.nxnx_batched_tput.nxnx_f32_attn_paged.nxnx_f32_llama_v4b.nxnx_f32_llama_v4p.nxnx_llm_batch_serve.nxnx_llm_sched.nxnx_llm_sched_gate.nxnx_paged_fwd_gate.nxnx_paged_kv_gate.nxnx_reason_paged_probe.nxnx_reasoning_paged.nx

structs

40struct NxPagedPool
107struct NxPagedSeq

consts

30const NX_PKV_BS: nx_int = 16 // tokens per block
32const NX_PKV_OK: nx_int = 0
33const NX_PKV_ERR_NULL: nx_int = 1
34const NX_PKV_ERR_BAD_DIM: nx_int = 2
35const NX_PKV_ERR_EXHAUST: nx_int = 3
36const NX_PKV_ERR_BOUNDS: nx_int = 4
51const NX_PKV_POOL_BYTES: nx_int = 64
115const NX_PKV_SEQ_BYTES: nx_int = 48

functions

53func nx_pkv_pool_new(n_blocks: nx_int, n_layers: nx_int,
called by 7: mainmainmainmainmainmain+1 calls 1: sys_mmap
77func nx_pkv_alloc_block(p: *NxPagedPool) -> i64
85func nx_pkv_ref_block(p: *NxPagedPool, b: i64) -> i64
called by 1: nx_pkv_seq_fork
90func nx_pkv_unref_block(p: *NxPagedPool, b: i64) -> i64
101func _pkv_base(p: *NxPagedPool, b: i64, L: nx_int, r: nx_int) -> i64
117func nx_pkv_seq_new(p: *NxPagedPool, max_tokens: nx_int) -> *NxPagedSeq
130func nx_pkv_seq_fork(src: *NxPagedSeq) -> *NxPagedSeq
148func nx_pkv_seq_free(s: *NxPagedSeq) -> i64
161func nx_pkv_seq_truncate(s: *NxPagedSeq, new_len: nx_int) -> nx_int
called by 1: main calls 1: nx_pkv_unref_block
178func nx_pkv_ensure_append(s: *NxPagedSeq, n_new: nx_int) -> nx_int
227func nx_pkv_append_layer(s: *NxPagedSeq, L: nx_int,
called by 1: nx_f32_attn_with_paged calls 1: _pkv_base
248func nx_pkv_advance(s: *NxPagedSeq, n_new: nx_int) -> nx_int
255func nx_pkv_gather_head(s: *NxPagedSeq, L: nx_int, kv_head: nx_int,
called by 1: nx_f32_attn_with_paged calls 1: _pkv_base