code wiki / (root) / nx_specdec.nx

nx_specdec.nx

buildroot/runtime/nx_specdec.nx

9897 B238 linesdepth 14pulls 58 transitivereach 1 importersview sourcekind library
docsdependenciesstructsconstsfunctions

about

nx_specdec.nx -- sovereign SPECULATIVE DECODING (prompt-lookup drafted, greedy-verified; census OPPORTUNITY "Speculative decoding", momentum 3). Decode is MEMORY-bound (weights re-read per token). Speculation turns k sequential m=1 forwards into ONE m=k+1 forward: DRAFT k tokens cheaply, VERIFY them in a single multi-token pass (weight bytes read ONCE for the whole block -- the same amortization that makes prefill cheap). With GREEDY acceptance (argmax match) the output is BIT-IDENTICAL to plain greedy decode: pure measured speedup, zero quality change. Draft source here = PROMPT-LOOKUP (n-gram continuation from the model's own context; Saxena 2023 "prompt lookup decoding", the draft-model-free member of the speculative family: Leviathan 2023 / Chen 2023; EAGLE-class trained drafts can replace it later behind the same verify loop). Acceptance is WORKLOAD-DEPENDENT: echo/quote/list-heavy text accepts big chunks; free-form text accepts little (then cost ~= plain decode plus the occasional rejected block). Cache contract: the verify forward appends k+1 rows; on partial accept the cache is TRUNCATED back to the accepted prefix (seq_len rewind -- rows are plain arrays, later appends overwrite). Positions/causality inside the multi-token pass are the offset-causal q_pos = total_k - n_q + i already implemented by nx_f32_attn_multi (proven by nx_specdec_gate EQUIV rows). genealogy_id: leviathan_2023_speculative + saxena_2023_prompt_lookup lineage_id: substrate_specdec_v1

dependencies 8 imports · 1 importers

nx_syscalls.nx nx_tier.nx nx_bpe.nx nx_f32_kv_cache.nx nx_f32_llm.nx nx_f32_llm_v4.nx nx_f32_sampler.nx nx_reasoning.nx nx_specdec.nx nx_specdec_gate.nx

imports: nx_syscalls.nxnx_tier.nxnx_bpe.nxnx_f32_kv_cache.nxnx_f32_llm.nxnx_f32_llm_v4.nxnx_f32_sampler.nxnx_reasoning.nx

imported by: nx_specdec_gate.nx

structs

88struct NxSpecStats

consts

37const NX_SPEC_NGRAM_MAX: nx_int = 3 // longest suffix n-gram to match
38const NX_SPEC_NGRAM_MIN: nx_int = 2 // 1-grams match spuriously; stop at 2
39const NX_SPEC_K: nx_int = 6 // max drafted tokens per round
96const NX_SPEC_STATS_BYTES: nx_int = 40

functions

46func nx_spec_draft_lookup(ctx: *i64, n_ctx: nx_int, k: nx_int,
98func nx_spec_stats_alloc() -> *NxSpecStats
called by 1: main calls 1: sys_mmap
102func _spec_trunc(cache: *NxF32KVCache, new_len: nx_int) -> i64
107func _spec_stop_tok(rc: *NxReasonCfg, t: nx_int) -> nx_int
117func nx_spec_plain_greedy(rc: *NxReasonCfg, toks: *i64, nt: nx_int,
152func nx_spec_decode_greedy(rc: *NxReasonCfg, toks: *i64, nt: nx_int,