nx_specdec.nx
buildroot/runtime/nx_specdec.nx
about
nx_specdec.nx -- sovereign SPECULATIVE DECODING (prompt-lookup drafted,
greedy-verified; census OPPORTUNITY "Speculative decoding", momentum 3).
Decode is MEMORY-bound (weights re-read per token). Speculation turns k
sequential m=1 forwards into ONE m=k+1 forward: DRAFT k tokens cheaply,
VERIFY them in a single multi-token pass (weight bytes read ONCE for the
whole block -- the same amortization that makes prefill cheap). With
GREEDY acceptance (argmax match) the output is BIT-IDENTICAL to plain
greedy decode: pure measured speedup, zero quality change.
Draft source here = PROMPT-LOOKUP (n-gram continuation from the model's
own context; Saxena 2023 "prompt lookup decoding", the draft-model-free
member of the speculative family: Leviathan 2023 / Chen 2023; EAGLE-class
trained drafts can replace it later behind the same verify loop).
Acceptance is WORKLOAD-DEPENDENT: echo/quote/list-heavy text accepts big
chunks; free-form text accepts little (then cost ~= plain decode plus the
occasional rejected block).
Cache contract: the verify forward appends k+1 rows; on partial accept the
cache is TRUNCATED back to the accepted prefix (seq_len rewind -- rows are
plain arrays, later appends overwrite). Positions/causality inside the
multi-token pass are the offset-causal q_pos = total_k - n_q + i already
implemented by nx_f32_attn_multi (proven by nx_specdec_gate EQUIV rows).
genealogy_id: leviathan_2023_speculative + saxena_2023_prompt_lookup
lineage_id: substrate_specdec_v1
dependencies 8 imports · 1 importers
imports: nx_syscalls.nxnx_tier.nxnx_bpe.nxnx_f32_kv_cache.nxnx_f32_llm.nxnx_f32_llm_v4.nxnx_f32_sampler.nxnx_reasoning.nx
imported by: nx_specdec_gate.nx
structs
| 88 | struct NxSpecStats |
consts
| 37 | const NX_SPEC_NGRAM_MAX: nx_int = 3 // longest suffix n-gram to match |
| 38 | const NX_SPEC_NGRAM_MIN: nx_int = 2 // 1-grams match spuriously; stop at 2 |
| 39 | const NX_SPEC_K: nx_int = 6 // max drafted tokens per round |
| 96 | const NX_SPEC_STATS_BYTES: nx_int = 40 |
functions
| 46 | func nx_spec_draft_lookup(ctx: *i64, n_ctx: nx_int, k: nx_int, |
| 98 | func nx_spec_stats_alloc() -> *NxSpecStats |
| 102 | func _spec_trunc(cache: *NxF32KVCache, new_len: nx_int) -> i64 called by 1: nx_spec_decode_greedy |
| 107 | func _spec_stop_tok(rc: *NxReasonCfg, t: nx_int) -> nx_int |
| 117 | func nx_spec_plain_greedy(rc: *NxReasonCfg, toks: *i64, nt: nx_int, called by 1: main calls 5: nx_f32_kv_cache_resetsys_mmapnx_f32_llm_forward_v4nx_f32_sampler_argmax_spec_stop_tok |
| 152 | func nx_spec_decode_greedy(rc: *NxReasonCfg, toks: *i64, nt: nx_int, |