nx_speculative_verify.nx
buildroot/runtime/nx_speculative_verify.nx
about
nx_speculative_verify.nx -- H3 speculative decoding bits-up.
Per NISHI_ELDER_AI_OFF_DOCKER_2026_05_20.md ยง2.1 H3: small fast
draft model proposes K tokens; large target model verifies them
in one parallel forward pass; substrate keeps the longest
accept-prefix + 1 bonus token (the target's own next-token
prediction at the rejection boundary is free). Leviathan 2023
+ Chen 2023 "Accelerating Large Language Model Decoding with
Speculative Sampling." Practical wins: 2-3x throughput at
equivalent quality when draft + target agree on common tokens.
V1 mechanics: the substrate primitive does NOT run a model.
It accepts the draft tokens (what the small model proposed)
and a parallel accept_flag array (1 == target verified, 0 ==
target rejected) computed upstream by the verification kernel,
PLUS the target's own next-token at the rejection boundary
(the "free" bonus token). It returns the longest accept-prefix
length and the count of tokens actually committed to the
caller's output buffer.
Composition path:
- nx_speculative_session_new(K, request_id) on admit
- For each verification round: nx_speculative_verify(session,
draft_tokens, accept_flags, K_actual, target_next_token,
out_committed) returns count_committed
- Per-session stats (drafted / accepted / rounds) accumulate
so the consumer can monitor the acceptance rate and back
off speculative depth when the draft model is mispredicting
- Acceptance rate exposed as Q16 fixed-point (rate * 65536)
to keep the substrate primitive integer-only
Pure substrate logic. No Linux features. Composes with shipped
nx_batch_scheduler (each batch slot can carry a NxSpeculativeSession
pointer for its decoding state).
V1 honest scope:
- Fixed max-K (NX_SPEC_MAX_K = 16) per round
- Caller provides verification flags (substrate doesn't run the
target forward pass; that's a tensor kernel one layer up)
- Greedy verification only (no sampled-rejection-resampling;
dependencies 1 imports · 2 importers
imports: nx_syscalls.nx
imported by: nx_hackers_algo_compose_test.nxnx_speculative_verify_test.nx
structs
| 97 | struct NxSpeculativeSession |
consts
| 74 | const NX_MAGIC_65536: i64 = 65536 |
| 77 | const NX_SPEC_MAX_K: i64 = 16 |
| 80 | const NX_SPEC_OK: i64 = 0 |
| 81 | const NX_SPEC_BAD_INPUT: i64 = 1 |
| 82 | const NX_SPEC_BAD_K: i64 = 2 |
| 83 | const NX_SPEC_TAMPER: i64 = 3 |
| 84 | const NX_SPEC_N_VERDICTS: i64 = 4 |
| 93 | const NX_SPEC_SESSION_CANARY_PRE: i64 = 0x537065635365737A // "SpecSesz" |
| 94 | const NX_SPEC_SESSION_CANARY_POST: i64 = 0x53657373456E6464 // "SessEndd" |
functions
| 86 | func nx_spec_verdict_is_valid(v: i64) -> i64 called by 1: main |
| 108 | func nx_spec_session_is_valid(s: *NxSpeculativeSession) -> i64 |
| 124 | func nx_speculative_session_new(request_id: i64, max_k: i64) -> *NxSpeculativeSession |
| 154 | func nx_speculative_verify(s: *NxSpeculativeSession, draft_tokens: *i64, accept_flags: *i64, k_actual: i64, target_next: i64, out_committed: *i64) -> i64 |
| 194 | func nx_spec_total_drafted(s: *NxSpeculativeSession) -> i64 |
| 199 | func nx_spec_total_accepted(s: *NxSpeculativeSession) -> i64 |
| 204 | func nx_spec_total_committed(s: *NxSpeculativeSession) -> i64 |
| 209 | func nx_spec_total_rounds(s: *NxSpeculativeSession) -> i64 |
| 216 | func nx_spec_acceptance_rate_q16(s: *NxSpeculativeSession) -> i64 |
| 223 | func nx_spec_session_request_id(s: *NxSpeculativeSession) -> i64 |
| 228 | func nx_spec_session_max_k(s: *NxSpeculativeSession) -> i64 |