code wiki / _hdl_build / nx_nofloat_linattn.nx
nx_nofloat_linattn.nx
buildroot/runtime/_hdl_build/nx_nofloat_linattn.nx
about
nx_nofloat_linattn.nx -- SOVEREIGN no-float LINEAR ATTENTION recurrent kernel = the foundation of Kimi
Delta Attention (KDA, the K3 namesake; operator 2026-07-19 "state of the art"). Linear attention replaces
softmax(QK^T)V with a RECURRENT matrix state: S_t = S_{t-1} + k_t (x) v_t (outer product, d x d);
o_t = q_t S_t. This is O(1) memory per step (no growing KV cache) -> the 75% KV cut + 6x decode @1M that
KDA delivers. Done in PURE INTEGER Q16 (outer product + matvec, accumulate-then-shift) = BIT-EXACT
deterministic linear attention -- a float linear-attn stack drifts by accumulation order; ours does not.
KDA extends THIS with (a) the delta rule S_t = S_{t-1}(I - b k k^T) + b k v^T and (b) channel-wise gating
alpha_t (diagonal forget) -- both follow-ons on this proven kernel. Grounded arXiv 2510.26692 (Kimi Linear).
license_tier: ORIGINAL No hw writes (Rule 26).
dependencies 1 imports · 4 importers
imports: nx_syscalls.nx
imported by: nx_nofloat_k3block.nxnx_nofloat_kda.nxnx_nofloat_kda_gate.nxnx_nofloat_linattn_gate.nx
structs
| none |
consts
| 12 | const LA_QBITS: i64 = 16 |
| 13 | const LA_DMAX: i64 = 16 // max head dim |
functions
| 16 | func la_zero(s: *i64, d: i64) -> i64 { var i: i64 = 0; while i < d*d { s[i] = 0; i = i + 1 } return 0 } |
| 18 | func la_accum(s: *i64, k: *i64, v: *i64, d: i64) -> i64 |
| 29 | func la_readout(s: *i64, q: *i64, o: *i64, d: i64) -> i64 |
| 42 | func la_forward(qm: *i64, km: *i64, vm: *i64, om: *i64, t: i64, d: i64) -> i64 |