code wiki / _hdl_build / nx_nofloat_linattn.nx

nx_nofloat_linattn.nx

buildroot/runtime/_hdl_build/nx_nofloat_linattn.nx

2387 B52 linesdepth 2pulls 2 transitivereach 9 importersview sourcekind librarytopic nofloat
docsdependenciesstructsconstsfunctions

about

nx_nofloat_linattn.nx -- SOVEREIGN no-float LINEAR ATTENTION recurrent kernel = the foundation of Kimi Delta Attention (KDA, the K3 namesake; operator 2026-07-19 "state of the art"). Linear attention replaces softmax(QK^T)V with a RECURRENT matrix state: S_t = S_{t-1} + k_t (x) v_t (outer product, d x d); o_t = q_t S_t. This is O(1) memory per step (no growing KV cache) -> the 75% KV cut + 6x decode @1M that KDA delivers. Done in PURE INTEGER Q16 (outer product + matvec, accumulate-then-shift) = BIT-EXACT deterministic linear attention -- a float linear-attn stack drifts by accumulation order; ours does not. KDA extends THIS with (a) the delta rule S_t = S_{t-1}(I - b k k^T) + b k v^T and (b) channel-wise gating alpha_t (diagonal forget) -- both follow-ons on this proven kernel. Grounded arXiv 2510.26692 (Kimi Linear). license_tier: ORIGINAL No hw writes (Rule 26).

dependencies 1 imports · 4 importers

nx_syscalls.nx nx_nofloat_linattn.nx nx_nofloat_k3block.nx nx_nofloat_kda.nx nx_nofloat_kda_gate.nx nx_nofloat_linattn_gate.nx

imports: nx_syscalls.nx

imported by: nx_nofloat_k3block.nxnx_nofloat_kda.nxnx_nofloat_kda_gate.nxnx_nofloat_linattn_gate.nx

structs

none

consts

12const LA_QBITS: i64 = 16
13const LA_DMAX: i64 = 16 // max head dim

functions

16func la_zero(s: *i64, d: i64) -> i64 { var i: i64 = 0; while i < d*d { s[i] = 0; i = i + 1 } return 0 }
18func la_accum(s: *i64, k: *i64, v: *i64, d: i64) -> i64
29func la_readout(s: *i64, q: *i64, o: *i64, d: i64) -> i64
42func la_forward(qm: *i64, km: *i64, vm: *i64, om: *i64, t: i64, d: i64) -> i64