nx_nofloat_moe.nx
buildroot/runtime/nx_nofloat_moe.nx
about
nx_nofloat_moe.nx -- no-float Mixture-of-Experts FFN (2026-07-15, MoE rung 1: the layer MECHANISM).
The July-2026 open-weight leaders (DeepSeek V4 / Qwen3.5 / Llama 4-5 / Kimi K2) are ALL sparse-MoE --
the arch census's #1 gap. The f32 TRAINER half exists (nx_f32_moe_train_gate 6/6, 07-10); this is the
INFERENCE half in pure Q16 integer, composed from the proven no-float primitives:
router logits r = x @ Wr^T ([out,in] GGUF convention, mm_out_in)
-> deterministic top-K (strict >, LOWEST index on ties -- positional, never order-dependent)
-> fx_exp softmax over the K selected (the attention softmax's own transcendental)
-> K x SwiGLU experts (silu(x@Wg) (*) x@Wu) @ Wd -- only the SELECTED experts are computed
-> weighted mix out = sum_i qmul(w_i, expert_i(x)).
cnt[] increments per COMPUTED expert = the sparsity instrument (K of E, measured not asserted).
Deterministic by construction: exact integer sums (order-free), positional tie-breaks, Q16 fx_exp.
mp = [0]=x(D) [1]=Wr(E*D) [2]=Wg(E blocks of H*D) [3]=Wu(E blocks of H*D) [4]=Wd(E blocks of D*H)
[5]=out(D) [6]=D [7]=H [8]=E [9]=K [10]=cnt(E) [11]=scr(i64 scratch >= E+2K+3H+D slots)
license_tier: ORIGINAL (lib: no main -- build standalone gives rc=102 by design)
dependencies 2 imports · 2 importers
imports: nx_syscalls.nxnx_nofloat_llm.nx
imported by: nx_nofloat_moe_block_gate.nxnx_nofloat_moe_gate.nx
structs
| none |
consts
| none |
functions
| 22 | func nmoe_block_forward(bp: *i64) -> i64 |
| 55 | func nmoe_forward(mp: *i64) -> i64 |