code wiki / (root) / nx_nofloat_moe.nx

nx_nofloat_moe.nx

buildroot/runtime/nx_nofloat_moe.nx

5207 B122 linesdepth 8pulls 16 transitivereach 2 importersview sourcekind librarytopic nofloat
docsdependenciesstructsconstsfunctions

about

nx_nofloat_moe.nx -- no-float Mixture-of-Experts FFN (2026-07-15, MoE rung 1: the layer MECHANISM). The July-2026 open-weight leaders (DeepSeek V4 / Qwen3.5 / Llama 4-5 / Kimi K2) are ALL sparse-MoE -- the arch census's #1 gap. The f32 TRAINER half exists (nx_f32_moe_train_gate 6/6, 07-10); this is the INFERENCE half in pure Q16 integer, composed from the proven no-float primitives: router logits r = x @ Wr^T ([out,in] GGUF convention, mm_out_in) -> deterministic top-K (strict >, LOWEST index on ties -- positional, never order-dependent) -> fx_exp softmax over the K selected (the attention softmax's own transcendental) -> K x SwiGLU experts (silu(x@Wg) (*) x@Wu) @ Wd -- only the SELECTED experts are computed -> weighted mix out = sum_i qmul(w_i, expert_i(x)). cnt[] increments per COMPUTED expert = the sparsity instrument (K of E, measured not asserted). Deterministic by construction: exact integer sums (order-free), positional tie-breaks, Q16 fx_exp. mp = [0]=x(D) [1]=Wr(E*D) [2]=Wg(E blocks of H*D) [3]=Wu(E blocks of H*D) [4]=Wd(E blocks of D*H) [5]=out(D) [6]=D [7]=H [8]=E [9]=K [10]=cnt(E) [11]=scr(i64 scratch >= E+2K+3H+D slots) license_tier: ORIGINAL (lib: no main -- build standalone gives rc=102 by design)

dependencies 2 imports · 2 importers

nx_syscalls.nx nx_nofloat_llm.nx nx_nofloat_moe.nx nx_nofloat_moe_block_gate.nx nx_nofloat_moe_gate.nx

imports: nx_syscalls.nxnx_nofloat_llm.nx

imported by: nx_nofloat_moe_block_gate.nxnx_nofloat_moe_gate.nx

structs

none

consts

none

functions

22func nmoe_block_forward(bp: *i64) -> i64
called by 1: main calls 2: rmsnorm_gamma_rownmoe_forward
55func nmoe_forward(mp: *i64) -> i64