nx_rmsnorm.nx

nx_rmsnorm.nx -- Root Mean Square Layer Normalisation. Closes a substrate gap: prior to this commit, NishiLang had matmul + attention + softmax + quantization shipped, but NO normalisation primitive. That makes the substrate unable to run a transformer block end-to-end (every modern transformer normalises between attention and FFN sublayers). RMSNorm vs LayerNorm choice (Zhang & Sennrich 2019): LayerNorm (Ba/Kiros/Hinton 2016): y = (x - mean(x)) / sqrt(var(x) + eps) * gamma + beta RMSNorm (Zhang & Sennrich 2019, simpler): y = x / sqrt(mean(x^2) + eps) * gamma RMSNorm drops the centering term (mean subtraction). Faster (~50% speedup per Zhang 2019 Table 2) at same downstream quality on translation + speech recognition + summarisation benchmarks. Modern transformer architectures use RMSNorm exclusively: Llama 1/2/3, Mistral, Mixtral, Qwen, Phi-3, Gemma Z-Image, Stable Diffusion 3, Flux Mamba state-space models LayerNorm support remains queued -- the math composes against the same nx_isqrt_q10 with one extra centering pass. ===== Math ======================================================= For each token i, hidden dim D, input x[i, :] of length D: sum_sq = sum_{d=0}^{D-1} x[i, d]^2 mean_sq = sum_sq / D rms = sqrt(mean_sq + eps) y[i, d] = x[i, d] / rms * gamma[d] gamma is the per-channel learned scale vector (length D), initialised to 1.0 (Q10=1024) at network construction time.