nx_gelu.nx

nx_gelu.nx -- Gaussian Error Linear Unit activation. The activation used by every non-Llama transformer family: GPT-2 / GPT-3 / GPT-Neo / GPT-J BERT / RoBERTa / DistilBERT / ALBERT T5 / mT5 / ByT5 ViT / DeiT / Swin / CLIP / SigLIP PaLM, Gemma (uses GeGLU variant = GELU * linear gate) Llama / Mistral / Qwen use SiLU instead (closely related; nx_silu.nx already ships). Substrate now ships both -- model loader picks per-config. ===== Math ======================================================= Exact GELU (Hendrycks & Gimpel 2016 eq. 1): GELU(x) = x * Phi(x) = x * 0.5 * (1 + erf(x / sqrt(2))) Exact requires erf, which we don't ship. Two practical approximations: 1. Tanh form (most common): GELU(x) ~= 0.5 * x * (1 + tanh(sqrt(2/pi) * (x + 0.044715*x^3))) 2. Sigmoid form (Hendrycks 2016 alt): GELU(x) ~= x * sigmoid(1.702 * x) We use form #2 because nx_sigmoid_q10 already ships in nx_silu.nx and it's a one-multiply + sigmoid composition. Max relative error vs exact GELU is ~0.5% across [-5, 5] -- well within substrate quantization budget. Tanh-form upgrade queued -- needs nx_tanh.nx primitive. For q4_K-quant transformer inference the sigmoid form is more than sufficient. Q-format: x in Q10, output in Q10. Constant 1.702 in Q10 = 1743.