nx_silu.nx

nx_silu.nx -- SiLU / Swish activation: x * sigmoid(x). Closes the FFN activation gap. Together with nx_rmsnorm (shipped dfce1e32) this lets NishiLang run a full Llama-class transformer block end-to-end: input -> RMSNorm -> attention -> RMSNorm -> FFN (matmul -> SiLU -> matmul) -> output. SiLU = Sigmoid Linear Unit = x * sigmoid(x). * Ramachandran et al. 2017 _Searching for Activation Functions_ * Elfwing et al. 2018 _Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning_ Modern transformer FFN uses SiLU exclusively (or its closely- related variant GeGLU which gates by gelu instead). Used by: Llama 2/3 SwiGLU FFN (uses SiLU as the gate) Mistral, Mixtral, Qwen, Gemma SwiGLU FFN Z-Image / Flux / Stable Diffusion 3 attention FFN ===== Math ======================================================= sigmoid(x) = 1 / (1 + exp(-x)) silu(x) = x * sigmoid(x) Q10 fixed-point: For x >= 0: e_neg = exp(-x) ; sigmoid = Q10 / (Q10 + e_neg) For x < 0: e_neg = exp(x) ; sigmoid = e_neg / (Q10 + e_neg) (Both branches use only the negative-domain exp lookup -- no overflow risk.) Then silu = x * sigmoid / Q10. Per the bits-up cardinal: composes against nx_exp_q10_neg (canonical exp primitive) NxTensor (canonical L1 container) LoopVerdict (bounded-loop discipline) genealogy_id: ramachandran_2017_silu + elfwing_2018_sil + hendrycks_gimpel_2016_gelu (sibling activation) lineage_id: substrate_silu_v1