nx_rope.nx

nx_rope.nx -- Rotary Position Embeddings (Su et al. 2021). Position-encoding primitive for modern transformers. Replaces sinusoidal-absolute (Vaswani 2017) and learned-absolute encoding with a multiplicative rotation that injects RELATIVE position into attention scores. RoPE has been adopted by every modern decoder- only LLM: Llama 1/2/3, Mistral, Mixtral, Qwen, Phi-3, Gemma Z-Image / Flux / Stable Diffusion 3 attention Mamba state-space hybrid layers Per the no-skipping cardinal: composes the just-shipped `nx_trig.nx` for cos/sin and `nx_root.nx` for inverse-frequency precompute. No skipping, no inlining trig; full bits-up. ===== Math ======================================================= For a query/key vector q of dimension d (d must be even), pair adjacent dimensions (2i, 2i+1). Each pair has its own frequency theta_i = 1 / base^(2i/d) where base = 10000 (Su's published recommendation). For position m, the rotation angle for pair i is angle_{m,i} = m * theta_i radians. The 2D rotation: q'[2i] = q[2i] * cos(angle_{m,i}) - q[2i+1] * sin(angle_{m,i}) q'[2i+1] = q[2i] * sin(angle_{m,i}) + q[2i+1] * cos(angle_{m,i}) This is a 2x2 orthogonal rotation -- norm-preserving by construction. When Q'K^T is computed downstream, the dot product gains a position-dependent term that depends ONLY on the relative position (m_q - m_k), not the absolute positions. That's the key RoPE property. ===== Inverse-frequency precompute ============================== theta_i = base^(-2i/d) for i in [0, d/2). Computing base^(-2i/d) in i64 + Q10: