code wiki / _hdl_build / nx_nofloat_adamw32_gate.nx
nx_nofloat_adamw32_gate.nx
buildroot/runtime/_hdl_build/nx_nofloat_adamw32_gate.nx
about
nx_nofloat_adamw32_gate.nx -- the REAL win from the R4 scale-up work: a numerically STABLE fixed-point Adam.
DIAGNOSIS: the Q16 AdamW (lib nfa_adamw) DIVERGES/STALLS on small gradients because g^2 UNDERFLOWS in Q16 --
e.g. g=0.06 (Q16 3932) -> qmul(g,g)=235 (0.0036) and the (1-b2)*g^2 increment qmul(66,235)=0, so the 2nd
moment v never accumulates, the denominator collapses to eps, and the update mhat/eps explodes (or stalls).
FIX: accumulate v in Q32 and compute g^2 = g*g at FULL precision (no >>16) so small gradients survive.
T1: Q32-moment AdamW CONVERGES on a small-gradient fit (loss -> ~0).
T2 (measured exceed): Q32 final loss << Q16 final loss on the SAME problem (the underflow fix is real).
Self-contained controlled experiment (n=8 fit, hand-coded grad 2*(w-t), no transformer). Pure integer.
expect_exit: 0 Sovereign: nx_nofloat_autograd (for nfa_adamw + nfa_isqrt) + nx_syscalls.
dependencies 3 imports · 0 importers
imports: nx_nofloat_autograd.nxnx_syscalls.nxnx_gate_emit_lib.nx
imported by: nobody (leaf or entry point)
call flow from main pre-order; caps 40 nodes / depth 6 declared; ↻ = already shown
structs
| none |
consts
| 13 | const Q16: i64 = 65536 |
functions
| 18 | func adamw32_one(w: *i64, g: *i64, m: *i64, v: *i64, n: i64, lr: i64, b1: i64, b2: i64, eps: i64, t: i64) -> i64 |
| 39 | func loss_q32(w: *i64, t: *i64, n: i64) -> i64 { var s: i64=0; var i: i64=0; while i<n { let d: i64=w[i]-t[i]; s=s+d*d; i=i+1 } return s } called by 1: main |
| 41 | func fill_grad(w: *i64, t: *i64, g: *i64, n: i64) -> i64 { var i: i64=0; while i<n { g[i]=2*(w[i]-t[i]); i=i+1 } return 0 } called by 1: main |
| 43 | func main() -> i64 |