code wiki / _hdl_build / nx_fnet_mlm_gate.nx
nx_fnet_mlm_gate.nx
buildroot/runtime/_hdl_build/nx_fnet_mlm_gate.nx
about
nx_fnet_mlm_gate.nx -- GATE for MODEL-002: a tiny MASKED LANGUAGE MODEL on the FNet mixer. FNet's Fourier
mix is BIDIRECTIONAL (every position sees every other), so its native LM objective is masked-LM (BERT-style),
NOT causal next-token (which would let the model see the answer). Here: one position of a 4-token palindrome
[a,b,b,a] is replaced by a MASK token; the model fills it in -- which requires MIXING the mirror position into
the masked one (a pure bag-of-words model cannot). No attention anywhere.
tokens(+MASK) -> EMBED(trained) -> FNET mix -> relu FFN -> softmax-CE over the vocab at the masked slot
G_train masked-token accuracy >= 7/8 AND final loss < first loss.
G_repro bit-exact: train twice -> identical accuracy + final-loss bits.
Evidence -> knowledge/status/fnet_mlm.log (FNETMLMGATE authored=organ ... verdict=GREEN). license_tier: ORIGINAL
dependencies 2 imports · 0 importers
imports: nx_autograd_tensor.nxnx_syscalls.nx
imported by: nobody (leaf or entry point)
call flow from main pre-order; caps 40 nodes / depth 6 declared; ↻ = already shown
structs
| none |
consts
| 15 | const MV: i64 = 5 // input vocab: tokens 0..3 + MASK=4 |
| 16 | const MN: i64 = 4 // sequence length |
| 17 | const MD: i64 = 4 // d_model |
| 18 | const MH: i64 = 8 // FFN hidden |
| 19 | const MC: i64 = 4 // output vocab (predict token 0..3) |
| 20 | const MND: i64 = 16 // MN*MD |
| 22 | const ML_LOG: *u8 = "knowledge/status/fnet_mlm.log" |
functions
| 24 | func ml_w(fd: i64, s: *u8) -> i64 { var n: i64 = 0; while s[n] != (0 as u8) { n = n + 1 } sys_write(fd, s, n); return 0 } |
| 25 | func ml_wn(fd: i64, v: i64) -> i64 |
| 36 | func ml_embed(E: *i64, seq: *i64, soff: i64, xout: *i64) -> i64 |
| 47 | func ml_fwd(tape: *i64, vals: *i64, st: *i64, x: *i64, nW1: i64, nb1: i64, nW2: i64, nb2: i64) -> i64 |
| 54 | func ml_build(tape: *i64, vals: *i64, st: *i64, E: *i64, W1: *i64, b1: *i64, W2: *i64, b2: *i64, |
| 85 | func ml_predict(tape: *i64, vals: *i64, st: *i64, E: *i64, W1: *i64, b1: *i64, W2: *i64, b2: *i64, seqs: *i64, s: i64) -> i64 |
| 105 | func ml_adamw(p: *i64, m: *i64, v: *i64, g: *i64, n: i64, lr: i64, beta1: i64, beta2: i64, om1: i64, om2: i64, eps: i64, c1: i64, c2: i64) -> i64 |
| 117 | func ml_train(tape: *i64, vals: *i64, grads: *i64, st: *i64, E: *i64, W1: *i64, b1: *i64, W2: *i64, b2: *i64, |
| 195 | func main() -> i64 |