code wiki / _hdl_build / _t2_bigram_lm_authored.nx

_t2_bigram_lm_authored.nx

buildroot/runtime/_hdl_build/_t2_bigram_lm_authored.nx

18130 B429 linesdepth 6pulls 9 transitivereach 0 importersview sourcekind tool
docsdependenciesstructsconstsfunctions

about

_t2_bigram_lm_authored.nx -- T2 rung 2a: the team's FIRST LANGUAGE MODEL. A next-token model over the TRAINED BPE vocab (reloaded from knowledge/store/bpe_vocab_v1.txt -- this rung is also the proof the persisted vocab round-trips into a consumer) on the team corpus. Smallest honest LM: logits = W[cur] (vocab x vocab table; multiclass logistic regression = CONVEX, so zero-init is exact and no symmetry trap exists), verified-CE loss, full-batch accumulation + gated AdamW, deterministic early stop. THE ORACLE IS ANALYTIC: from the integer bigram/unigram counts of the SAME token stream the optimal achievable cross-entropies are computed in closed form -- H_uni = -(1/N) sum_b c(b) ln(c(b)/N) (context-free baseline) H_bi = -(1/N) sum_ab c(ab) ln(c(ab)/c(a)) (the bigram optimum) GATES: V vocab reloads (64 BPEMERGE rows parse) | O structure exists (H_bi < H_uni) | L LEARNED: final CE < H_uni AND CE <= midpoint (H_uni+H_bi)/2 -- the model provably uses context; distance-to-optimum reported in permil, never asserted | B bit-exact two-run (the full 320x320 table identical). Emits the TRIAGE training report (train_report_t2lm.log; joins train_reports.conf for the pulse sweep) + MODELWRIGHT T2LM evidence rows + a deterministic greedy sample as evidence. LAWS: struct-free, flat ifs, no &&/||, measured never asserted. license_tier: ORIGINAL

dependencies 1 imports · 0 importers

nx_tgrad_core.nx _t2_bigram_lm_authored.nx

imports: nx_tgrad_core.nx

imported by: nobody (leaf or entry point)

call flow from main pre-order; caps 40 nodes / depth 6 declared; ↻ = already shown

main _tg_puts sys_write sys_openat_append lm_bp _tg_puts ↻ lm_fp lm_bn _tg_num sys_mmap sys_write ↻ lm_fn sys_now_realtime_sec sys_mmap ↻ sys_clock_gettime_real lm_load_vocab sys_read_file sys_openat_rd sys_lseek sys_mmap ↻ sys_read sys_close lm_match lm_keynum lm_match ↻ lm_load_corpus sys_read_file ↻ lm_match ↻ lm_encode nx_f32_div nx_f32_classify nx_f32_exp_field nx_f32_mant_field nx_f32_sign nx_f32_mant_field ↻ nx_f32_exp_field ↻ nx_i32_to_f32 nx_f32_sub nx_f32_add nx_f32_classify ↻

structs

none

consts

20const LM_VCAP: i64 = 320
21const LM_MAXTOK: i64 = 1200
22const LM_EPOCHS: i64 = 200
23const LM_LOGEVERY: i64 = 10

functions

25func lm_fp(fd: i64, s: *u8) -> i64 { var n: i64=0; while s[n]!=(0 as u8){n=n+1} sys_write(fd,s,n); return 0 }
called by 2: lm_bpmain
26func lm_fn(fd: i64, v: i64) -> i64 { let bb: *u8=sys_mmap(28); var m: i64=v; if m<0{m=0-m;sys_write(fd,"-" as *u8,1)}; let t: *u8=sys_mmap(28); var k: i64=0; if m==0{t[0]=48;k=1}; while m>0{t[k]=48+(m%10);m=m/10;k=k+1}; var i: i64=0; while i<k{bb[i]=t[k-1-i];i=i+1}; sys_write(fd,bb,k); return 0 }
called by 2: lm_bnmain
27func lm_bp(fd: i64, s: *u8) -> i64 { _tg_puts(s); if fd >= 0 { lm_fp(fd, s) } return 0 }
called by 2: lm_trainmain calls 2: _tg_putslm_fp
28func lm_bn(fd: i64, v: i64) -> i64 { _tg_num(v); if fd >= 0 { lm_fn(fd, v) } return 0 }
called by 2: lm_trainmain calls 2: _tg_numlm_fn
29func lm_match(b: *u8, i: i64, n: i64, lit: *u8) -> i64
38func lm_keynum(b: *u8, ls: i64, le: i64, key: *u8) -> i64
called by 1: lm_load_vocab calls 1: lm_match
59func lm_load_vocab(path: *u8, merges: *i64) -> i64
called by 1: main calls 3: sys_read_filelm_matchlm_keynum
87func lm_load_corpus(conf: *u8, buf: *u8, cap: i64) -> i64
called by 1: main calls 2: sys_read_filelm_match
127func lm_encode(bytes: *u8, n: i64, merges: *i64, mcount: i64, seq: *i64) -> i64
called by 1: main
159func lm_decode_tok(tok: i64, merges: *i64, out: *u8, opos: i64) -> i64
called by 1: main
184func lm_train(tape: *i64, nb: *i64, arena: *i64, ab: *i64, toks: *i64, ntok: i64, w: *i64, stopthr: i64, logfd: i64, out2: *i64, curve: *i64, curven: *i64) -> i64
254func main(argc: i64, argv: *i64) -> i64