code wiki / _hdl_build / _bpe_vocab_authored.nx

_bpe_vocab_authored.nx

buildroot/runtime/_hdl_build/_bpe_vocab_authored.nx

12900 B322 linesdepth 2pulls 2 transitivereach 0 importersview sourcekind tool
docsdependenciesstructsconstsfunctions

about

_bpe_vocab_authored.nx -- T2 rung 1 of the training-substrate ladder: TRAIN a BPE vocab on the TEAM CORPUS (the read side existed: nx_f32_bpe_load; the TRAINING side was the research-flagged gap -- this rung lands it at the smallest honest scale: greedy byte-pair, deterministic tie-break = lowest-left-then-lowest-right, no regex pretokenization; honest scope noted in the registry). Corpus is conf-driven (law 11): knowledge/status/bpe_corpus.conf rows "C <path>". GATES: K hand-computed merge oracle on the classic aaabdaaabac corpus (merges MUST come out (97,97),(97,98),(256,257) -- construction-known, computed by hand in this header: aa x4 wins; then tie Za=2 vs ab=2 -> ab by lowest-left; then Z,257 x2) | D determinism (two trains -> identical merge tables) | R LOSSLESS roundtrip (encode->decode == original bytes for EVERY corpus line) | C compression MEASURED (tokens < bytes; permil reported, never asserted). Vocab persisted durable + reloadable: knowledge/store/bpe_vocab_v1.txt BPEMERGE rows. Evidence: MODELWRIGHT T2 rows -> modelwright.log. Tamper hook: argv[1] conf override. LAWS: struct-free, flat ifs, no &&/||. license_tier: ORIGINAL

dependencies 1 imports · 0 importers

nx_syscalls.nx _bpe_vocab_authored.nx

imports: nx_syscalls.nx

imported by: nobody (leaf or entry point)

call flow from main pre-order; caps 40 nodes / depth 6 declared; ↻ = already shown

main bp_w sys_write sys_openat_append sys_exit bp_bp bp_w ↻ bp_bn bp_wn sys_mmap sys_write ↻ sys_now_realtime_sec sys_mmap ↻ sys_clock_gettime_real sys_mmap ↻ bp_train bp_load sys_mmap ↻ sys_read_file sys_openat_rd sys_lseek sys_mmap ↻ sys_read sys_close bp_match sys_close ↻ bp_encode bp_decode_tok sys_mmap ↻ bp_wn ↻ sys_openat_wr

structs

none

consts

18const BP_MERGES: i64 = 64
19const BP_VCAP: i64 = 320
20const BP_MAXB: i64 = 65536

functions

22func bp_w(fd: i64, s: *u8) -> i64 { var n: i64=0; while s[n]!=(0 as u8){n=n+1} sys_write(fd,s,n); return 0 }
called by 2: bp_bpmain calls 1: sys_write
23func bp_wn(fd: i64, v: i64) -> i64 { let bb: *u8=sys_mmap(28); var m: i64=v; if m<0{m=0-m;sys_write(fd,"-" as *u8,1)}; let t: *u8=sys_mmap(28); var k: i64=0; if m==0{t[0]=48;k=1}; while m>0{t[k]=48+(m%10);m=m/10;k=k+1}; var i: i64=0; while i<k{bb[i]=t[k-1-i];i=i+1}; sys_write(fd,bb,k); return 0 }
called by 2: bp_bnmain calls 2: sys_mmapsys_write
24func bp_bp(fd: i64, s: *u8) -> i64 { bp_w(1, s); if fd >= 0 { bp_w(fd, s) } return 0 }
called by 1: main calls 1: bp_w
25func bp_bn(fd: i64, v: i64) -> i64 { bp_wn(1, v); if fd >= 0 { bp_wn(fd, v) } return 0 }
called by 1: main calls 1: bp_wn
26func bp_match(b: *u8, i: i64, n: i64, lit: *u8) -> i64
called by 1: bp_load
37func bp_load(conf: *u8, buf: *u8) -> i64
called by 1: main calls 3: sys_mmapsys_read_filebp_match
79func bp_train(bytes: *u8, n: i64, seq: *i64, counts: *i64, merges: *i64) -> i64
called by 1: main
131func bp_encode(bytes: *u8, off: i64, n: i64, merges: *i64, mcount: i64, seq: *i64) -> i64
called by 1: main
165func bp_decode_tok(tok: i64, merges: *i64, out: *u8, opos: i64) -> i64
called by 1: main calls 1: sys_mmap
188func main(argc: i64, argv: *i64) -> i64
calls 14: bp_wsys_openat_appendsys_exitbp_bpbp_bnsys_now_realtime_sec+8