code wiki / _hdl_build / nx_nofloat_bpe_gate.nx

nx_nofloat_bpe_gate.nx

buildroot/runtime/_hdl_build/nx_nofloat_bpe_gate.nx

4967 B84 linesdepth 3pulls 3 transitivereach 0 importersview sourcekind gate/prooftopic nofloat
docsdependenciesstructsconstsfunctions

about

nx_nofloat_bpe_gate.nx -- R2 (efficiency) from the sovereign-researcher roadmap (knowledge/research/2026-06-23-nofloat-affordable-land-roadmap.md, grounded in nfs_bpe.raw): a LEARNED Byte-Pair-Encoding tokenizer. Greedily merge the most-frequent adjacent token pair K times -> frequent substrings become single sub-word tokens -> FEWER tokens per text -> cheaper LM training + longer effective context for the same compute. Pure integer, self-contained (nx_syscalls only). T1 COMPRESSION: BPE token count << char count (ratio >= 1.4) = the efficiency win. T2 LOSSLESS (teeth): recursively expand the BPE tokens back to bytes == the original corpus, byte-exact. T3 learned K real merges (each a repeated pair). expect_exit: 0 license_tier: ORIGINAL

dependencies 2 imports · 0 importers

nx_syscalls.nx nx_gate_emit_lib.nx nx_nofloat_bpe_gate.nx

imports: nx_syscalls.nxnx_gate_emit_lib.nx

imported by: nobody (leaf or entry point)

call flow from main pre-order; caps 40 nodes / depth 6 declared; ↻ = already shown

main g_puts sys_write slen sys_mmap g_pn sys_mmap ↻ sys_write ↻ expand expand ↻ g_check g_puts ↻ sys_openat_append sys_write ↻ sys_close sys_exit

structs

none

consts

12const STRIDE: i64 = 320 // pair key = a*STRIDE + b (ids stay < 256+K < 320)

functions

15func slen(s: *u8) -> i64 { var n: i64=0; while s[n]!=(0 as u8){n=n+1} return n }
called by 1: main
17func expand(tok: i64, ma: *i64, mb: *i64, out: *u8, w: *i64) -> i64
called by 2: expandmain calls 1: expand
25func main() -> i64
calls 10: g_putsslensys_mmapg_pnexpandg_check+4