code wiki / (root) / nx_tok_probe.nx

nx_tok_probe.nx

buildroot/runtime/nx_tok_probe.nx

3645 B94 linesdepth 14pulls 63 transitivereach 0 importersview sourcekind probetopic tok
docsdependenciesstructsconstsfunctions

about

nx_tok_probe.nx -- tokenizer-parity DIAGNOSTIC: dump the exact token ids (and each id's decoded string) that nx_bpe_encode_bytelevel produces for an arithmetic prompt vs a known-good word prompt. Loads GGUF vocab ONLY (no weights) -- fast. Answers mechanically whether digits are being merged (Qwen2.5 pre-tokenizer splits every digit into its OWN token; a merged "47" would be a context the model never saw in arithmetic). expect_exit: 0

dependencies 9 imports · 0 importers

nx_syscalls.nx nx_tier.nx nx_le.nx nx_strconv.nx nx_bpe.nx nx_gguf.nx nx_gguf_load.nx nx_f32_bpe_load.nx nx_reasoning.nx nx_tok_probe.nx

imports: nx_syscalls.nxnx_tier.nxnx_le.nxnx_strconv.nxnx_bpe.nxnx_gguf.nxnx_gguf_load.nxnx_f32_bpe_load.nxnx_reasoning.nx

imported by: nobody (leaf or entry point)

call flow from main pre-order; caps 40 nodes / depth 6 declared; ↻ = already shown

main sys_mmap nxa_die sys_write sys_exit nxa_lock_take nxa_lock_addr sys_write ↻ nxa_lock_give nxa_lock_addr ↻ nxa_report_overrun sys_write ↻ nxa_dump_printable sys_write ↻ nxa_dump_sizes sys_write ↻ sys_read_file sys_openat_rd sys_lseek sys_mmap ↻ sys_read sys_munmap sys_close nx_gguf_parse nx_le_read_u32 nx_le_read_u64 nx_le_read_u32 ↻ _gguf_skip_value nx_le_read_u64 ↻ nx_le_read_u32 ↻ _gguf_skip_value ↻ sys_mmap ↻ nx_bpe_vocab_new sys_mmap ↻ nx_intern_new nx_intern_next_pow2 nx_hash_new nx_f32_bpe_load_from_gguf _fbl_load_tokens sys_mmap ↻

structs

none

consts

none

functions

19func tp_w(s: *u8) -> i64 { var n: i64 = 0; while s[n] != (0 as u8) { n = n + 1 } sys_write(1, s, n); return 0 }
called by 2: tp_wntp_dump calls 1: sys_write
20func tp_wn(v: i64) -> i64
called by 1: tp_dump calls 4: sys_mmaptp_wnx_strconv_format_i64sys_write
28func tp_wtext(b: *u8, n: i64) -> i64
called by 1: tp_dump calls 2: sys_mmapsys_write
41func tp_dump(v: *NxBpeVocab, label: *u8, text: *u8, n: nx_int) -> i64
called by 1: main calls 6: tp_wtp_wtextsys_mmapnx_bpe_encode_byteleveltp_wnnx_bpe_decode_bytelevel
63func main() -> i64
calls 7: sys_mmapsys_read_filenx_gguf_parsenx_bpe_vocab_newnx_f32_bpe_load_from_ggufnx_reason_build_arith_prompt+1