code wiki / (root) / nx_tok_probe.nx

nx_tok_probe.nx

buildroot/runtime/nx_tok_probe.nx

3645 B94 linesdepth 14pulls 62 transitivereach 0 importersview sourcekind probetopic tok
docsdependenciesstructsconstsfunctions

about

nx_tok_probe.nx -- tokenizer-parity DIAGNOSTIC: dump the exact token ids (and each id's decoded string) that nx_bpe_encode_bytelevel produces for an arithmetic prompt vs a known-good word prompt. Loads GGUF vocab ONLY (no weights) -- fast. Answers mechanically whether digits are being merged (Qwen2.5 pre-tokenizer splits every digit into its OWN token; a merged "47" would be a context the model never saw in arithmetic). expect_exit: 0

dependencies 9 imports · 0 importers

nx_syscalls.nx nx_tier.nx nx_le.nx nx_strconv.nx nx_bpe.nx nx_gguf.nx nx_gguf_load.nx nx_f32_bpe_load.nx nx_reasoning.nx nx_tok_probe.nx

imports: nx_syscalls.nxnx_tier.nxnx_le.nxnx_strconv.nxnx_bpe.nxnx_gguf.nxnx_gguf_load.nxnx_f32_bpe_load.nxnx_reasoning.nx

imported by: nobody (leaf or entry point)

call flow from main pre-order; caps 40 nodes / depth 6 declared; ↻ = already shown

main sys_mmap sys_read_file sys_openat_rd sys_lseek sys_mmap ↻ sys_read sys_close nx_gguf_parse nx_le_read_u32 nx_le_read_u64 nx_le_read_u32 ↻ _gguf_skip_value nx_le_read_u64 ↻ nx_le_read_u32 ↻ _gguf_skip_value ↻ sys_mmap ↻ nx_bpe_vocab_new sys_mmap ↻ nx_intern_new nx_intern_next_pow2 nx_hash_new nx_f32_bpe_load_from_gguf _fbl_load_tokens sys_mmap ↻ nx_gguf_meta_find nx_le_read_u64 ↻ nx_le_read_u32 ↻ _gmeta_bytes_equal nx_gguf_skip_value _gguf_skip_value ↻ nx_gguf_meta_array_inner_t nx_le_read_u32 ↻ nx_gguf_meta_array_count nx_le_read_u64 ↻ nx_gguf_meta_array_first_e nx_le_read_u64 ↻ nx_bpe_add_token nx_intern_get nx_hash_fnv1a_bytes

structs

none

consts

none

functions

19func tp_w(s: *u8) -> i64 { var n: i64 = 0; while s[n] != (0 as u8) { n = n + 1 } sys_write(1, s, n); return 0 }
called by 2: tp_wntp_dump calls 1: sys_write
20func tp_wn(v: i64) -> i64
called by 1: tp_dump calls 4: sys_mmaptp_wnx_strconv_format_i64sys_write
28func tp_wtext(b: *u8, n: i64) -> i64
called by 1: tp_dump calls 2: sys_mmapsys_write
41func tp_dump(v: *NxBpeVocab, label: *u8, text: *u8, n: nx_int) -> i64
called by 1: main calls 6: tp_wtp_wtextsys_mmapnx_bpe_encode_byteleveltp_wnnx_bpe_decode_bytelevel
63func main() -> i64
calls 7: sys_mmapsys_read_filenx_gguf_parsenx_bpe_vocab_newnx_f32_bpe_load_from_ggufnx_reason_build_arith_prompt+1