nx_nofloat_tok.nx
buildroot/runtime/nx_nofloat_tok.nx
about
nx_nofloat_tok.nx -- canonical sovereign BPE tokenizer library for the GGUF vocab (ONE source of truth).
text <-> token ids using tokenizer.ggml.tokens (vocab) + tokenizer.ggml.merges (ranked merge rules).
tk_find_id(target) -> id (string -> id; search vocab)
tk_decode_off(id) -> byte offset of the id's vocab string (read len/ptr via nx_gguf_meta)
tk_bpe_encode(input) -> tokens + ids (single-char start; merge lowest-rank adjacent pair until none)
Gates import this (DRY -- no copy-paste tokenizer code). ASCII inputs only for now (byte-level pretokenize
is identity for printable ASCII; the GPT-2 byte->unicode map for spaces/non-ASCII is the follow-on).
No main (pure library). license_tier: ORIGINAL
dependencies 6 imports · 16 importers
diagram shows first 10 each side; +0 more imports, +6 more importers in the complete lists below.
imports: nx_syscalls.nxnx_tier.nxnx_le.nxnx_tensor.nxnx_gguf.nxnx_gguf_meta.nx
imported by: nx_gpu_export.nxnx_gpu_serve_gate.nxnx_llm_capcheck.nxnx_llm_ppl_bench_gate.nxnx_nofloat_arch_config_gate.nxnx_nofloat_olmoe_forward_gate.nxnx_nofloat_qwen_diag_gate.nxnx_nofloat_qwen_dqprobe_gate.nxnx_nofloat_qwen_fastgen_gate.nxnx_nofloat_qwen_fastgen_i32_gate.nxnx_nofloat_qwen_fastgen_i8_gate.nxnx_nofloat_qwen_gen_gate.nxnx_nofloat_qwen_kvgen_gate.nxnx_nofloat_qwen_text_gate.nxnx_nofloat_qwen_tok_gate.nxnx_nofloat_serve_core.nx
structs
| none |
consts
| 61 | const TK_TCAP: i64 = 262144 |
functions
| 16 | func tk_bytes_eq(a: *u8, b: *u8, n: i64) -> i64 { var i: i64=0; while i<n { if a[i]!=b[i] { return 0 } i=i+1 } return 1 } |
| 19 | func tk_find_id(buf: *u8, first: i64, vocab: i64, target: *u8, tlen: i64) -> i64 |
| 25 | func tk_decode_off(buf: *u8, first: i64, tid: i64) -> i64 |
| 31 | func tk_merge_rank(buf: *u8, mfirst: i64, nm: i64, xp: *u8, xl: i64, yp: *u8, yl: i64, tmp: *u8) -> i64 |
| 43 | func tk_byte_char(b: i64, out: *u8) -> i64 |
| 63 | func tk_h64(p: *u8, n: i64) -> i64 |
| 76 | func tk_tab_build(buf: *u8, first: i64, count: i64, tab: *i64) -> i64 |
| 94 | func tk_tab_get(buf: *u8, tab: *i64, q: *u8, ql: i64) -> i64 |
| 114 | func tk_rank_fast(buf: *u8, mtab: *i64, xp: *u8, xl: i64, yp: *u8, yl: i64, tmp: *u8) -> i64 |
| 129 | func tk_bpe_encode(buf: *u8, mfirst: i64, nm: i64, vfirst: i64, vocab: i64, input: *u8, ilen: i64, tok_ptr: *i64, tok_len: *i64, ids: *i64) -> i64 called by 21: ge2_gpu_embedmainmaingdx_generatemainmain+15 calls 6: sys_mmaptk_byte_chartk_tab_buildtk_rank_fasttk_tab_getsys_munmap |