code wiki / (root) / nx_f32_bpe_load.nx

nx_f32_bpe_load.nx

buildroot/runtime/nx_f32_bpe_load.nx

6694 B182 linesdepth 7pulls 13 transitivereach 28 importersview sourcekind librarytopic f32

docs dependencies structs consts functions

about

nx_f32_bpe_load.nx -- load BPE tokenizer vocabulary from GGUF. Walks the GGUF metadata section for tokenizer.ggml.tokens (the vocabulary as an array of strings) and tokenizer.ggml.merges (BPE merge rules), populating an NxBpeVocab ready for nx_bpe_encode / nx_bpe_decode. This closes the last 'text-in' gap to running real Qwen2.5 / Llama / etc. inference end-to-end on the f32 substrate. genealogy_id: ggml_tokenizer_metadata_keys + gpt2_byte_bpe_spec lineage_id: substrate_f32_bpe_load_v1

dependencies 8 imports · 28 importers

diagram shows first 10 each side; +0 more imports, +18 more importers in the complete lists below.

imports: nx_syscalls.nxnx_tier.nxnx_le.nxnx_intern.nxnx_bpe.nxnx_gguf.nxnx_gguf_load.nxnx_gguf_meta.nx

imported by: nx_batched_gate.nxnx_batched_tput.nxnx_chat_warmcache.nxnx_coder_gen_f32.nxnx_companion_chat.nxnx_companion_voice.nxnx_embed_bench.nxnx_f32_bpe_load_test.nxnx_f32_embed_gate.nxnx_f32_llm_bench_test.nxnx_f32_llm_chat_test.nxnx_f32_llm_cohere_test.nxnx_f32_llm_live_gen_test.nxnx_f32_llm_serve.nxnx_f32_qwen_capital_gate.nxnx_llm_argmax_probe.nxnx_llm_batch_serve.nxnx_llm_france_probe.nxnx_llm_layerlens_probe.nxnx_llm_sched_gate.nxnx_paged_fwd_gate.nxnx_qwen_native_gate.nxnx_qwen_wsl_timing_gate.nxnx_reason_paged_probe.nxnx_reason_probe.nxnx_reasoning_gate.nxnx_specdec_gate.nxnx_tok_probe.nx

structs

none

consts

23	const NX_FBL_OK: nx_int = 0
24	const NX_FBL_ERR_NULL: nx_int = 1
25	const NX_FBL_ERR_NO_TOKENS: nx_int = 2
26	const NX_FBL_ERR_NOT_ARRAY: nx_int = 3
27	const NX_FBL_ERR_NOT_STRING: nx_int = 4
28	const NX_FBL_ERR_OOM: nx_int = 5
29	const NX_FBL_N_VERDICTS: nx_int = 6

functions

31	func nx_fbl_verdict_is_valid(v: nx_int) -> nx_int called by 1: main
40	func _fbl_load_tokens(buf: u8, len: i64, hdr: NxGgufHeader, called by 1: nx_f32_bpe_load_from_gguf calls 7: sys_mmap nx_gguf_meta_find nx_gguf_meta_array_inner_type nx_gguf_meta_array_count nx_gguf_meta_array_first_elt_off nx_le_read_u64+1
80	func _fbl_load_merges(buf: u8, len: i64, hdr: NxGgufHeader, called by 1: nx_f32_bpe_load_from_gguf calls 8: sys_mmap nx_gguf_meta_find nx_gguf_meta_array_inner_type nx_gguf_meta_array_count nx_gguf_meta_array_first_elt_off nx_le_read_u64+2
158	func nx_f32_bpe_load_from_gguf(buf: u8, len: i64, hdr: NxGgufHeader, called by 28: main main main main main main+22 calls 2: _fbl_load_tokens _fbl_load_merges