code wiki / (root) / nx_f32_llm_special_tokens.nx

nx_f32_llm_special_tokens.nx

buildroot/runtime/nx_f32_llm_special_tokens.nx

3773 B74 linesdepth 7pulls 9 transitivereach 25 importersview sourcekind librarytopic f32
docsdependenciesstructsconstsfunctions

about

nx_f32_llm_special_tokens.nx -- read BOS/EOS/PAD token ids from GGUF. Composes with nx_f32_llm_run: the caller passes the eos_token_id arg. Pre-this-brick, callers had to hardcode it. Post: read it from the GGUF and pass it through. Production runs no longer loop forever on models that emit EOS to terminate generation. Metadata keys (GGUF v3, gpt2-style tokenizers + Llama tokenizers): tokenizer.ggml.bos_token_id (u32) beginning-of-sequence tokenizer.ggml.eos_token_id (u32) end-of-sequence tokenizer.ggml.padding_token_id (u32) optional padding token All three return -1 if not present in metadata. genealogy_id: ggml_tokenizer_special_token_keys lineage_id: substrate_f32_llm_special_tokens_v1

dependencies 4 imports · 25 importers

nx_syscalls.nx nx_tier.nx nx_gguf.nx nx_gguf_meta.nx nx_f32_llm_special_tokens.nx nx_batched_gate.nx nx_batched_tput.nx nx_chat_warmcache.nx nx_coder_gen_f32.nx nx_companion_chat.nx nx_companion_voice.nx nx_f32_llm_bench_test.nx nx_f32_llm_chat_test.nx nx_f32_llm_cohere_test.nx nx_f32_llm_live_gen_test.nx

diagram shows first 10 each side; +0 more imports, +15 more importers in the complete lists below.

imports: nx_syscalls.nxnx_tier.nxnx_gguf.nxnx_gguf_meta.nx

imported by: nx_batched_gate.nxnx_batched_tput.nxnx_chat_warmcache.nxnx_coder_gen_f32.nxnx_companion_chat.nxnx_companion_voice.nxnx_f32_llm_bench_test.nxnx_f32_llm_chat_test.nxnx_f32_llm_cohere_test.nxnx_f32_llm_live_gen_test.nxnx_f32_llm_serve.nxnx_f32_llm_special_tokens_test.nxnx_f32_qwen_capital_gate.nxnx_llm_argmax_probe.nxnx_llm_batch_serve.nxnx_llm_france_probe.nxnx_llm_layerlens_probe.nxnx_llm_sched_gate.nxnx_paged_fwd_gate.nxnx_qwen_native_gate.nxnx_qwen_wsl_timing_gate.nxnx_reason_paged_probe.nxnx_reason_probe.nxnx_reasoning_gate.nxnx_specdec_gate.nx

structs

none

consts

none

functions

23func _flst_read_u32_key(buf: *u8, len: i64, hdr: *NxGgufHeader,
35func nx_f32_llm_read_bos(buf: *u8, len: i64, hdr: *NxGgufHeader) -> nx_int
called by 1: main calls 2: sys_mmap_flst_read_u32_key
49func nx_f32_llm_read_eos(buf: *u8, len: i64, hdr: *NxGgufHeader) -> nx_int
63func nx_f32_llm_read_pad(buf: *u8, len: i64, hdr: *NxGgufHeader) -> nx_int
called by 1: main calls 2: sys_mmap_flst_read_u32_key