nx_nofloat_tokdec.nx

nx_nofloat_tokdec.nx -- byte-level-BPE piece decoder (GPT-2/Qwen convention): GGUF vocab pieces are UTF-8 strings of REMAPPED codepoints (every raw byte 0..255 maps to a printable codepoint; e.g. space -> U+0120 'G-dot' and newline -> U+010A). This is the INVERSE map, so generated token pieces render as REAL text. Mapping (the standard gpt2 bytes_to_unicode inverse): codepoints 33-126, 161-172, 174-255 are themselves; the 68 excluded bytes (0-32, 127-160, 173) were assigned 256+n in ascending byte order -> cp 256..288 -> byte cp-256 (0..32; cp 288 = U+0120 -> byte 32 = SPACE) cp 289 -> byte 127 cp 290..322 -> byte 128+(cp-290) (128..160) cp 323 -> byte 173 Pieces therefore only contain codepoints <= 0x143 (1- or 2-byte UTF-8); anything else passes through raw (defensive -- never drop bytes). Pure functions, no main. license_tier: ORIGINAL