nx_bpe.nx

nx_bpe.nx -- Byte-Pair Encoding tokenizer. Closes the TEXT-INTERFACE gap. With this brick the substrate can convert input text -> token IDs (encode) and token IDs -> text (decode), completing the input/output bookends around the transformer compute pipeline. Combined with nx_gguf (model loader, 7b4fafc6), nx_embedding (token lookup, 84725593), nx_transformer_block (forward, 8efc7edb), and nx_token_sample (logits sampling, 68e63138), the substrate now has the COMPLETE end-to-end inference path -- pure NishiLang, zero PyTorch, zero CUDA, zero llama.cpp dependency. ===== Algorithm (Sennrich/Haddow/Birch 2016) ===================== BPE starts with a base vocabulary of bytes (256 entries 0..255). Training collects an ordered list of merge rules: pairs of token IDs that recur frequently get merged into a new token ID. Encoding (Sennrich 2016, Algorithm 1): 1. Split input text into single bytes (each is an initial token). 2. Find the adjacent token-pair with the LOWEST rank (= highest priority) in the merge table. 3. Apply that merge: replace the pair with the merged token ID. 4. Repeat until no pair has a rank. Decoding: concatenate the byte sequences of each token in order. ===== Substrate composition ===================================== nx_intern.nx -- canonical byte-string <-> ID table (the vocab itself) nx_loop -- bounded loops; budget = n_input * n_merges hard ceiling per JPL Rule 2 v1 perf: linear scan over the merge table per merge-decision step. For 50k merges + 100-token output that's ~5M comparisons total. O(1) pair lookup via nx_map (key = a*MAX_VOCAB + b) is the queued v2 perf upgrade.