nx_wordpiece_lib.nx

nx_wordpiece_lib.nx -- SOVEREIGN WordPiece tokenizer over a BERT uncased vocab.txt, a library so the cross-encoder arm of nx_beir_eval and its gate drive it in-process (search rung R0, 2026-09-14). ABSENT-PROVEN before writing (nx_absent wordpiece buildroot/runtime nx: matches=0 coverage_complete=1). The vocab is read whole through sys_read_file (no cap), every token is hashed into an open-addressing table sized from the token count, and every hit is verified by bytes, so a hash collision can never return the wrong id. Basic tokenisation follows the HF BertTokenizer with do_lower_case: ASCII lowercase, every byte below 33 splits, every ASCII punctuation byte is its own token, a word longer than WP_MAX_WORD_CHARS is [UNK] whole, then greedy longest-match wordpiece with the two-byte continuation prefix. DEVIATION, stated: bytes above 0x7f are kept as word characters (no NFD accent stripping, no CJK isolation), so an accented word normally becomes [UNK] pieces. license_tier: ORIGINAL