code wiki / tok
topic: tok
4 modules sharing the tok name family (derived from the tree's prefix discipline).
| module | description | lines | funcs |
|---|---|---|---|
| nx_tok_attr_probe.nx | ISOLATE the attribute-text leak (task #9, 2026-07-27). | 98 | 5 |
| nx_tok_debug.nx | GROUND-TRUTH: how does nx_html_next_token tokenize <style>...</style><cite>? | 104 | 7 |
| nx_tok_fold_gate.nx | KATs for the accent-folding UTF-8 tokenizer (the Unicode rung): Latin diacritics | 182 | 7 |
| nx_tok_probe.nx | tokenizer-parity DIAGNOSTIC: dump the exact token ids | 94 | 5 |