code wiki / (root) / nx_token_permutation.nx

nx_token_permutation.nx

buildroot/runtime/nx_token_permutation.nx

9893 B274 linesdepth 2pulls 2 transitivereach 1 importersview sourcekind librarytopic token
docsdependenciesstructsconstsfunctions

about

nx_token_permutation.nx -- order-effect analyzer for prompt token sequences. Foundation primitive for reasoning about PROMPT ORDER EFFECTS in image-gen models. Given two ordered token-id sequences that share the same multi-set (i.e., a permutation of each other), measure: 1. Kendall tau distance # of pairwise inversions; the canonical permutation-distance metric 2. Position-0 changed did the first token swap? (CLIP and T5 both weight position-0 heavily) 3. Position-last changed did the trailing token swap? (some text encoders weight the EOS- adjacent position differently) 4. New-neighbor pairs pairs of tokens now adjacent that weren't before (attention is denser between adjacent tokens; new pairs means new latent joint-meaning) 5. Same-multi-set check did the two sequences share tokens? (if not, this analysis doesn't apply and the verdict is NOT_PERMUTATION) Output also carries a SEALED-ENUM verdict per the dual-reading cardinal: NX_PERMUT_IDENTICAL identical sequences NX_PERMUT_ADJACENT_SWAP single adjacent transposition NX_PERMUT_MINOR_REORDER tau >= 0.85 NX_PERMUT_MODERATE_REORDER tau in [0.5, 0.85) NX_PERMUT_SEVERE_REORDER tau < 0.5 NX_PERMUT_NOT_PERMUTATION different multi-sets USE CASES: - "beautiful tall girl" vs "tall beautiful girl" -> ADJACENT_SWAP position-0 changed -> for CLIP-based models, predict ~35% attention shift; for T5-based models (Z-Image), predict ~15% shift. - "beautiful tall girl" vs "girl tall beautiful" -> SEVERE_REORDER tau ~ 0; position-0 + position-last both changed; new neighbor pairs (girl-tall, tall-beautiful where before was beautiful-tall). - "tall girl" vs "tall beautiful girl" -> NOT_PERMUTATION (different multi-sets; "added a word" case the user asked about).

dependencies 2 imports · 1 importers

nx_syscalls.nx nx_tier.nx nx_token_permutation.nx nx_token_permutation_test.nx

imports: nx_syscalls.nxnx_tier.nx

imported by: nx_token_permutation_test.nx

structs

75struct PermutationReport

consts

64const NX_PERMUT_Q: nx_int = 1024
67const NX_PERMUT_IDENTICAL: nx_int = 0
68const NX_PERMUT_ADJACENT_SWAP: nx_int = 1
69const NX_PERMUT_MINOR_REORDER: nx_int = 2
70const NX_PERMUT_MODERATE_REORDER: nx_int = 3
71const NX_PERMUT_SEVERE_REORDER: nx_int = 4
72const NX_PERMUT_NOT_PERMUTATION: nx_int = 5
73const NX_PERMUT_N_VERDICTS: nx_int = 6

functions

92func _permut_same_multi_set(a: *i64, n_a: nx_int, b: *i64, n_b: nx_int) -> nx_int
127func _permut_build_position_map(a: *i64, b: *i64, n: nx_int, pos_buf: *i64) -> nx_int
153func _permut_count_inversions(pos_buf: *i64, n: nx_int) -> nx_int
170func _permut_count_new_neighbors(a: *i64, b: *i64, n: nx_int) -> nx_int
196func nx_token_permutation_compute(
270func nx_token_permutation_verdict_is_valid(v: nx_int) -> nx_int
called by 1: main