nx_ngram.nx

nx_ngram.nx -- n-gram extraction + frequency + dispersion. Second brick of the corpus-linguistics S-class substrate per cardinal feedback-corpus-linguistics-s-class-substrate-sketch-engine-displacement. === What this primitive answers ==================================== Given a stream of token IDs and N (unigram / bigram / trigram / ...), produce: - extraction of every n-gram in the stream - hash-interned IDs for each n-gram (FNV-1a over the token-ID sequence) - frequency count per n-gram - dispersion: how evenly the n-gram is distributed across the corpus (Juilland's D — close to 1 = uniform, close to 0 = clumped) - top-K most-frequent extraction === Composition ==================================================== nx_string_ops.nx (nx_str_hash_fnv1a) -- token-sequence interning nx_essentials.nx (nx_log2_floor) -- log2 for dispersion nx_tier.nx -- nx_int alias === Sealed-enum frequency bands ==================================== Per dual-reading cardinal: every primitive emits BOTH a quantitative count AND a qualitative sealed-enum band derived from log-rank. log2(freq) buckets: 0..3 -> SINGLETON_RARE (hapax / once or twice) 3..7 -> RARE (3-127) 7..12 -> COMMON (128-4095) 12.. -> VERY_COMMON (4096+) === Dispersion bands =============================================== Juilland's D in Q10: 0..256 -> CLUMPED (very uneven; <25% uniform) 256..614 -> MODERATELY_SPREAD 614..870 -> WELL_SPREAD 870..1024 -> EVEN