code wiki / (root) / sketch_entropy.nx

sketch_entropy.nx

buildroot/runtime/sketch_entropy.nx

4407 B128 linesdepth 5pulls 5 transitivereach 1 importersview sourcekind sketch/demotopic sketch
docsdependenciesstructsconstsfunctions

about

sketch_entropy.nx -- streaming Shannon entropy estimator. Maintains exact frequency counts via the sovereign hash-map primitive (sketch_hash_map.nx), then computes entropy at query time: H = -Σ p_i log_2(p_i) bits = log_2(N) - (Σ f_i log_2(f_i)) / N USE CASES: - anomaly detection (sudden entropy drop = repetition spike) - change-point detection (entropy distribution shift) - feature extraction for ML over streams - DDoS detection (low entropy = single-source flood) - load-balancer fairness (high entropy = uniform distribution) EXACT WITHIN BOUNDED UNIVERSE: works for streams where distinct keys fit in the hash map (bounded by caller-chosen capacity). For unbounded universes, compose with CountSketch or SpaceSaving for approximate frequencies first. INTEGER LOG-BASE-2 (no f64): log_2(n) ≈ bitlen(n) - 1 (floor) For finer resolution: use top-bit position + fractional bits via mantissa interpolation. v1 uses floor(log_2) -- entropy under- estimates by O(1/N) per term. LOSSLESS-LANGUAGE DISCIPLINE: Returns entropy in PPM ([0, log_2(N) * 1_000_000]). ApproxI64 with NX_ENV_ABS, param_a = n_distinct (quantization bound from floor-log). Maturity = ReferenceImpl (Production when we ship the precise log-fp lookup table).

dependencies 3 imports · 1 importers

syscalls.nx sketch_hash_map.nx sketch_types.nx sketch_entropy.nx sketch_entropy_test.nx

imports: syscalls.nxsketch_hash_map.nxsketch_types.nx

imported by: sketch_entropy_test.nx

structs

37struct StreamingEntropy {

consts

none

functions

44func nx_ent_alloc(capacity: i64) -> *StreamingEntropy {
called by 1: main calls 1: nx_hmap_alloc
56func nx_ent_add(e: *StreamingEntropy, key: i64) -> i64 {
called by 1: main calls 2: nx_hmap_getnx_hmap_put
68func nx_ent_log2_floor(x: i64) -> i64 {
called by 1: nx_ent_bits_ppm
85func nx_ent_bits_ppm(e: *StreamingEntropy) -> i64 {
107func nx_ent_n_distinct(e: *StreamingEntropy) -> i64 {
called by 1: main calls 1: nx_hmap_size
111func nx_ent_n_total(e: *StreamingEntropy) -> i64 {
called by 1: main
117func nx_ent_query(e: *StreamingEntropy) -> *ApproxI64 {
126func nx_ent_memory_bytes(e: *StreamingEntropy) -> i64 {