sketch_entropy.nx
buildroot/runtime/sketch_entropy.nx
about
sketch_entropy.nx -- streaming Shannon entropy estimator.
Maintains exact frequency counts via the sovereign hash-map primitive
(sketch_hash_map.nx), then computes entropy at query time:
H = -Σ p_i log_2(p_i) bits
= log_2(N) - (Σ f_i log_2(f_i)) / N
USE CASES:
- anomaly detection (sudden entropy drop = repetition spike)
- change-point detection (entropy distribution shift)
- feature extraction for ML over streams
- DDoS detection (low entropy = single-source flood)
- load-balancer fairness (high entropy = uniform distribution)
EXACT WITHIN BOUNDED UNIVERSE: works for streams where distinct
keys fit in the hash map (bounded by caller-chosen capacity).
For unbounded universes, compose with CountSketch or SpaceSaving
for approximate frequencies first.
INTEGER LOG-BASE-2 (no f64):
log_2(n) ≈ bitlen(n) - 1 (floor)
For finer resolution: use top-bit position + fractional bits via
mantissa interpolation. v1 uses floor(log_2) -- entropy under-
estimates by O(1/N) per term.
LOSSLESS-LANGUAGE DISCIPLINE:
Returns entropy in PPM ([0, log_2(N) * 1_000_000]).
ApproxI64 with NX_ENV_ABS, param_a = n_distinct (quantization
bound from floor-log). Maturity = ReferenceImpl (Production
when we ship the precise log-fp lookup table).
dependencies 3 imports · 1 importers
imports: syscalls.nxsketch_hash_map.nxsketch_types.nx
imported by: sketch_entropy_test.nx
structs
| 37 | struct StreamingEntropy { |
consts
| none |
functions
| 44 | func nx_ent_alloc(capacity: i64) -> *StreamingEntropy { |
| 56 | func nx_ent_add(e: *StreamingEntropy, key: i64) -> i64 { |
| 68 | func nx_ent_log2_floor(x: i64) -> i64 {
called by 1: nx_ent_bits_ppm |
| 85 | func nx_ent_bits_ppm(e: *StreamingEntropy) -> i64 { |
| 107 | func nx_ent_n_distinct(e: *StreamingEntropy) -> i64 { |
| 111 | func nx_ent_n_total(e: *StreamingEntropy) -> i64 {
called by 1: main |
| 117 | func nx_ent_query(e: *StreamingEntropy) -> *ApproxI64 { |
| 126 | func nx_ent_memory_bytes(e: *StreamingEntropy) -> i64 {
calls 1: nx_hmap_memory_bytes |