code wiki / (root) / nx_sketch_tdigest.nx

nx_sketch_tdigest.nx

buildroot/runtime/nx_sketch_tdigest.nx

12025 B340 linesdepth 3pulls 3 transitivereach 19 importersview sourcekind sketch/demotopic sketch
docsdependenciesstructsconstsfunctions

about

sketch_tdigest.nx -- T-Digest (Dunning 2019) tail-tight quantile sketch. Maintain centroids (mean, weight) sorted by mean. Each centroid covers a window of the cumulative-distribution domain [q_left, q_right]. The MAX WEIGHT a centroid may absorb depends on its q-position via a scale function that gives more room to centroids in the middle of the distribution and TIGHTLY BOUNDS centroids near the tails -- the headline T-Digest property. COMPLEMENTS KLL: - KLL: provable uniform rank-error bound, lighter at middle quantiles, looser at tails. - T-Digest: empirically tail-tight (p99/p99.9 accuracy is the headline), no global rank-error guarantee. Together: both shipping means callers pick the right tool per query. DUNNING k1 vs SIMPLIFIED SCALE FUNCTION (v1 trade-off): Dunning's k1(q) = (delta / 2π) · arcsin(2q - 1) → w_max_norm(q) = k1_inv(k1(q) + 1) - q exact: tail w_max ~ sqrt-like; middle ~ delta-bounded. v1 here: w_max_norm(q) = 4·q·(1-q) · π / delta The simplified scale UNDERESTIMATES tail capacity (more conservative = MORE centroids near tails, better tail accuracy than spec). At q=0.5 it gives 0.0314·N matching Dunning within 1%. Trade-off: memory uses slightly more centroids than necessary; queried accuracy is no worse than Dunning's bound, often better at tails. v2 will swap in a tabulated arcsin-based scale. LOSSLESS-LANGUAGE DISCIPLINE (doc 20): Query returns ApproxI64 with envelope_kind = NX_ENV_RANK_ERROR, param_a = ~1% absolute rank error (conservative bound for delta=100), maturity = ReferenceImpl, adv = Honest.

dependencies 2 imports · 1 importers

nx_syscalls.nx nx_sketch_types.nx nx_sketch_tdigest.nx nx_dataframe_approx.nx

imports: nx_syscalls.nxnx_sketch_types.nx

imported by: nx_dataframe_approx.nx

structs

50struct Centroid
55struct TDigest

consts

43const NX_TD_DELTA_DEFAULT: i64 = 100
44const NX_TD_BUFFER_CAP: i64 = 1000
45const NX_TD_MAX_CENTROIDS: i64 = 800 // delta * 8 headroom
47const NX_TD_PI_PPB: i64 = 3141592654 // π · 10^9

functions

66func nx_tdigest_alloc(delta: i64) -> *TDigest
called by 1: dfa_quantile calls 1: sys_mmap
82func nx_tdigest_centroid_at(td: *TDigest, i: i64) -> *Centroid
96func nx_tdigest_w_max(td: *TDigest, cum_w: i64, total: i64) -> i64
called by 1: nx_tdigest_merge
126func nx_tdigest_sort_buffer(td: *TDigest) -> i64
called by 1: nx_tdigest_merge
155func nx_tdigest_merge(td: *TDigest) -> i64
271func nx_tdigest_add(td: *TDigest, value: i64) -> i64
called by 1: dfa_quantile calls 1: nx_tdigest_merge
286func nx_tdigest_quantile(td: *TDigest, p_milli: i64) -> i64
316func nx_tdigest_rank_error_ppb(delta: i64) -> i64
called by 1: nx_tdigest_query
323func nx_tdigest_query(td: *TDigest, p_milli: i64) -> *ApproxI64
334func nx_tdigest_memory_bytes(td: *TDigest) -> i64
338func nx_tdigest_n_centroids(td: *TDigest) -> i64