nx_sketch_tdigest.nx
buildroot/runtime/nx_sketch_tdigest.nx
about
sketch_tdigest.nx -- T-Digest (Dunning 2019) tail-tight quantile sketch.
Maintain centroids (mean, weight) sorted by mean. Each centroid covers
a window of the cumulative-distribution domain [q_left, q_right]. The
MAX WEIGHT a centroid may absorb depends on its q-position via a scale
function that gives more room to centroids in the middle of the
distribution and TIGHTLY BOUNDS centroids near the tails -- the
headline T-Digest property.
COMPLEMENTS KLL:
- KLL: provable uniform rank-error bound, lighter at middle
quantiles, looser at tails.
- T-Digest: empirically tail-tight (p99/p99.9 accuracy is the
headline), no global rank-error guarantee.
Together: both shipping means callers pick the right tool per query.
DUNNING k1 vs SIMPLIFIED SCALE FUNCTION (v1 trade-off):
Dunning's k1(q) = (delta / 2π) · arcsin(2q - 1)
→ w_max_norm(q) = k1_inv(k1(q) + 1) - q
exact: tail w_max ~ sqrt-like; middle ~ delta-bounded.
v1 here: w_max_norm(q) = 4·q·(1-q) · π / delta
The simplified scale UNDERESTIMATES tail capacity (more conservative
= MORE centroids near tails, better tail accuracy than spec). At
q=0.5 it gives 0.0314·N matching Dunning within 1%. Trade-off:
memory uses slightly more centroids than necessary; queried accuracy
is no worse than Dunning's bound, often better at tails.
v2 will swap in a tabulated arcsin-based scale.
LOSSLESS-LANGUAGE DISCIPLINE (doc 20):
Query returns ApproxI64 with envelope_kind = NX_ENV_RANK_ERROR,
param_a = ~1% absolute rank error (conservative bound for delta=100),
maturity = ReferenceImpl, adv = Honest.
dependencies 2 imports · 1 importers
imports: nx_syscalls.nxnx_sketch_types.nx
imported by: nx_dataframe_approx.nx
structs
| 50 | struct Centroid |
| 55 | struct TDigest |
consts
| 43 | const NX_TD_DELTA_DEFAULT: i64 = 100 |
| 44 | const NX_TD_BUFFER_CAP: i64 = 1000 |
| 45 | const NX_TD_MAX_CENTROIDS: i64 = 800 // delta * 8 headroom |
| 47 | const NX_TD_PI_PPB: i64 = 3141592654 // π · 10^9 |
functions
| 66 | func nx_tdigest_alloc(delta: i64) -> *TDigest |
| 82 | func nx_tdigest_centroid_at(td: *TDigest, i: i64) -> *Centroid |
| 96 | func nx_tdigest_w_max(td: *TDigest, cum_w: i64, total: i64) -> i64 called by 1: nx_tdigest_merge |
| 126 | func nx_tdigest_sort_buffer(td: *TDigest) -> i64 called by 1: nx_tdigest_merge |
| 155 | func nx_tdigest_merge(td: *TDigest) -> i64 |
| 271 | func nx_tdigest_add(td: *TDigest, value: i64) -> i64 |
| 286 | func nx_tdigest_quantile(td: *TDigest, p_milli: i64) -> i64 |
| 316 | func nx_tdigest_rank_error_ppb(delta: i64) -> i64 called by 1: nx_tdigest_query |
| 323 | func nx_tdigest_query(td: *TDigest, p_milli: i64) -> *ApproxI64 |
| 334 | func nx_tdigest_memory_bytes(td: *TDigest) -> i64 |
| 338 | func nx_tdigest_n_centroids(td: *TDigest) -> i64 |