sketch_tdigest.nx
buildroot/runtime/sketch_tdigest.nx
about
sketch_tdigest.nx -- T-Digest (Dunning 2019) tail-tight quantile sketch.
Maintain centroids (mean, weight) sorted by mean. Each centroid covers
a window of the cumulative-distribution domain [q_left, q_right]. The
MAX WEIGHT a centroid may absorb depends on its q-position via a scale
function that gives more room to centroids in the middle of the
distribution and TIGHTLY BOUNDS centroids near the tails -- the
headline T-Digest property.
COMPLEMENTS KLL:
- KLL: provable uniform rank-error bound, lighter at middle
quantiles, looser at tails.
- T-Digest: empirically tail-tight (p99/p99.9 accuracy is the
headline), no global rank-error guarantee.
Together: both shipping means callers pick the right tool per query.
DUNNING k1 vs SIMPLIFIED SCALE FUNCTION (v1 trade-off):
Dunning's k1(q) = (delta / 2π) · arcsin(2q - 1)
→ w_max_norm(q) = k1_inv(k1(q) + 1) - q
exact: tail w_max ~ sqrt-like; middle ~ delta-bounded.
v1 here: w_max_norm(q) = 4·q·(1-q) · π / delta
The simplified scale UNDERESTIMATES tail capacity (more conservative
= MORE centroids near tails, better tail accuracy than spec). At
q=0.5 it gives 0.0314·N matching Dunning within 1%. Trade-off:
memory uses slightly more centroids than necessary; queried accuracy
is no worse than Dunning's bound, often better at tails.
v2 will swap in a tabulated arcsin-based scale.
LOSSLESS-LANGUAGE DISCIPLINE (doc 20):
Query returns ApproxI64 with envelope_kind = NX_ENV_RANK_ERROR,
param_a = ~1% absolute rank error (conservative bound for delta=100),
maturity = ReferenceImpl, adv = Honest.
dependencies 2 imports · 8 importers
imports: syscalls.nxsketch_types.nx
imported by: _a2a_td_x100.nx_a2a_tdigest_subst.nx_py_quant_substrate.nxsketch_ddsketch_vs_tdigest_tail_bench.nxsketch_observability_dashboard_bench.nxsketch_tdigest_test.nxsketch_tdigest_v2_vs_v1_bench.nxsketch_tdigest_vs_materialized_quantile_bench.nx
structs
| 48 | struct Centroid { |
| 53 | struct TDigest { |
consts
| 37 | const NX_TD_DELTA_DEFAULT: i64 = 100 |
| 38 | const NX_TD_BUFFER_CAP: i64 = 1000 |
| 39 | const NX_TD_MAX_CENTROIDS: i64 = 2000 // bits-up scale: was 800; now supports |
| 45 | const NX_TD_PI_PPB: i64 = 3141592654 // π · 10^9 |
functions
| 66 | func nx_tdigest_alloc(delta: i64) -> *TDigest { |
| 84 | func nx_tdigest_centroid_at(td: *TDigest, i: i64) -> *Centroid { |
| 98 | func nx_tdigest_w_max(td: *TDigest, cum_w: i64, total: i64) -> i64 {
called by 1: nx_tdigest_merge |
| 128 | func nx_tdigest_sort_buffer(td: *TDigest) -> i64 {
called by 1: nx_tdigest_merge |
| 157 | func nx_tdigest_merge(td: *TDigest) -> i64 { |
| 271 | func nx_tdigest_add(td: *TDigest, value: i64) -> i64 { |
| 286 | func nx_tdigest_quantile(td: *TDigest, p_milli: i64) -> i64 { |
| 316 | func nx_tdigest_rank_error_ppb(delta: i64) -> i64 {
called by 1: nx_tdigest_query |
| 323 | func nx_tdigest_query(td: *TDigest, p_milli: i64) -> *ApproxI64 { |
| 334 | func nx_tdigest_memory_bytes(td: *TDigest) -> i64 { |
| 338 | func nx_tdigest_n_centroids(td: *TDigest) -> i64 { |