code wiki / (root) / sketch_kmv.nx

sketch_kmv.nx

buildroot/runtime/sketch_kmv.nx

10291 B310 linesdepth 4pulls 5 transitivereach 5 importersview sourcekind sketch/demotopic sketch
docsdependenciesstructsconstsfunctions

about

sketch_kmv.nx -- KMV (K-Minimum-Values) sketch. Bar-Yossef et al. 2002 + Beyer 2007. Maintain the K smallest 32-bit hash values seen. Direct ancestor of DataSketches' Theta sketches; we ship the foundation primitive first then build Theta on top. CAPABILITIES (set-operation cardinality): - cardinality: (K-1) * 2^32 / kth_smallest when |set| > K total_inserted_unique when |set| <= K - union: K smallest across A.values ∪ B.values - Jaccard: |A.kmins ∩ B.kmins| / |A.kmins ∪ B.kmins| unbiased estimator of true Jaccard Error: rel_stddev ~ 1/sqrt(K-1) at 1-sigma. For K=4096: 1.56% stddev. For K=16384: 0.78%. COMPLEMENTS HLL: - HLL: better space per accuracy point for pure cardinality. - KMV: SAME sketch supports union AND intersection (Jaccard). The trade-off is well-studied; both ship in DataSketches (HLL.java and Theta.java) -- we ship the same axis pair. Hash domain: 32-bit (murmur3_32). We store hashes as i64 to keep arithmetic simple; the value space is [0, 2^32). Hash 0 is mapped to 1 (matches Cuckoo's empty-sentinel pattern; KMV uses the special "all-slots-filled-with-max" sentinel via the n_items counter instead, but mapping 0->1 avoids edge cases in the cardinality formula).

dependencies 3 imports · 3 importers

syscalls.nx murmur3.nx sketch_types.nx sketch_kmv.nx sketch_kmv_jaccard_vs_naive_bench. sketch_kmv_test.nx sketch_theta.nx

imports: syscalls.nxmurmur3.nxsketch_types.nx

imported by: sketch_kmv_jaccard_vs_naive_bench.nxsketch_kmv_test.nxsketch_theta.nx

structs

41struct Kmv {

consts

35const NX_KMV_K_MIN: i64 = 16
36const NX_KMV_K_MAX: i64 = 65536
39const NX_KMV_HASH_MAX: i64 = 4294967296

functions

50func nx_kmv_alloc(k: i64, seed: i64) -> *Kmv {
65func nx_kmv_hash(kmv: *Kmv, key: *u8, len: i64) -> i64 {
called by 1: nx_kmv_add calls 1: murmur3_32
78func nx_kmv_add(kmv: *Kmv, key: *u8, len: i64) -> i64 {
called by 2: mainmain calls 1: nx_kmv_hash
118func nx_kmv_add_hash(kmv: *Kmv, raw_h: i64) -> i64 {
called by 1: main
161func nx_kmv_estimate(kmv: *Kmv) -> i64 {
172func nx_kmv_stddev_rel_ppb(k: i64) -> i64 {
called by 1: nx_kmv_query
182func nx_kmv_query(kmv: *Kmv) -> *ApproxI64 {
198func nx_kmv_union(a: *Kmv, b: *Kmv) -> *Kmv {
called by 2: mainmain calls 1: nx_kmv_alloc
262func nx_kmv_jaccard_ppm(a: *Kmv, b: *Kmv) -> i64 {
called by 2: mainmain
308func nx_kmv_memory_bytes(kmv: *Kmv) -> i64 {
called by 1: main