Data Engineering & Analytics
Nishi's sovereign, integer-exact analytics stack — measured against dbt, Spark, DuckDB, ClickHouse, Arrow, pandas/Polars, Superset, DataSketches and OpenLineage. Every score is graded by nx_dataeng_sota_census with a liar-kill (each axis opens a real organ); this page is generated from that state, never hand-asserted.
616meet-SOTA / 1000
10Have
13Partial
5Gap
2Exceed
30 axes · 0 ungrounded (liar-kill armed). Started this arc at 333 — the area flagged "toy level"; +283 across 8 gated organs.
The honest read: real primitives existed but sat unwired. A gated analytics engine was built on them — point at columns (in-memory or from the sovereign store), get exact or approximate aggregates, a plain-English profile, the relationships between them, trends over time, and query them. Integer-exact means bit-reproducible aggregates — a determinism edge float tools cannot match.
The gated analytics stack
| Organ | Gate | Capability |
|---|---|---|
nx_dataframe | 9/9 | exact aggregation: count / sum / min / max / mean / variance / stddev / quantile / filter / GROUP BY / join |
nx_dataframe_approx | 5/5 | HLL distinct + t-digest quantile at scale, measured vs exact truth (HLL 6 permil error) |
nx_analyst_data | 6/6 | profile any column + plain-English read (categorical / id / constant / skew / outliers) |
nx_analyst_multi | 8/8 | Pearson correlation + strength class + key-driver finder |
nx_analyst_report | 5/5 | "analyze this dataset" front door — composes the whole stack into one report |
nx_timeseries | 6/6 | rolling mean + least-squares trend + growth over time |
nx_analyst_store | 4/4 | runs the whole stack on real stored data — seg_store scan → extract → analyze |
nx_query | 7/7 | query executor: SELECT count/sum/avg/min/max(col) WHERE col op val GROUP BY col, integer-exact |
Census axes vs SOTA
| Axis | Status | Grounding |
|---|---|---|
| Aggregation front door | HAVE | count/sum/avg/min/max/percentile over a column, integer-exact (nx_dataframe) |
| Descriptive stats (general) | HAVE | mean/var/stddev/quantile over arbitrary columns |
| Approx analytics at scale | HAVE | HLL distinct + t-digest, measured vs exact (killed the unmeasured-claim debt) |
| Multivariate correlation | HAVE | Pearson + key-driver; discovered price~5*size-2*age from data alone |
| General data analyst | HAVE | profile + find patterns + explain a whole dataset (nx_analyst_report) |
| Stored-data analytics | HAVE | the analyst runs on real seg_store records (round-trip proven) |
| Query engine | PARTIAL | SELECT agg WHERE GROUP BY executor, integer-exact (nx_query 7/7); string-syntax parser = finer gap |
| Timeseries analytics | PARTIAL | rolling/trend/growth done; resample + seasonality = finer gap |
| Quantile sketch | PARTIAL | t-digest tail-tight (p99 10 permil) but median coarse; interpolation follow-on |
| Columnar format | PARTIAL | nx_seg_columnar RLE+dict primitive; unwired to a query layer |
| Data lineage | PARTIAL | nx_store_lineage: organ->store PRODUCER/CONSUMER edges (OpenLineage-class) |
| Data contracts | PARTIAL | nx_seg_schema: Avro-class schema + backward/forward evolution-safety |
| Transform DAG (dbt-class) | GAP | models + data tests + docs + semantic layer |
| Vectorized / distributed exec | GAP | batch/vectorized + multi-node (blocked-adjacent: the 1-core threading gap) |
| BI dashboard | GAP | query-driven interactive dashboards (needs the UI arc) |
| Arrow interchange | GAP | zero-copy columnar interchange format |
Sovereign exceeds
| Axis | Status | Why beyond mainstream |
|---|---|---|
| Integer-exact analytics | EXCEED | 100%-integer aggregates are bit-reproducible; DuckDB/pandas float sums are order-dependent, ours are not |
| Own-stack, zero deps | EXCEED | the whole primitive stack (HLL++/t-digest/columnar/store) is sovereign — no Arrow/DataSketches/JVM |
Honest remaining gaps
Five GAPs — a transform-DAG, vectorized + distributed execution (blocked on the 1-core threading gap), a BI dashboard, and Arrow interchange. The next make-it-real step is scaling the stored-data path past the 256-per-key store cap. Every score here is measured and re-runnable.
Generated by
nx_dataeng_compare_page composing the sovereign S-class kit — TLS 1.3, no third-party web server, no external fonts or assets. Back to the compare hub