nishi code wiki / research / generational SOTA analysis
Generational SOTA analysis: how to score a layered ecosystem and cluster its work
Compiled 2026-08-01. Corpus slug: project-nishi-generational-sota-2026-08-01. Labels: SHIPPING = in production use in the field · RESEARCH = paper/preprint · OURS = measured in this ecosystem. Everything unverifiable is in the declared UNVERIFIED section.
Question this brief answers. The ecosystem's family tree has 8 generations (gen 0 god primitives → gen 7 products). What does July-2026 research say about how to (a) score each generation against state-of-the-art, (b) prioritize across them, and (c) cluster the resulting work into logical parts? Product of this brief: /code/sota, the generational SOTA analysis.
Forks off — the lineage
This library builds on itself: every brief declares what it forks off, so future work forks off this one rather than re-deriving it.
| parent | what this brief inherits from it |
|---|---|
project-nishi-debt-intelligence-sota-2026-07-31 (corpus; partially transcribed at /code/families) | The July-2026 incident-clustering canon: normalize-before-embed, density-based clustering with an explicit noise class, topology-constrained correlation, risk-reduction/effort ranking capped at 5–8 recommendations, ladder grading. Inherited wholesale as the clustering half of this method. |
reference-agentic-sota-july-2026-and-what-we-adopted (corpus) | The frontier positioning: agentic-coding benchmarks are near-saturated (SWE-bench Verified 77.2%); the real SOTA gap is the substrate. Inherited as the reason per-generation analysis matters more than chasing one saturated axis. |
reference-substrate-program-order-and-costs-2026-08-01 (corpus) | Order-of-work doctrine: check whether sovereignty is free before budgeting to fight for it; silicon→gpu→kernel ordering. Inherited as the effort-tiering instinct in the work order. |
| Real-time rendering brief (published 2026-07-31) | The format contract this area enforces: sourced+dated claims, evidence labels, units, declared gaps. |
Claims — sourced, dated, labelled
| # | claim | source · date | label |
|---|---|---|---|
| 1 | Defect cost concentrates in the tightly-coupled core. Components in the core/central portion of a core–periphery architecture carry significantly more defect-related activity than peripheral ones; a small number of components generate a large proportion of total system cost. Measured via Design Structure Matrices over real systems. | Sturtevant / MacCormack / Baldwin et al., “Technical debt and system architecture: the impact of coupling on defect-related activity”, J. Systems & Software · 2016 (foundational, still the cited baseline in 2026 ATD work) | RESEARCH |
| 2 | Architecture technical debt is traced and prioritized along the dependency graph, not per-file: the dependency approach reveals lifecycle (introduction → propagation → repayment) and makes automated identification/prioritization tractable. | arXiv 2501.15387, “Tracing the Lifecycle of Architecture Technical Debt: a Dependency Approach” · Jan 2025 | RESEARCH |
| 3 | Platform maturity is graded ordinally on named dimensions, and gaps are prioritized per dimension (CNCF: investment, adoption, interfaces, operations, measurement; Gartner: 8 capability areas). The 2026 state-of-platform data shows a persistent measurement crisis: 29.6% of teams measure no success metric at all — the differentiator is measuring, not building. | CNCF TAG App Delivery maturity model (standing) · platformengineering.org, “Platform engineering maturity in 2026” / State of Platform Engineering Vol 4 · 2026 | SHIPPING |
| 4 | Golden paths are the adoption mechanism: coverage grows only as fast as someone hand-writes plumbing unless extending a plane costs data, not code. | CNCF model (above) · proven here 2026-07-31: deploy plane 27 near-duplicate implementations → 1 row-driven path, coverage 333‰→666‰ by rows alone (/code/sotaplan R2) | SHIPPING + OURS |
| 5 | Clustering work: topology beats text; keep an explicit noise class; cap recommendations at 5–8. Correlation constrained by the real dependency graph (TopoEvo arXiv 2605.15611); density methods that decline to cluster outliers (GPTrace, ICSE’26, arXiv 2512.01609); postmortem practice on recommendation caps. Inherited via the debt-intelligence brief and confirmed on our own ledger (anchor clustering, B³ F1 865‰). | arXiv 2605.15611 · May 2026; arXiv 2512.01609 · ICSE 2026; parent brief 2026-07-31 | RESEARCH + OURS |
| 6 | The agentic-coding benchmark axis is near-saturated and misaligned (SWE-bench Verified SOTA 77.2%, <10%→>70% in ~a year; position paper argues the benchmarks no longer measure agentic engineering) — so “get to SOTA” effort belongs on the layers beneath, which is exactly what a generational analysis surfaces. | llm-stats SWE-bench Verified · July 2026; arXiv 2606.17799 · June 2026 | RESEARCH |
| 7 | Autonomous root-cause/remediation agents are not yet reliable (every frontier model below 50% on ITBench-AA, launched 2026-05-27) — build the deterministic measurement/ranking layer; leave closed-loop autonomy out of the design. | ITBench-AA (Artificial Analysis + IBM Research) · May 2026, via parent brief | RESEARCH |
The synthesized method (what /code/sota implements)
| step | rule | from |
|---|---|---|
| 1 | Strata = the real topology. Generations of the import DAG are the unit of analysis — never thematic buckets (claim 2, 5). | DSM / TopoEvo |
| 2 | A different SOTA bar per stratum. Roots: external-oracle proof. Hubs: mutation-proof, ruler-readable coverage. Bulk: golden path. Libraries: adoption. Services: readiness. Products: field benchmarks (claims 1, 3, 4). | core–periphery economics + CNCF dimensions |
| 3 | Headline = MIN across strata, never the mean; and lower generations first — evidence on a leaf buys nothing until its chain lifts. | ours (banked law), consistent with claim 1 |
| 4 | Orphans are a declared noise class, never force-clustered (claim 5). | HDBSCAN practice |
| 5 | 5–8 clusters, each naming ONE act, blast radius a measured term (descendant reach), effort tiers declared as assessed (claims 5, 7). | postmortem practice |
| 6 | Every number names its instrument and date; assessed judgments are labelled assessed (claim 3's measurement crisis, inverted). | CNCF 2026 data |
Declared UNVERIFIED
| 1 | The Gartner platform-engineering maturity model (doc 7348030) is paywalled — cited from its public abstract only; the “8 capability areas” count is from the abstract, unread in full. |
| 2 | Vendor-blog adoption figures for 2026 platform engineering (e.g. “80% adoption”) are unreplicated marketing numbers — excluded from the claims table. |
| 3 | The SSRN comparative multivocal study of maturity models (abstract 6710038) was not read beyond its abstract. |
| 4 | ITBench-AA numbers are inherited from the parent brief, not re-fetched this session. |
| 5 | The RED/AMBER colors on /code/sota are assessed from named evidence, not derived by an instrument — the per-generation rollup organ that would derive them is declared future work there. Until it exists, this method is only partially self-enforcing. |
What this brief fed, and how to fork it
Fed: /code/sota — SOTA by generation (2026-08-01): 8 generation bands, each with its own bar; 7 work clusters, generation-ordered.
To fork this brief: copy the format, add a “Forks off” row pointing here (research_gensota / corpus slug above), and supersede any claim you re-verify with a newer source rather than deleting it — additive, like everything else here.