nishi code wiki / research / measuring code quality qualitatively
Measuring code quality qualitatively: the ungameable rulers
Researched 2026-07-29 (2-agent verified web sweep, primary sources); transcribed to this library 2026-08-01. Corpus slug: reference-code-quality-eval-sota-2026-07-29; full copy also on NAS knowledge/status/quality_ruler_research_20260729.md. Labels: RESEARCH paper/preprint · SHIPPING production practice · OURS measured in this ecosystem.
Question this brief answers. How do you measure code quality when numeric gates get gamed? The field's July-2026 consensus states our own finding exactly: “gates green” ≠ quality. The headline measurements: METR (03/2026) had 4 maintainers review 296 agent PRs — ~half of test-passing PRs would be rejected; Cursor (06/2026) found 63% of Opus “successes” RETRIEVED the fix (web/git history) rather than derived it — a hardened sandbox drops the score 87.1%→73.0%.
Forks off — the lineage
| parent | what this brief inherits |
|---|---|
project-nishi-model-tier-transcript-census-2026-07-29 (corpus, STUB) | The incident that motivated the sweep: tool-error rate mistaken for quality. |
| research_gensota · sibling | Mutation testing as a non-vacuity exit criterion — this brief is where that practice was sourced; gensota applied it to the gate fleet (R4). |
Fed: the D001 non-vacuity program (mutation sweeps, negative controls, BITE-proven cells in nx_gate_verdict), the R4 rung on /code/sotaplan, and the render-critique loop already live in the uigen lane (design #4).
The 10 transferable designs — ranked for a self-hosted harness with local judges
| # | design | source · date | label |
|---|---|---|---|
| 1 | Blocker/rubric split: mechanical blockers FIRST (build, lint, tests, diff-size/file-boundary scope caps); the rubric judge sees only blocker-passers and can never rescue a blocker failure. Claimed +81% lower misclassification vs SWE-bench Pro. | Cognition FrontierCode · 06/2026 | SHIPPING |
| 2 | Non-vacuity + protect the oracle: every agent-submitted test must FAIL on pre-change code (kills assert-True gaming); oracle/test paths outside the agent-writable tree, any write there auto-RED; strip .git, default-deny egress. | EvilGenie · Cursor sandbox · 2026 | SHIPPING + OURS (the gate-bite law) |
| 3 | Mutation score = the ungameable numeric gate: plant N mutants, the suite must kill ≥k or green doesn't count. Hollow suites exposed mechanically. | Meta at scale 2025 · Stryker incremental (1–5min/PR) | SHIPPING + OURS (R4 sweep: 7 survivors/16, 3 fixed) |
| 4 | Checklist-anchored VLM judging: render → TIMED screenshots incl post-interaction → local VLM verifies ONE checklist item per call. 94.4% human-ranking agreement; score visual and code modalities separately. Small judges are reliable only when verifying concrete items against evidence. | ArtifactsBench, arXiv 2507.04952 (open-source) | RESEARCH + OURS (render-critique loop live) |
| 5 | Judge hygiene: pairwise beats pointwise BUT an order flip swings >10% — run both orders + 2–3 paraphrases, unstable verdict = tie; a panel of 2–3 diverse small judges beats one big one; report judge-vs-human agreement beside every score (OpenAI's own trained grader: 66% vs experts; 95% self-consistency ≠ validity). | CodeJudgeBench · CriterAlign · Braintrust · 2026 | RESEARCH |
| 6 | Trajectory anti-pattern detectors — deterministic over the action log, no judge: verification-skip, search-loop, oracle-edit, derived-vs-retrieved, length-vs-baseline. Failures are epistemic and start in the first few steps; failure runs are 12–82% longer ⇒ leading indicators. | TraceProbe 2607.06184 · AgentLens 2607.06624 (the “Lucky Pass Problem”: 2605.12925) | RESEARCH |
| 7 | Self-punishing chained tasks: stage-2 agent builds ON stage-1 output; B's success rate IS A's maintainability score — no judge. Measured: agent code 2.3× more verbose, 2× more structurally eroded than human baselines. | SlopCodeBench 2603.24755 · CodeThread 2606.21804 | RESEARCH |
| 8 | Lagging truth from git (post-hoc, ungameable in-session): 2-week churn (<12% healthy, >25% critical), duplicate-block delta (+81% under AI), error-masking constructs (+47%), moved-vs-added (refactoring collapsed 21%→3.8%). Segment by authoring tier. DORA 2025: AI throughput UP, instability UP ⇒ revert/rework/incident are the honest quality gauges; throughput is a cost gauge. | GitClear 2026 (623M lines) · DORA 2025 | SHIPPING |
| 9 | Held-out randomized tests + an honest ceiling: grade on a random draw the agent never saw; cap the max honest score — TOO-PERFECT IS ITSELF AN ALARM. | CapCode 2606.07379 | RESEARCH |
| 10 | Contamination defense: date-stamped post-cutoff tasks, a private never-published holdout, grep every pass for solution-text overlap with the task description. | SWE-bench+ / -Live / -Pro · 2025–26 | SHIPPING |
Declared UNVERIFIED / caveats (the agents flagged these themselves)
| 1 | Judge-human agreement is only “moderate” everywhere measured — nobody claims a local judge matches experts unassisted; anchor judges in deterministic evidence first. |
| 2 | Long transcripts degrade judge reliability (RuVerBench) — one rubric item per call on an evidence slice. |
| 3 | LLMs cannot judge semantic equivalence (EquiBench ≈ coin-flip) — use differential execution, never a judge. |
| 4 | Vendor acceptance/retention statistics are marketing-grade and excluded. |
| 5 | The 28.6%→0.6% hacked-rate-with-trajectory-monitor figure is real but unattributed to an exact paper. |
| 6 | ✓ CLOSED 2026-08-01: nx_page_ingest fixed (debt 1785623390 EATEN); the arXiv citations above (2507.04952, 2607.06184, 2607.06624, 2603.24755, 2606.07379) are ARCHIVED in the sovereign library. |
How this lands on Nishi
Designs 2 and 3 are already doctrine here (non-vacuity teeth, BITE-proven cells, the R4 mutation sweep with 7 surviving mutants found and 3 fixed); design 4 is live as the uigen render-critique loop; design 1's blocker/rubric split maps onto the promote guard + gate_run bound. The open adoptions, in leverage order: trajectory anti-pattern detectors over our own harness journals (design 6 — deterministic, no judge, leading indicators), chained-task maintainability scoring for the generation lanes (design 7), and the GitClear-style lagging gauges over sovgit history (design 8).