Honest scope, stated first. Every Nishi number below is measured on our own instruments and carries its method. The resolve rate is a sovereign NishiLang analog under the SWE-bench contract (FAIL_TO_PASS pass AND PASS_TO_PASS hold, fresh compile+run the sole judge) — not the Python 500-instance Verified set the rival agents run; no parity is claimed until our F787 public-set oracle leg lands. Rival cells stay fetch-pending until a sourced public number is ingested — this board never fabricates a competitor's score.
| axis | Nishi (measured) | Claude Code · Gemini · Kimi · GLM · OpenAI · OSS (SWE-agent/OpenHands) |
|---|---|---|
| Resolve rate (SWE-bench contract) |
92% (13/14), adopted 3-maker ensemble (Qwen 1.5B + coder-1.5B + official Coder-3B), every counted green additionally property-verified; the one holdout is an ill-posed multi-function instance. Live: /compare/autograde | Sourced 2026-08-09 (on their benchmarks — a different, harder task; our F787 public-set leg is calibrating): SWE-bench Verified frontier cluster — GPT-5.5 88.7% · Claude Opus 4.8 88.6% · Gemini 3.1 Pro 80.6% (Verified is saturated, and a 2026 contamination audit found frontier models reproducing verbatim gold patches — memorization, not always solving). SWE-bench Pro now discriminates: Claude Mythos 5 80.3% · GLM-5.2 top open-weight 62.1% (vs GPT-5.5 58.6%) · Kimi K2.7 Code unsubmitted (MCP Mark Verified 81.1%). OSS agent recipes on Verified with 32B-class open makers: 37–62% (OpenHands-LM 37.2 · SWE-Agent-LM 40.2 · SWE-Master-32B 61.4) — our seat runs 1.5–3B-class makers. |
| Crash & billing durability | Zero greens lost across 3 host kills + 2 VM wipes in one day (2026-08-09): state lives in the ecosystem (per-episode tree-side ledgers, TTL-expiring claims), never in the seat. | A rented seat's mid-rung death loses the rung; spend-limit stops end the run. Unmeasured pending public incident data. |
| Cost per closed rung | Near-zero marginal — local open-weight makers under our own harness; the engine-shift gauge (frontier-tokens-per-closed-workstream) fell 210,913 → 171,807 and is still falling. | Per-token at provider rates. Unmeasured pending published pricing × measured rungs. |
| Zero-frontier-LLM autonomy | The grading heartbeat re-measures every target and republishes this site daily with zero frontier-LLM tokens — both the grade leg and the publish leg proven live 2026-08-09. | Operator-driven sessions; scheduled autonomy unmeasured. |
| Verification depth of greens | 100% of counted greens are property-verified over exhaustive input sweeps (symbolic judge, EXH mode); a tests-green overfit is demoted to MISS, never counted. | Test-suite-only acceptance; no property layer. |
Resolve rate is the field's own yardstick. The other four are where a sovereign seat structurally differs from a rented one: durability (crash-immune by construction), cost (weights are a config row), autonomy (the ecosystem grades itself on a schedule), and verification depth (an unyielding symbolic judge makes ensembling diverse makers sound — union greens cannot admit a wrong fix).
Data: knowledge/store/s3bench- (v2 render, 2026-08-09; page-generator
organ is a filed rung). Resolve-rate provenance: official ledger batch ts=1786301334, graded by
nx_swebench_local_gate, published by the autograde pulse. Rival numbers live-fetched
2026-08-09: codeant.ai/blogs/swe-bench-scores · codingfleet.com/blog/swe-bench-pro-leaderboard-2026 ·
emergent.sh/learn/glm-5-2-vs-kimi-k2-7-code · vm0.ai/en/models/kimi-k2-7-code · arXiv 2602.03411
(SWE-Master) · arXiv 2407.16741 (OpenHands) · tokenmix.ai/blog/swe-bench-2026 — full trail in the
s3b-sources plane row. External-set progress: the F806 oracle swept the FULL 500-instance Verified population —
24 CALIBRATED (the sovereign attempt worklist) · 25 near-miss contract mismatches · 451 honest
env refusals (the bare-py3.12 era ceiling; per-era envs are the named expansion). First sovereign
attempt cell executed (declared oracle-file retrieval, tests as sole judge): honestly unresolved —
the baseline the recursive loop now grows. swebvcal-/swebvres- planes.