Nishi vs the Field -- Evaluation and Grading (Referee)

Evaluation and Grading (Referee) — mechanically measured, liar-killed, sovereign.

Capability	Nishi	Scorecard	openai-evals	EvalPlus	HELM	Notes
Property-based independent oracles (never author-supplied answers)	Yes	No	Part	Part	No	Sort proven ordered AND a permutation, isqrt by bounds, pow by cross-method agreement; a deliberately BUGGY sort is CAUGHT (the verifier has teeth, gate GREEN 2026-07-06 era); EvalPlus test augmentation is the partial peer
Liar-killed census generation (fabricated cells die)	Yes	No	No	No	No	Every Nishi cell requires the implementing symbol on disk; neg-control + exceed-bounds + row-count checks fail the build -- 27 live comparisons carry it
Anti-drift claim coaching (only COACH-OK claims a rung)	Yes	No	No	No	No	U-axes, dated bars, claim-vs-proof ratio, freshness; it forced a real correction on the supervisor census 2026-07-10 (421 to 347 honest)
Multi-profile industry quality grading	Yes	Part	No	No	Part	Seven industry profiles (Elm, JPL, CERT...) with majority voting; Scorecard checks and HELM multi-metrics are partial peers
Safety-critical 8-axis grading that ABSTAINS	Yes	No	No	No	No	JPL / IEC61508 / ISO26262 / DO-178C axes; abstains via UNMEASURED instead of guessing -- abstention as a grading verdict is rare in the field
Honesty and overclaim grading of prose	Yes	No	No	No	No	Grades claims vs evidence verbs; HONEST FLAG: found orphaned 2026-07-09 (zero importers) -- wiring it to a consumer is a named rung
QA benchmark harness with real question sets	Yes	No	Best	Yes	Yes	251-question bar; SCORING is the named wall (2026-07-09); openai-evals registry harness is the bar
Census product-surface audit (U-axes enforcement)	Yes	No	No	No	No	Flagged 83 of 160 censuses missing U-axes (2026-07-09) -- the instrument that audits the instruments
Live-derived ecosystem grading (anti-staleness)	Yes	Part	No	No	No	Grades re-derive from evidence files on every read; Scorecard per-commit re-runs are the partial peer
Public measured-results publication	Yes	Part	Part	Part	Best	The /compare hub publishes 27 liar-killed comparisons as REST+MCP; HELM public leaderboards are the bar
LLM-as-judge evaluation	No	No	Best	No	Part	openai-evals model-graded evals are the bar; our reverse-AI-judge exists for IMAGES (nx_natstat, photoreal domain) but no text judge
Standardized benchmark suites (MMLU HumanEval class)	No	No	Part	Part	Best	HELM living-benchmark breadth is the bar; our benches are domain-organ KATs
Contamination and memorization detection	No	No	No	Part	Part	Critical for the coming TRAIN-our-own-model arc; EvalPlus and HELM carry partial answers
Supply-chain security scoring of repos	No	Best	No	No	No	Scorecard IS this; ties to the janitor census SBOM gap (mom 1864)
Statistical rigor (confidence intervals, paired tests)	No	No	No	No	Part	The Stabilizer-validated paired bench (2026-05-20, layout variance) set the precedent but no organ generalizes it
Human-preference evaluation	No	No	No	No	Part	Arena-style preference collection absent; HELM instruction-following slices partial
Longitudinal regression dashboards	No	Part	No	No	Part	Ratchet floors track direction but no eval-over-time surface exists
Cost and latency-aware evaluation	No	No	No	No	Part	HELM efficiency metrics are the bar; our zero-token loops make cost moot internally but externally-facing evals need it
Cheat-proof grading BY CONSTRUCTION	Best	No	No	No	No	Held-out scoring + property oracles + teeth-tested (the buggy-sort catch) + abstention verdicts: the grader CANNOT be gamed by the graded, in-substrate; the field's harnesses trust dataset answers and model judges
The evaluator grades ITSELF on the public plane	Best	No	No	No	No	The maturity rollup grades the ecosystem INCLUDING the grading fleet, the coach gates the coach's own censuses, and this very page is liar-killed by the organ it measures -- self-measurement as culture, not feature

Honest verdict. The Nishi referee is the culture the rest of the roster answers to: graders with TEETH. Capabilities are proven only by INDEPENDENT oracles (defining properties or cross-method agreement -- a deliberately buggy sort MUST be caught), censuses are liar-killed so fabricated cells die at generation, the coach gates every SOTA claim, the safety grader ABSTAINS as UNMEASURED rather than guess, and the whole apparatus publishes its own measured results on the public compare plane it uses to grade everything else. Against the evaluation field it is honestly narrow: no LLM-as-judge, no standardized benchmark suites, no contamination detection, no confidence intervals or paired statistics, no human-preference evaluation, no longitudinal regression dashboards -- HELM and openai-evals own breadth and statistics. The climb: score qabench (scoring is the named wall, bar 251), wire the orphaned honesty grader to a consumer, contamination checks for the coming trained models, then statistical rigor on every bench.

Generated by nx_swcompare_matrix (sovereign NishiLang organ) from knowledge/compare/referee.matrix — every Nishi cell verified against real organ source on disk. Zero JS, zero trackers.