nx_benchmark_harness.nx
buildroot/runtime/nx_benchmark_harness.nx
about
nx_benchmark_harness.nx -- head-to-head perf verdict primitive.
Substrate-level enforcer of the cardinal feedback-honest-perf-
verdict-no-aspirational-claims. Every NishiLang vs incumbent
benchmark emits a SEALED-ENUM verdict per axis: WIN / LOSE / TIE /
UNMEASURABLE. No "competitive with" weasel words. Every LOSE row
MUST be paired with a named-improvement string the caller fills in.
IDEA-PROVENANCE (per the language-design-research discipline:
learn from, never copy):
- JMH (OpenJDK) -- multi-warmup, multi-fork, GC-quiet
- Criterion.rs -- bootstrap CI, outlier detection (MAD)
- Google Benchmark -- statistical-rigor patterns
- Hyperfine -- multi-trial CLI command timing
Every idea was re-derived from the published papers + project READMEs;
no source code was incorporated. See genealogy_id for paper citations.
Six measurable axes per benchmark run:
1. THROUGHPUT ops/sec or items/sec
2. LATENCY_P50 median single-op time
3. LATENCY_P99 99th-percentile single-op time (tail latency)
4. MEMORY_PEAK peak bytes during run
5. DETERMINISM same-input-same-output across N trials (Q10)
6. PORTABILITY runs without modification across target archs (Q10)
Each axis emits a per-axis sealed-enum verdict:
NX_BENCH_AXIS_WIN challenger >= incumbent * 1.01 (1% delta floor)
NX_BENCH_AXIS_TIE within 1% either direction
NX_BENCH_AXIS_LOSE challenger < incumbent * 0.99
NX_BENCH_AXIS_UNMEASURABLE measurement not available this run
Composite verdict:
NX_BENCH_DECISIVE_WIN >= 4 of 6 axes WIN, zero LOSE
NX_BENCH_WIN majority WIN, fewer LOSE
NX_BENCH_TIE majority TIE or balanced WIN/LOSE
NX_BENCH_LOSE majority LOSE
NX_BENCH_DECISIVE_LOSE >= 4 of 6 axes LOSE
NX_BENCH_INCONCLUSIVE too many UNMEASURABLE to call
Why 1% floor: per cardinal feedback-one-percent-minimum-delta-substrate
dependencies 2 imports · 4 importers
imports: nx_syscalls.nxnx_tier.nx
imported by: nx_benchmark_harness_test.nxnx_daemon_bench.nxnx_daemon_bench_test.nxnx_polyglot_bench.nx
structs
| 92 | struct AxisMeasurement |
| 99 | struct AxisVerdict |
| 105 | struct BenchmarkReport |
consts
| 58 | const NX_BENCH_Q: nx_int = 1024 |
| 61 | const NX_BENCH_AXIS_UNMEASURABLE: nx_int = 0 |
| 62 | const NX_BENCH_AXIS_LOSE: nx_int = 1 |
| 63 | const NX_BENCH_AXIS_TIE: nx_int = 2 |
| 64 | const NX_BENCH_AXIS_WIN: nx_int = 3 |
| 65 | const NX_BENCH_N_AXIS_VERDICTS: nx_int = 4 |
| 68 | const NX_BENCH_AXIS_THROUGHPUT: nx_int = 0 |
| 69 | const NX_BENCH_AXIS_LATENCY_P50: nx_int = 1 |
| 70 | const NX_BENCH_AXIS_LATENCY_P99: nx_int = 2 |
| 71 | const NX_BENCH_AXIS_MEMORY_PEAK: nx_int = 3 |
| 72 | const NX_BENCH_AXIS_DETERMINISM: nx_int = 4 |
| 73 | const NX_BENCH_AXIS_PORTABILITY: nx_int = 5 |
| 74 | const NX_BENCH_N_AXES: nx_int = 6 |
| 77 | const NX_BENCH_INCONCLUSIVE: nx_int = 0 |
| 78 | const NX_BENCH_DECISIVE_LOSE: nx_int = 1 |
| 79 | const NX_BENCH_LOSE: nx_int = 2 |
| 80 | const NX_BENCH_TIE: nx_int = 3 |
| 81 | const NX_BENCH_WIN: nx_int = 4 |
| 82 | const NX_BENCH_DECISIVE_WIN: nx_int = 5 |
| 83 | const NX_BENCH_N_COMPOSITE_VERDICTS: nx_int = 6 |
| 86 | const NX_BENCH_DELTA_FLOOR_Q10: nx_int = 10 |
| 90 | const NX_BENCH_MIN_TRIALS: nx_int = 8 |
functions
| 129 | func _bench_axis_verdict(m: *AxisMeasurement, v: *AxisVerdict) -> nx_int called by 1: nx_benchmark_compute |
| 177 | func _bench_composite(report: *BenchmarkReport) -> nx_int called by 1: nx_benchmark_compute |
| 256 | func _bench_worst_axis(report: *BenchmarkReport) -> nx_int called by 1: nx_benchmark_compute |
| 306 | func nx_benchmark_compute(report: *BenchmarkReport) -> nx_int |
| 320 | func nx_benchmark_axis_verdict_is_valid(v: nx_int) -> nx_int called by 1: main |
| 326 | func nx_benchmark_composite_is_valid(c: nx_int) -> nx_int called by 1: main |
| 332 | func nx_benchmark_axis_index_is_valid(a: nx_int) -> nx_int |
| 342 | func nx_benchmark_report_is_cardinal_compliant(report: *BenchmarkReport) -> nx_int called by 1: main |