code wiki / _hdl_build / nx_referee_v2.nx

nx_referee_v2.nx

buildroot/runtime/_hdl_build/nx_referee_v2.nx

4288 B78 linesdepth 3pulls 3 transitivereach 6 importersview sourcekind librarytopic referee
docsdependenciesstructsconstsfunctions

about

nx_referee_v2.nx -- the Referee, upgraded by the cited research (knowledge/research/2026-06-04- objective-testing-judging.md). v1 had separation + ground-truth + determinism + decisiveness; v2 adds the mechanisms the literature proves are necessary, because the DOMINANT failure is a weak test set, not the scoring math: R2 TPR/TNR -- test quality on TWO axes (correctness AND coverage); a single metric hides failure R3 verdicts -- fixed codes (ACC/WA/TLE/RE) + all-ACC aggregation R5 validity -- TASK validity + OUTCOME validity + anti-gaming isolation (no oracle access, reset) R6 leakage -- a judge related to a competitor (same model / inheritance / same family) is INVALID R7 holdout -- the test set must be unseen (post-cutoff), or results are contaminated A head-to-head is RIGOROUS only if ALL gates pass. license_tier: ORIGINAL Cites: testlib, isolate, Wasik2018, CodeContests+ (TPR/TNR), Zhu/Jin NeurIPS2025 (validity+gaming), PreferenceLeakage ICLR2026, Kapoor&Narayanan 2023.

dependencies 2 imports · 4 importers

nx_referee.nx nx_syscalls.nx nx_referee_v2.nx nx_andelinwest_crew.nx nx_curate.nx nx_paper_panel.nx nx_referee_v2_test.nx

imports: nx_referee.nxnx_syscalls.nx

imported by: nx_andelinwest_crew.nxnx_curate.nxnx_paper_panel.nxnx_referee_v2_test.nx

structs

none

consts

30const RF2_ACC: i64 = 0
31const RF2_WA: i64 = 1 // wrong answer
32const RF2_TLE: i64 = 2 // time-limit exceeded
33const RF2_RE: i64 = 3 // runtime error

functions

18func rf2_permil(part: i64, whole: i64) -> i64 { if whole <= 0 { return 0 } return (part * 1000) / whole }
called by 2: rf2_tprrf2_tnr
20func rf2_tpr(correct_accepted: i64, correct_total: i64) -> i64 { return rf2_permil(correct_accepted, correct_total) }
called by 2: mainmain calls 1: rf2_permil
21func rf2_tnr(incorrect_caught: i64, incorrect_total: i64) -> i64 { return rf2_permil(incorrect_caught, incorrect_total) }
called by 2: mainmain calls 1: rf2_permil
23func rf2_testset_trustworthy(tpr: i64, tnr: i64, floor: i64) -> i64
called by 3: mainmainmain
35func rf2_aggregate(verdicts: *i64, n: i64) -> i64
called by 1: main
43func rf2_rigorous(task_valid: i64, outcome_valid: i64) -> i64
called by 1: main
50func rf2_gaming_proof(can_read_oracle: i64, can_write_oracle: i64, state_reset: i64) -> i64
called by 1: main
58func rf2_judge_independent(same_model: i64, inheritance: i64, same_family: i64) -> i64
called by 2: pp_grader_independentmain
66func rf2_holdout_clean(seen_in_training: i64) -> i64 { if seen_in_training == 0 { return 1 } return 0 }
called by 1: main
70func rf2_rigorous_headtohead(v1_valid: i64, testset_trustworthy: i64, rigorous: i64, gaming_proof: i64, judge_independent: i64, holdout_clean: i64) -> i64
called by 1: main