nx_calibration.nx
buildroot/runtime/nx_calibration.nx
about
nx_calibration.nx -- statistical calibration of predicted confidences.
When the substrate emits "predicted age 22 with 90% confidence"
(or any probability claim), DOES 90% confidence empirically
produce a hit 90% of the time? Without calibration, model
confidences are opaque numbers users learn to ignore. With
calibration, they become RELIABILITY-GUARANTEED measurements.
References:
Brier 1950 "Verification of Forecasts Expressed in Terms of
Probability" -- the Brier score
BS = (1/N) * sum_i (p_i - o_i)^2
where p_i in [0,1] is forecast probability and o_i in {0,1}.
Naeini/Cooper/Hauskrecht 2015 "Obtaining Well Calibrated
Probabilities Using Bayesian Binning" -- ECE / MCE.
DeGroot/Fienberg 1983 -- "well-calibrated" definition.
Closes calibration field in
nxc2/docs/TOPDOWN_BOTTOMUP_GAP_ANALYSIS.md. No existing image-
gen / LLM stack measures calibration of its OWN confidences in
a bits-up form; "confidence" floats around with no empirical
anchor.
genealogy_id: brier_1950_naeini_2015
lineage_id: substrate_calibration_v1
5W+H+GLP linkage: emits VERDICT records with what=VERDICT,
how=brier|ece|mce, performance=score_q10.
dependencies 3 imports · 0 importers
imports: nx_syscalls.nxnx_runtime.nxnx_tier.nx
imported by: nobody (leaf or entry point)
call flow from main pre-order; caps 40 nodes / depth 6 declared; ↻ = already shown
structs
| 76 | struct NxCalibrationResult |
consts
| 39 | const NX_MAGIC_1024: i64 = 1024 |
| 43 | const NX_CAL_Q10: nx_int = 1024 |
| 47 | const NX_CAL_MAX_SAMPLES: nx_int = 1048576 |
| 48 | const NX_CAL_MAX_BINS: nx_int = 100 |
| 49 | const NX_CAL_DEFAULT_BINS: nx_int = 10 |
| 55 | const NX_CAL_VERDICT_EXCELLENT: nx_int = 0 // ECE < 0.02 |
| 56 | const NX_CAL_VERDICT_GOOD: nx_int = 1 // ECE < 0.05 |
| 57 | const NX_CAL_VERDICT_FAIR: nx_int = 2 // ECE < 0.10 |
| 58 | const NX_CAL_VERDICT_POOR: nx_int = 3 // ECE < 0.20 |
| 59 | const NX_CAL_VERDICT_BROKEN: nx_int = 4 // ECE >= 0.20 |
| 61 | const NX_CAL_THR_EXCELLENT_Q10: nx_int = 20 // 0.02 * 1024 |
| 62 | const NX_CAL_THR_GOOD_Q10: nx_int = 51 // 0.05 * 1024 |
| 63 | const NX_CAL_THR_FAIR_Q10: nx_int = 102 // 0.10 * 1024 |
| 64 | const NX_CAL_THR_POOR_Q10: nx_int = 204 // 0.20 * 1024 |
| 88 | const NX_CAL_RESULT_BYTES: nx_size = 72 |
functions
| 66 | func nx_cal_verdict_from_ece(ece_q10: nx_int) -> nx_int |
| 100 | func nx_cal_brier_q10(p_q10: *nx_int, o: *nx_int, n_samples: nx_int) -> nx_int |
| 121 | func _cal_bin_idx(p_q10: nx_int, n_bins: nx_int) -> nx_int |
| 136 | func nx_cal_bin( |
| 184 | func nx_cal_ece_q10( called by 1: nx_cal_report |
| 208 | func nx_cal_mce_q10( called by 1: nx_cal_report |
| 227 | func nx_cal_report( called by 1: main calls 6: sys_mmapnx_cal_binnx_cal_brier_q10nx_cal_ece_q10nx_cal_mce_q10nx_cal_verdict_from_ece |
| 270 | func main() -> nx_int |