nx_debtcluster.nx source

1// nx_debtcluster.nx -- THE LEDGER STOPS BEING A LIST AND BECOMES A RECOMMENDATION. 2// license_tier: ORIGINAL No hw writes (Rule 26). expect_exit: 0 3// 4// ─── THE DEFECT THIS EXISTS TO KILL ──────────────────────────────────────────────────────────────────── 5// MEASURED 2026-07-31: the debt plane holds 2068 rows, 905 open, 281 open at sev>=8, and it is FLAT. Every 6// seat reads it top-down by severity and picks a row. But the rows are not independent -- in the first FORTY 7// sev>=8 rows alone there are at least five families, each one incident wearing many row numbers: 8// mgmt_api availability seq698 crash-loop · seq710 hang · the live sev-9 :18098 outage 9// resource exhaustion seq883 23.6GiB alloc-without-free · seq774 20,440 respawns · seq783 · seq727 · seq975 10// dup-source seq207 · seq701 · seq945 · seq965 · seq974 (965/974 report GREEN divergent=0 11// WHILE THE PARENTS STAY OPEN) 12// law-warden magic nums seq274 L001 (_hdl_build, 2348) · seq315 L006 (core, 3476) -- same rule, same 13// detector, two corpora, filed as two unrelated problems 14// gx-body seq804·805·808·819·820·824·826·828·900 = NINE rows, ONE skull-architecture arc 15// A seat that fixes the top row by severity fixes ONE MEMBER of a family and leaves the family. The ledger 16// is sorted by how loud a row is, never by how much work a single act would close. 17// ★LAW: A LEDGER SORTED BY SEVERITY ANSWERS "WHAT SCREAMS LOUDEST", NEVER "WHAT SHOULD I BUILD". 18// 19// ─── WHAT SOTA SAYS (researched 2026-07-31, and what we deliberately DID NOT copy) ───────────────────── 20// 1. NORMALIZE BEFORE YOU EMBED. LLM-augmented ticket clustering (FLAIRS 2026, ABCD) beats raw-text 21// embedding on silhouette AND human-rated coherence purely by extracting a canonical statement first. 22// GPTrace (ICSE'26, arXiv 2512.01609) does the same for crash dedup: normalize -> embed -> HDBSCAN, 23// scored on silhouette + DBCV + P/R/F1, beating CrashWalk and Igor. 24// 2. DENSITY-BASED, NOT PARTITIONAL. HDBSCAN is chosen in that literature precisely because it DECLINES to 25// cluster outliers. A debt ledger is mostly singletons; a partitional method (k-means, plain 26// agglomerative) has nowhere to put them and pollutes every cluster with noise. We implement the 27// principle, not the library: a core-density test with an EXPLICIT noise class that is always reported. 28// 3. TOPOLOGY BEATS TEXT SIMILARITY FOR CAUSALITY. TopoEvo (arXiv 2605.15611) is unambiguous -- correlation 29// is constrained by the ACTUAL dependency graph, and topology-agnostic baselines manufacture spurious 30// links by treating all relations as equivalent. TWO ROWS BOTH SAYING "LEAK" IN UNCONNECTED ORGANS ARE 31// TWO PROBLEMS. This is the ic_ident_gate below, and it is the single most important line in the organ. 32// 4. RANK BY RISK-REDUCTION OVER EFFORT, and cap the recommendation list at 5-8 (postmortem practice). 33// 5. ★DO NOT BUILD THE AGENTIC RCA TIER. ITBench-AA (Artificial Analysis + IBM Research, launched 34// 2026-05-27, 59 live Kubernetes SRE tasks): EVERY frontier model scores below 50% -- Claude Opus 4.7 35// 47%, GPT-5.5 46%; IBM's own ITBench reports best-agent 13.8% SRE / 25.2% CISO / 0% FinOps. Meanwhile 36// the plain correlate-and-rank layer delivers a reliable 80-85% noise reduction in production. So we 37// build the deterministic layer that demonstrably works and refuse the one that demonstrably does not. 38// 6. GRADE ORDINALLY. Arvo's AI Investigation Capability Ladder (May 2026, upd. July 2026): L0 manual · 39// L1 correlation · L2 summarized timeline · L3 single-shot diagnosis · L4 agentic · L5 closed-loop. 40// THIS ORGAN IS L2, EVIDENCE-CAPPED, AND SAYS SO IN ITS OWN OUTPUT. Claiming L4 would be the banked 41// coverage-gaming defect verbatim. 42// 43// ─── WHY LEXICAL AND NOT THE DENSE MODEL (a deliberate choice, not a shortcut) ────────────────────────── 44// nx_recall_dense runs a real 41MB PPMI model (vocab 102,318 / 2,558,014 triples, integer late-interaction, 45// deterministic). It is available and it is the obvious next rung. It is NOT rung one, because this corpus 46// is identifier-dense: the signal that says two rows are the same problem is the literal token 47// `nx_mgmt_api`, not a distributional neighbourhood. A general-text embedder DILUTES an exact identifier 48// match into a similarity score; IDF-weighted exact matching preserves it. This is the same reason BM25 49// stays competitive on technical corpora and the reason hybrid RRF fusion exists at all. The dense axis is 50// named in the output as a declared GAP (nx_recall_fuse is gate 7/7, nDCG 494->659, not yet in serve) so 51// the tier can never be read as complete. 52// 53// ─── NON-VACUITY: THIS INSTRUMENT CAN FAIL, AND IT PROVES IT ON ITSELF ───────────────────────────────── 54// `selftest` runs the REAL pipeline over planted fixtures. The decisive tooth is T2, a NEGATIVE CONTROL: 55// two rows written to be lexically near-identical ("memory grows unbounded on every call, never freed") 56// but sited in DISJOINT organs. A text-only clusterer merges them -- that is the spurious correlation 57// TopoEvo names. This organ MUST keep them apart, and selftest FAILS if it does not. A clusterer that 58// cannot refuse a merge is not measuring anything. 59// Further teeth: zero rows loaded -> INSTRUMENT-BLIND exit 2 (never "no clusters found") · plane short-read 60// -> coverage_complete=0 and verdict PARTIAL · noise_rows and the row denominator are ALWAYS printed 61// (A SAMPLE FILED WITHOUT ITS DENOMINATOR IS A LIE) · the headline is the MIN emitted priority, never the 62// mean (HEADLINE=MIN law) · effort is reported as -1 UNMEASURED and is NEVER folded into the score, because 63// the plane records file-time and not close-time so effort is genuinely not derivable here (-1=UNMEASURED 64// never the min, per nx_capaxes). 65// 66// nx_debtcluster run [prefix] [sim_join_permil] cluster the live plane -> ranked JSON 67// nx_debtcluster selftest planted fixtures incl. the negative control 68// ONE import only -- nx_store_seed_lib.nx already pulls in nx_seg_store.nx and nx_syscalls.nx, so naming 69// them again is redundant. That redundancy is hygiene, NOT the cause of anything: an earlier note here 70// claimed siblings naming all three would fail to build, and that claim was REFUTED by measurement -- 71// dropping the extra imports moved the duplicate-definition error's line numbers by exactly the number of 72// comment lines added, which proved the duplicate sts_lock was INTRINSIC to nx_store_seed_lib.nx (defined 73// at both line 40 and line 377) rather than an expansion artifact. Fixed at the library; see debt 1785525535. 74import "nx_store_seed_lib.nx" 75import "nx_ppmi_lib.nx" 76const IC_MAGIC_1784947125: i64 = 1784947125 77const IC_MAGIC_1784947134: i64 = 1784947134 78const IC_MAGIC_1784947786: i64 = 1784947786 79const IC_MAGIC_1784951793: i64 = 1784951793 80const IC_MAGIC_1784952157: i64 = 1784952157 81const IC_MAGIC_1784954507: i64 = 1784954507 82const IC_MAGIC_1784955224: i64 = 1784955224 83const IC_MAGIC_1784955875: i64 = 1784955875 84const IC_MAGIC_1784956205: i64 = 1784956205 85const IC_MAGIC_1784995126: i64 = 1784995126 86const IC_MAGIC_1784995721: i64 = 1784995721 87const IC_MAGIC_1784995947: i64 = 1784995947 88const IC_MAGIC_1784996406: i64 = 1784996406 89const IC_MAGIC_1784996558: i64 = 1784996558 90const IC_MAGIC_1785000800: i64 = 1785000800 91const IC_MAGIC_1784925304: i64 = 1784925304 92const IC_MAGIC_1784927702: i64 = 1784927702 93const IC_MAGIC_1784958766: i64 = 1784958766 94const IC_MAGIC_1784978119: i64 = 1784978119 95const IC_MAGIC_1784999221: i64 = 1784999221 96const IC_MAGIC_1785001189: i64 = 1785001189 97const IC_MAGIC_1785002038: i64 = 1785002038 98const IC_MAGIC_1024: i64 = 1024 99const IC_MAGIC_1784856230: i64 = 1784856230 100const IC_MAGIC_1784857155: i64 = 1784857155 101const IC_MAGIC_1784480242: i64 = 1784480242 102const IC_MAGIC_1784856346: i64 = 1784856346 103const IC_MAGIC_1785030820: i64 = 1785030820 104const IC_MAGIC_1785031111: i64 = 1785031111 105const IC_MAGIC_1784995429: i64 = 1784995429 106const IC_MAGIC_1784995511: i64 = 1784995511 107const IC_MAGIC_1784996221: i64 = 1784996221 108const IC_MAGIC_1785002845: i64 = 1785002845 109const IC_MAGIC_1785002909: i64 = 1785002909 110const IC_MAGIC_1785003511: i64 = 1785003511 111const IC_MAGIC_1785003779: i64 = 1785003779 112const IC_MAGIC_1785003935: i64 = 1785003935 113const IC_MAGIC_1785027945: i64 = 1785027945 114const IC_MAGIC_1784661149: i64 = 1784661149 115const IC_MAGIC_1784661346: i64 = 1784661346 116const IC_MAGIC_1784664001: i64 = 1784664001 117const IC_MAGIC_1785027242: i64 = 1785027242 118const IC_MAGIC_1784908046: i64 = 1784908046 119const IC_MAGIC_2048: i64 = 2048 120const IC_MAGIC_16384: i64 = 16384 121 122// Plane read cap. MIRRORS nx_debt.nx's DB_CAP exactly. Reading the same plane with a smaller buffer is how 123// an instrument silently sees a prefix of the truth (the nx_debt_view 512KB blindness, 734 rows of 1835). 124// 2026-08-06: RAISED 4194304 -> 33554432 in lockstep with nx_debt.nx DB_CAP and nx_debtlive.nx DL_CAP. 125// The plane CROSSED 4 MiB (4,217,666B / 3,326 rows) and every capped reader went blind to the NEWEST rows 126// first. ***THE MIRROR IS THE DEFECT***: three organs hand-copy this number, each with a comment swearing it 127// mirrors the others -- which is exactly when they stop. The 512KB precedent cited above is the SAME class, 128// so the codebase documented this failure and then reproduced it. Real fix: ONE exported cap in the plane lib. 129const IC_CAP: i64 = 33554432 130// Row headroom. Plane is ~2068 and grows ~500/wk (nx_debtlive's own measurement), so 8192 is ~12 weeks. 131// Rows beyond this are COUNTED into rows_capped and reported, never silently dropped. 132const IC_MAXROWS: i64 = 8192 133const IC_MAXTOK: i64 = 48 // tokens retained per row; long rows are informative in the head 134const IC_DICT: i64 = 65536 // open-addressed dict slots; ~8x expected distinct tokens keeps probes short 135const IC_POSTCAP: i64 = 393216 // IC_MAXROWS * IC_MAXTOK 136const IC_TOKMIN: i64 = 3 // 1-2 char tokens carry no discriminative signal 137const IC_TOKMAX: i64 = 64 138const IC_HMOD: i64 = 1000003 // keeps the rolling hash bounded and positive (no overflow reliance) 139// A token appearing in more than this share of rows has near-zero IDF and quadratically inflates the 140// postings scan. Skipped tokens are COUNTED and reported (tokens_df_skipped) -- this is why no stopword 141// list exists in this organ: stopwords are DATA-DRIVEN here, not a hardcoded table (rule 11). 142const IC_DF_PERMIL: i64 = 125 143// Default link threshold in permil of IDF-weighted overlap. Overridable as argv[3] and ALWAYS echoed in the 144// envelope, and the pair-count bands below it are reported so the choice is auditable rather than asserted. 145const IC_SIM_JOIN: i64 = 380 146const IC_BAND_LO: i64 = 250 147const IC_BAND_HI: i64 = 500 148// Density: a row must have at least this many GATE-QUALIFYING neighbours (over the similarity threshold AND 149// sharing an identifier) to be a CORE row. Rows that are not core cannot seed a cluster -- this is the 150// HDBSCAN property that makes a noise class possible. 151// ★WHY 1 AND NOT THE LITERATURE'S 2+: minPts must match the DOMAIN'S cluster-size distribution, and a debt 152// ledger's families START AT TWO -- {seq698 mgmt crash-loop, seq710 mgmt hang} is a complete, actionable 153// family. minPts=2 requires >=2 neighbours and therefore >=3 members, so it CANNOT form a pair and silently 154// discards every two-row family as noise. That is not a stricter instrument, it is a blind one: measured 155// here, minPts=2 returned families=0 / noise=5 on a fixture containing two obvious pairs. 156// The noise class does NOT depend on this number -- it is the IDENTIFIER GATE that refuses merges, and the 157// negative control (T2) is what proves the gate still bites at minPts=1. Loosening this without T2 surviving 158// would be tuning until green, which is why T2 is a hard failure and not a warning. 159const IC_MINPTS: i64 = 1 160// ─── ANCHOR CLUSTERING (replaced transitive similarity 2026-07-31, ON MEASUREMENT) ───────────────────── 161// The first implementation linked rows by IDF-weighted text similarity and took the transitive closure. 162// It CHAINED: at sim=250 the top family had 44 members spanning FIFTEEN scopes and the second had 20 across 163// EIGHTEEN, because A~B and B~C merges A,B,C even when A and C share nothing at all. Raising the threshold 164// to stop that just traded the chaining for silence (families 231->104, noise 1373->1747). There is no 165// threshold that fixes single linkage, because the defect is the TRANSITIVITY, not the cutoff. 166// ★THE REPLACEMENT: a family is the set of rows that literally contain ONE shared identifier -- the anchor. 167// Chaining is then impossible BY CONSTRUCTION (membership is a property of each row on its own, never of a 168// path through other rows), and the label stops being guesswork because THE ANCHOR IS THE NAME. 169// An anchor present in more than this share of rows is a THEME, not a family ("buildroot" is not a work 170// item); excluded anchors are COUNTED and reported so the exclusion is never silent. 171const IC_ANCHOR_DF_PERMIL: i64 = 80 172const IC_MINFAM: i64 = 2 173// Two families merge only when this share of the SMALLER family's rows each independently carry the other 174// family's anchor. 500 = a strict majority. This is the guard that separates evidence-based merging from 175// the transitive closure that chained in v1: one bridging row cannot reach a majority, ever. 176// DENSE (SEMANTIC) ARM. The merge pass proved a second LEXICAL signal only adds noise; the remaining 177// recall has to come from a signal that understands WORDS THAT DIFFER. ppl_maxsim_idf is integer 178// late-interaction maxsim over the 43MB PPMI model -- deterministic, no float. 179// SCOPE IS DELIBERATELY NARROW: it only places rows the anchor pass left UNASSIGNED. It never merges, 180// splits or re-labels an existing family, so recall can only rise and the precision risk is bounded to 181// exactly the rows it places. That is the discipline the merge pass lacked. 182// DENSE ARM DISABLED 2026-08-01 -- BUILT, LOADED, MEASURED, REFUTED. Kept as the record. 183// It worked mechanically (dense_model_loaded=1, dense_placed=248 of ~248 noise rows) and was 184// STRICTLY HARMFUL: B-cubed precision 900 -> 828, recall 628 -> 628 (NOT ONE POINT), F1 739 -> 714, 185// and the GATE WENT RED -- T4/T5 failed because it swallowed the deliberately-unrelated fixture row, 186// destroying the noise class outright. At threshold 700 it placed essentially EVERY noise row. 187// THE DIAGNOSIS THAT MATTERS, AND IT KILLS MY SCOPING ASSUMPTION: recall did not move AT ALL, which 188// proves the missing recall is NOT in unassigned rows. Every gold row was already assigned -- the 189// loss is gold-family members SPLIT ACROSS DIFFERENT families. Placing noise could never have fixed 190// it, so this arm was aimed at the wrong defect from the start. 191// LAW: BEFORE ADDING A SIGNAL, PROVE WHERE THE LOSS ACTUALLY IS -- I ASSUMED IT WAS THE NOISE PILE 192// AND SPENT A WHOLE RUNG THERE. The next attempt must target SPLIT families, not unassigned rows. 193// Does the ORGAN tier drive CLAIMING, or only labelling? Organ-first raised emitted_organ_anchored 194// 4->7 but recall sat at 628 and 4 of 5 multi-row gold families stayed split. This flag isolates the 195// grouping half so the two jobs can be measured apart instead of assumed together. 196const IC_OVERLAP: i64 = 1 197const IC_ORGAN_CLAIM: i64 = 0 198const IC_SCOPE_FIRST: i64 = 0 199const IC_DENSE_ENABLED: i64 = 0 200const IC_DENSE_MIN: i64 = 700 201const IC_DENSE_TOPF: i64 = 64 202const IC_MERGE_PERMIL: i64 = 500 203// MERGE PASS DISABLED 2026-07-31 -- BUILT, MEASURED, REFUTED, KEPT AS THE RECORD SO NOBODY REBUILDS IT. 204// Hypothesis: anchors are one blocking strategy, so merging families on majority co-occurrence should 205// recover the recall the ruler measured as missing (P840/R635/F1 723). 206// RESULT: B-cubed came back BIT-IDENTICAL -- 840/635/723, not one point of gain -- while the visible 207// output degraded badly: rank 1 became 'nx_connect_appd' with 161 members swallowing the stale-mirror 208// family, and ranks 2-8 turned into junk phrase anchors (MID-SESSION, OO-CONSOLIDATION, API-pure, 209// build-deploy). Families ballooned 152->161 and 87->115 while naming nothing. 210// TWO LESSONS, both banked: 211// 1. MAJORITY CO-OCCURRENCE IS NOT EVIDENCE OF SAME-SUBJECT. Rows about one organ naturally quote the 212// same handful of ecosystem nouns, so the majority test passes for families that merely share a 213// vocabulary. The anchor was already the right discriminator; a second lexical signal adds noise. 214// 2. THE RULER WAS FLAT WHILE QUALITY VISIBLY FELL. A 20-row gold set over 5 families cannot see a 215// regression that is obvious by eye -- so the gold set is UNDER-POWERED for this class, and that is a 216// finding about the instrument, not a reason to trust the change. 217// LAW: A RULER THAT CANNOT MOVE WHEN THE OUTPUT VISIBLY WORSENS IS NOT YET A RULER FOR THAT DEFECT. 218const IC_MERGE_ENABLED: i64 = 0 // two rows describing one thing is already a family worth naming 219const IC_MAXFAM: i64 = 8192 // <= IC_MAXROWS so the ranking array can always index every family 220// Ordinal escalation, deliberately larger than any achievable max_sev x open product on this plane, so the 221// urgent tier is a TIER and not a thumb on the scale that a big enough pile could still outweigh. 222const IC_URGENT_BOOST: i64 = 1000000 223// A family holding a sev>=9 row is an OUTAGE and preempts on ordinal grounds, exactly as an SRE rota does. 224// Without this, volume dominates severity in any product form and a live sev-9 sinks below a big sev-5 pile. 225const IC_URGENT_SEV: i64 = 9 226// Postmortem practice caps action items at 5-8. Emitting more is how a recommendation becomes a list again. 227const IC_EMIT: i64 = 8 228const IC_LABEL_TOKENS: i64 = 3 229const IC_IDS_PER_CLUSTER: i64 = 12 230const IC_OUTCAP: i64 = 262144 231const IC_LBUF: i64 = 512 // one cluster label ("tok|tok|tok"), reused per emitted family 232const IC_WRMODE: i64 = 420 // 0644 233const IC_TAB: i64 = 9 234const IC_NL: i64 = 10 235const IC_DAY: i64 = 86400 236const IC_MODE_RUN: i64 = 0 237const IC_MODE_SELFTEST: i64 = 1 238const IC_MODE_PAGE: i64 = 2 239const IC_MODE_EVAL: i64 = 3 240const IC_GTMAX: i64 = 64 241 242func ic_len(s: *u8) -> i64 { var i: i64 = 0; while s[i] != (0 as u8) { i = i + 1 } return i } 243func ic_puts(s: *u8) -> i64 { sys_write(1, s, ic_len(s)); return 0 } 244func ic_cat(d: *u8, o: i64, s: *u8) -> i64 { var i: i64 = 0; var o2: i64 = o; while s[i] != (0 as u8) { d[o2] = s[i]; o2 = o2 + 1; i = i + 1 } return o2 } 245func ic_catn(d: *u8, o: i64, v: i64) -> i64 { 246 let t: *u8 = sys_mmap(32) 247 var m: i64 = v 248 var neg: i64 = 0 249 if m < 0 { neg = 1; m = 0 - m } 250 var k: i64 = 0 251 if m == 0 { t[0] = 48 as u8; k = 1 } 252 while m > 0 { t[k] = ((m - (m / 10) * 10) + 48) as u8; m = m / 10; k = k + 1 } 253 var o2: i64 = o 254 if neg == 1 { d[o2] = 45 as u8; o2 = o2 + 1 } 255 var i: i64 = 0 256 while i < k { d[o2] = t[k - 1 - i]; o2 = o2 + 1; i = i + 1 } 257 return o2 258} 259func ic_catsl(d: *u8, o: i64, src: *u8, off: i64, len: i64) -> i64 { 260 var i: i64 = 0 261 var o2: i64 = o 262 while i < len { d[o2] = src[off + i]; o2 = o2 + 1; i = i + 1 } 263 return o2 264} 265func ic_atoi(s: *u8) -> i64 { 266 var v: i64 = 0 267 var i: i64 = 0 268 var go: i64 = 1 269 while go == 1 { 270 let c: i64 = s[i] as i64 271 if c < 48 { go = 0 } else { if c > 57 { go = 0 } else { v = v * 10 + (c - 48); i = i + 1 } } 272 } 273 return v 274} 275// digits of a byte slice -> i64 (the epoch field). Stops at the first non-digit. 276func ic_slice_num(q: *u8, off: i64, len: i64) -> i64 { 277 var v: i64 = 0 278 var i: i64 = 0 279 var go: i64 = 1 280 while go == 1 { 281 if i >= len { go = 0 } else { 282 let c: i64 = q[off + i] as i64 283 if c < 48 { go = 0 } else { if c > 57 { go = 0 } else { v = v * 10 + (c - 48); i = i + 1 } } 284 } 285 } 286 return v 287} 288 289// log2(x) scaled by 1024, integer-only: highest set bit + linear interpolation of the mantissa. 290// Max error ~0.086 bits, monotonic, deterministic, no float anywhere. Used for IDF so that a token in 1 row 291// and a token in 3 rows are actually distinguishable (a bare bit-position would collapse them). 292func ic_log2q(x: i64) -> i64 { 293 if x <= 1 { return 0 } 294 var b: i64 = 0 295 var v: i64 = x 296 while v > 1 { v = v / 2; b = b + 1 } 297 var p: i64 = 1 298 var i: i64 = 0 299 while i < b { p = p * 2; i = i + 1 } 300 let frac: i64 = ((x - p) * IC_MAGIC_1024) / p 301 return b * IC_MAGIC_1024 + frac 302} 303 304// ─── TOKEN CLASSIFICATION ────────────────────────────────────────────────────────────────────────────── 305// An IDENTIFIER token is one that names a THING IN THE SYSTEM rather than describing a symptom: it carries 306// an underscore or a hyphen (`nx_mgmt_api`, `gx-body`, `seg-store`). This is derived from the ecosystem's 307// own naming conventions, not invented. Identifier tokens are what the topology gate keys on -- symptom 308// words ("leak", "hang", "timeout") recur everywhere and must never be sufficient to merge two rows. 309// An IDENTIFIER must carry a separator AND AT LEAST ONE LETTER. The letter rule is not cosmetic: without 310// it '2026-07-31' is an identifier, and dates become the highest-volume anchors on the plane. MEASURED -- 311// the first anchor run ranked '2026-07-31' FIRST with 199 members across 163 scopes, plus '2026-07-25' and 312// '2026-07-23' at ranks 3 and 5. A date is the one thing every row filed that day shares, which makes it a 313// perfect anchor and a worthless family. 314// LAW: A TOKEN THAT EVERY ROW SHARES FOR A REASON UNRELATED TO ITS SUBJECT IS THE BEST-SCORING ANCHOR AND 315// THE WORST POSSIBLE ONE. 316func ic_is_ident(q: *u8, off: i64, len: i64) -> i64 { 317 var sep: i64 = 0 318 var alpha: i64 = 0 319 var i: i64 = 0 320 while i < len { 321 let c: i64 = q[off + i] as i64 322 if c == 95 { sep = 1 } 323 if c == 45 { sep = 1 } 324 if c >= 97 { if c <= 122 { alpha = 1 } } 325 if c >= 65 { if c <= 90 { alpha = 1 } } 326 i = i + 1 327 } 328 if sep == 1 { if alpha == 1 { return 1 } } 329 return 0 330} 331// IS THIS TOKEN THE NAME OF SOMETHING THAT ACTUALLY EXISTS? 332// The junk anchors all shared one property: MID-SESSION, API-pure, byte-identical and build-deploy are 333// phrases lifted from row prose, and NOTHING IN THE SYSTEM IS CALLED THAT. nx_mgmt_api.nx is a real file. 334// So the test is not a heuristic about shape -- it is a lookup against the tree. This is the topology 335// grounding TopoEvo actually prescribes (constrain correlation by the real system), where the earlier 336// separator-and-a-letter rule was only a shape approximation of it. 337// Cost is ~2 open() calls per candidate anchor (~1900), which is nothing. 338func ic_file_exists(path: *u8) -> i64 { 339 let fd: i64 = sys_openat_rd(path) 340 if fd < 0 { return 0 } 341 sys_close(fd) 342 return 1 343} 344func ic_try(pre: *u8, q: *u8, off: i64, len: i64, suf: *u8, scratch: *u8) -> i64 { 345 var o: i64 = ic_cat(scratch, 0, pre) 346 var i: i64 = 0 347 while i < len { scratch[o] = q[off + i]; o = o + 1; i = i + 1 } 348 o = ic_cat(scratch, o, suf) 349 scratch[o] = 0 as u8 350 return ic_file_exists(scratch) 351} 352// WIDENED 2026-08-01 ON MEASUREMENT. Organ-first raised emitted_organ_anchored 4/8 -> 6/8, but the 117-row 353// stale-mirror family KEPT the phrase anchor 'auto-filed' -- because its actual subject, nx_stale_sweep, 354// is a .cron.sh and NOT a .nx organ. The lookup correctly answered 'not a source file' and I had wrongly 355// equated that with 'not a real thing'. The system's real subjects are .nx SOURCES, .elf BINARIES and .sh 356// SCRIPTS alike. 357// LAW: A NAME TEST THAT KNOWS ONLY ONE FILE TYPE MISSES EVERY SUBJECT THAT LIVES IN ANOTHER. 358func ic_is_organ(q: *u8, off: i64, len: i64, scratch: *u8) -> i64 { 359 if ic_try("buildroot/runtime/" as *u8, q, off, len, ".nx" as *u8, scratch) == 1 { return 1 } 360 if ic_try("buildroot/runtime/_hdl_build/" as *u8, q, off, len, ".nx" as *u8, scratch) == 1 { return 1 } 361 if ic_try("" as *u8, q, off, len, ".elf" as *u8, scratch) == 1 { return 1 } 362 if ic_try("" as *u8, q, off, len, ".sh" as *u8, scratch) == 1 { return 1 } 363 if ic_try("" as *u8, q, off, len, ".cron.sh" as *u8, scratch) == 1 { return 1 } 364 return 0 365} 366 367// Pure-digit tokens are epochs, counts and dates. They are high-cardinality noise that would link unrelated 368// rows filed in the same second, so they are dropped at the tokenizer. 369func ic_all_digits(q: *u8, off: i64, len: i64) -> i64 { 370 var i: i64 = 0 371 while i < len { 372 let c: i64 = q[off + i] as i64 373 if c < 48 { return 0 } 374 if c > 57 { return 0 } 375 i = i + 1 376 } 377 return 1 378} 379func ic_tokchar(c: i64) -> i64 { 380 if c >= 97 { if c <= 122 { return 1 } } 381 if c >= 65 { if c <= 90 { return 1 } } 382 if c >= 48 { if c <= 57 { return 1 } } 383 if c == 95 { return 1 } 384 if c == 45 { return 1 } 385 return 0 386} 387func ic_alnum(c: i64) -> i64 { 388 if c >= 97 { if c <= 122 { return 1 } } 389 if c >= 65 { if c <= 90 { return 1 } } 390 if c >= 48 { if c <= 57 { return 1 } } 391 return 0 392} 393// case-insensitive byte compare of two slices in the same buffer 394func ic_slice_eq(q: *u8, a: i64, al: i64, b: i64, bl: i64) -> i64 { 395 if al != bl { return 0 } 396 var i: i64 = 0 397 while i < al { 398 var x: i64 = q[a + i] as i64 399 var y: i64 = q[b + i] as i64 400 if x >= 65 { if x <= 90 { x = x + 32 } } 401 if y >= 65 { if y <= 90 { y = y + 32 } } 402 if x != y { return 0 } 403 i = i + 1 404 } 405 return 1 406} 407// rolling hash, case-folded, kept inside IC_HMOD so it is always positive and never relies on overflow 408func ic_hash(q: *u8, off: i64, len: i64) -> i64 { 409 var h: i64 = 0 410 var i: i64 = 0 411 while i < len { 412 var c: i64 = q[off + i] as i64 413 if c >= 65 { if c <= 90 { c = c + 32 } } 414 h = h * 131 + c 415 h = h - (h / IC_HMOD) * IC_HMOD 416 i = i + 1 417 } 418 return h 419} 420// Intern a token. dh: [0]=hash [1]=off [2]=len [3]=df [4]=idf [5]=ident, stride 6. Returns slot, or -1 full. 421func ic_intern(q: *u8, off: i64, len: i64, dh: *i64) -> i64 { 422 let h: i64 = ic_hash(q, off, len) 423 var s: i64 = h - (h / IC_DICT) * IC_DICT 424 var probes: i64 = 0 425 var res: i64 = 0 - 1 426 var go: i64 = 1 427 while go == 1 { 428 if probes >= IC_DICT { go = 0 } else { 429 let base: i64 = s * 6 430 if dh[base + 2] == 0 { 431 dh[base + 0] = h 432 dh[base + 1] = off 433 dh[base + 2] = len 434 dh[base + 3] = 0 435 dh[base + 4] = 0 436 dh[base + 5] = ic_is_ident(q, off, len) 437 res = s 438 go = 0 439 } else { 440 if dh[base + 0] == h { 441 if ic_slice_eq(q, dh[base + 1], dh[base + 2], off, len) == 1 { res = s; go = 0 } 442 } 443 if go == 1 { 444 s = s + 1 445 if s >= IC_DICT { s = 0 } 446 probes = probes + 1 447 } 448 } 449 } 450 } 451 return res 452} 453 454// ─── UNION-FIND ──────────────────────────────────────────────────────────────────────────────────────── 455// THE RULER: HAND-VERIFIED GROUND TRUTH. 456// Every tuning decision in this organ up to 2026-07-31 was made BY INSPECTION -- I read the output, judged 457// the junk, changed a parameter. That is taste, not measurement, and it cannot tell a real improvement from 458// one that merely moved the junk somewhere I did not look. This table is the correction. 459// Each row below is a debt row I PERSONALLY READ IN FULL and assigned to a family by its subject. Keyed on 460// the EPOCH ID, never the row index: indices are positional and shift as the plane grows, so an index-keyed 461// ruler silently re-labels itself between runs and reports drift as quality change. 462// It is a SMALL SAMPLE (5 families / 20 rows of ~2259) and the output ALWAYS prints gt_found against 463// gt_declared so nobody can mistake this floor for a corpus-wide score. 464// Scored with B-CUBED, chosen over ARI/NMI on purpose: family sizes run 152 rows down to 2, and 465// pair-counting metrics are dominated by the largest cluster. B-cubed is per-ELEMENT, so a 2-row family 466// swallowed by a 152-row one costs exactly what it should. 467// families: 1=mgmt-api availability 2=dup-source 3=gx-body 4=untrusted-input security 5=memory leak 468func ic_gt_load(gtid: *i64, gtfam: *i64) -> i64 { 469 var n: i64 = 0 470 gtid[n] = IC_MAGIC_1784856230; gtfam[n] = 1; n = n + 1 471 gtid[n] = IC_MAGIC_1784857155; gtfam[n] = 1; n = n + 1 472 gtid[n] = IC_MAGIC_1784480242; gtfam[n] = 2; n = n + 1 473 gtid[n] = IC_MAGIC_1784856346; gtfam[n] = 2; n = n + 1 474 gtid[n] = IC_MAGIC_1785030820; gtfam[n] = 2; n = n + 1 475 gtid[n] = IC_MAGIC_1785031111; gtfam[n] = 2; n = n + 1 476 gtid[n] = IC_MAGIC_1784995429; gtfam[n] = 3; n = n + 1 477 gtid[n] = IC_MAGIC_1784995511; gtfam[n] = 3; n = n + 1 478 gtid[n] = IC_MAGIC_1784996221; gtfam[n] = 3; n = n + 1 479 gtid[n] = IC_MAGIC_1785002845; gtfam[n] = 3; n = n + 1 480 gtid[n] = IC_MAGIC_1785002909; gtfam[n] = 3; n = n + 1 481 gtid[n] = IC_MAGIC_1785003511; gtfam[n] = 3; n = n + 1 482 gtid[n] = IC_MAGIC_1785003779; gtfam[n] = 3; n = n + 1 483 gtid[n] = IC_MAGIC_1785003935; gtfam[n] = 3; n = n + 1 484 gtid[n] = IC_MAGIC_1785027945; gtfam[n] = 3; n = n + 1 485 gtid[n] = IC_MAGIC_1784661149; gtfam[n] = 4; n = n + 1 486 gtid[n] = IC_MAGIC_1784661346; gtfam[n] = 4; n = n + 1 487 gtid[n] = IC_MAGIC_1784664001; gtfam[n] = 4; n = n + 1 488 gtid[n] = IC_MAGIC_1785027242; gtfam[n] = 5; n = n + 1 489 gtid[n] = IC_MAGIC_1784908046; gtfam[n] = 5; n = n + 1 490 gtid[n] = IC_MAGIC_1784947125; gtfam[n] = 3; n = n + 1 491 gtid[n] = IC_MAGIC_1784947134; gtfam[n] = 3; n = n + 1 492 gtid[n] = IC_MAGIC_1784947786; gtfam[n] = 3; n = n + 1 493 gtid[n] = IC_MAGIC_1784951793; gtfam[n] = 3; n = n + 1 494 gtid[n] = IC_MAGIC_1784952157; gtfam[n] = 3; n = n + 1 495 gtid[n] = IC_MAGIC_1784954507; gtfam[n] = 3; n = n + 1 496 gtid[n] = IC_MAGIC_1784955224; gtfam[n] = 3; n = n + 1 497 gtid[n] = IC_MAGIC_1784955875; gtfam[n] = 3; n = n + 1 498 gtid[n] = IC_MAGIC_1784956205; gtfam[n] = 3; n = n + 1 499 gtid[n] = IC_MAGIC_1784995126; gtfam[n] = 3; n = n + 1 500 gtid[n] = IC_MAGIC_1784995721; gtfam[n] = 3; n = n + 1 501 gtid[n] = IC_MAGIC_1784995947; gtfam[n] = 3; n = n + 1 502 gtid[n] = IC_MAGIC_1784996406; gtfam[n] = 3; n = n + 1 503 gtid[n] = IC_MAGIC_1784996558; gtfam[n] = 3; n = n + 1 504 gtid[n] = IC_MAGIC_1785000800; gtfam[n] = 3; n = n + 1 505 gtid[n] = IC_MAGIC_1784925304; gtfam[n] = 6; n = n + 1 506 gtid[n] = IC_MAGIC_1784927702; gtfam[n] = 7; n = n + 1 507 gtid[n] = IC_MAGIC_1784958766; gtfam[n] = 8; n = n + 1 508 gtid[n] = IC_MAGIC_1784978119; gtfam[n] = 9; n = n + 1 509 gtid[n] = IC_MAGIC_1784999221; gtfam[n] = 10; n = n + 1 510 gtid[n] = IC_MAGIC_1785001189; gtfam[n] = 11; n = n + 1 511 gtid[n] = IC_MAGIC_1785002038; gtfam[n] = 12; n = n + 1 512 return n 513} 514 515// BOUNDED OVERLAP: a row belongs to its primary family and, at most, ALSO to its scope family. 516// Splits survived every single-membership design (gold_families_split stuck at 4-5) because one row 517// can only sit in one bucket, so a family whose members carry different anchors CANNOT be whole. 518// Cap is TWO deliberately: unbounded overlap would put a row in every family it name-drops and 519// precision would collapse. B-cubed scores overlapping clusters natively, so the ruler can judge this. 520func ic_in_fam(as1: *i64, as2: *i64, r: i64, c: i64) -> i64 { 521 if as1[r] == c { return 1 } 522 if as2[r] == c { return 1 } 523 return 0 524} 525func ic_share_fam(as1: *i64, as2: *i64, a: i64, b: i64) -> i64 { 526 if as1[a] >= 0 { if ic_in_fam(as1, as2, b, as1[a]) == 1 { return 1 } } 527 if as2[a] >= 0 { if ic_in_fam(as1, as2, b, as2[a]) == 1 { return 1 } } 528 return 0 529} 530func ic_find(par: *i64, x: i64) -> i64 { 531 var r: i64 = x 532 while par[r] != r { r = par[r] } 533 var c: i64 = x 534 while par[c] != c { let nxt: i64 = par[c]; par[c] = r; c = nxt } 535 return r 536} 537func ic_union(par: *i64, a: i64, b: i64) -> i64 { 538 let ra: i64 = ic_find(par, a) 539 let rb: i64 = ic_find(par, b) 540 if ra == rb { return 0 } 541 if ra < rb { par[rb] = ra } else { par[ra] = rb } 542 return 1 543} 544 545// ─── THE PIPELINE ────────────────────────────────────────────────────────────────────────────────────── 546// Runs over an in-memory TSV plane so that `run` and `selftest` execute the IDENTICAL code path. A selftest 547// that exercises a reimplementation of the thing it certifies proves nothing (A DELEGATED CHECK IS NOT A 548// CHECK UNTIL YOU WATCH THE DELEGATE PERFORM IT). 549// 550// arrays, all preallocated once (sys_mmap is page-granular with no free -- allocating inside a loop is the 551// banked unfreed-mmap class): 552// rmeta stride 8: [0]=epoch [1]=sev [2]=open [3]=ntok [4]=idfsum [5]=core [6]=scope_off [7]=scope_len 553// rtok: IC_MAXROWS * IC_MAXTOK slot ids 554// st: [0]=rows [1]=rows_capped [2]=open_rows [3]=distinct_tokens [4]=df_skipped [5]=shape_alt 555// [6]=pairs_lo [7]=pairs_join [8]=pairs_hi 556func ic_parse(q: *u8, n: i64, rmeta: *i64, rtok: *i64, dh: *i64, st: *i64) -> i64 { 557 var rows: i64 = 0 558 var pos: i64 = 0 559 // Field-bound scratch, allocated ONCE. sys_mmap is raw and page-granular with no free, so allocating 560 // these inside the per-row loop would leak 8KB x rows -- the exact unfreed-mmap class this ecosystem 561 // banked as a law. A leak in the organ that ranks leaks would be its own best finding. 562 let fo: *i64 = sys_mmap(64) as *i64 563 let fl: *i64 = sys_mmap(64) as *i64 564 while pos < n { 565 var e: i64 = pos 566 var fin: i64 = 0 567 while fin == 0 { 568 if e >= n { fin = 1 } else { if q[e] == (IC_NL as u8) { fin = 1 } else { e = e + 1 } } 569 } 570 if e > pos { 571 if rows >= IC_MAXROWS { st[1] = st[1] + 1 } else { 572 // ---- field split (bounded scan; we only need the first five field bounds) ---- 573 var f: i64 = 0 574 var fs: i64 = pos 575 var k: i64 = pos 576 while k <= e { 577 var sep: i64 = 0 578 if k == e { sep = 1 } else { if q[k] == (IC_TAB as u8) { sep = 1 } } 579 if sep == 1 { 580 if f < 7 { fo[f] = fs; fl[f] = k - fs } 581 f = f + 1 582 fs = k + 1 583 } 584 k = k + 1 585 } 586 // ---- severity: DEFENSIVE AT THE BOUNDARY (rule 12). The plane carries two row shapes -- 587 // the common `epoch/sev/scope/status/desc` and a rarer `id/title/sev/status/...` (D001). 588 // Reading sev from a fixed index gives the second shape a severity derived from the first 589 // letter of its title. So: accept the first of fields 1..2 that is a LONE digit 1-9. 590 var sev: i64 = 0 591 var sev_at: i64 = 0 - 1 592 var ti: i64 = 1 593 while ti <= 2 { 594 if sev_at < 0 { if f > ti { if fl[ti] == 1 { 595 let c: i64 = q[fo[ti]] as i64 596 if c >= 49 { if c <= 57 { sev = c - 48; sev_at = ti } } 597 } } } 598 ti = ti + 1 599 } 600 if sev_at != 1 { st[5] = st[5] + 1 } 601 // ---- status: an `open` field anywhere in the first five (both shapes put it at 3) ---- 602 var is_open: i64 = 0 603 var si: i64 = 2 604 while si <= 4 { 605 if f > si { if fl[si] == 4 { 606 let o0: i64 = fo[si] 607 if q[o0] == (111 as u8) { if q[o0+1] == (112 as u8) { if q[o0+2] == (101 as u8) { if q[o0+3] == (110 as u8) { is_open = 1 } } } } 608 } } 609 si = si + 1 610 } 611 // ---- scope: only trustworthy when the row is the common shape ---- 612 var sc_off: i64 = 0 613 var sc_len: i64 = 0 614 if sev_at == 1 { if f > 2 { sc_off = fo[2]; sc_len = fl[2] } } 615 // THE SCOPE IS THE FILER'S OWN STATEMENT OF WHAT THE ROW IS ABOUT -- the most 616 // authoritative family signal on the row, and I had been treating it as just another 617 // token. MEASURED: organ-first ordering split 4 of 5 multi-row gold families (worst 618 // into 5 buckets) because a row gets claimed by an organ it merely MENTIONS instead of 619 // by its subject. Marking the scope token outranks that. 620 if sc_len >= IC_TOKMIN { if sc_len <= IC_TOKMAX { 621 if ic_all_digits(q, sc_off, sc_len) == 0 { 622 let scsl: i64 = ic_intern(q, sc_off, sc_len, dh) 623 // SCOPE-FIRST DISABLED 2026-08-01 -- BUILT, MEASURED, REFUTED BY THE SECOND RULER. 624 // B-cubed said BETTER (precision 900->923, F1 739->747) but the label-free 625 // metric said WORSE: emitted_organ_anchored 7 -> 3, and the emitted labels 626 // degenerated to generic buckets -- gates(129 members across 108 SCOPES), 627 // sites, evidence, plane, CORRECTION. It also did NOT do what it was built 628 // for: gold_families_split stayed 4 and worst_split stayed 5. 629 // ROOT CAUSE: a slot rank is GLOBAL, so a scope word that is also common 630 // prose ('gates') outranks everything EVERYWHERE it appears, not only on the 631 // rows it is actually the scope OF. Fixing that needs PER-ROW anchor 632 // eligibility, which this design cannot express. 633 // LAW: WHEN TWO RULERS DISAGREE, THE ONE MEASURING THE THING THAT VISIBLY 634 // CHANGED WINS -- a +23 precision point does not buy 'CORRECTION' as the name 635 // of a work item. 636 if scsl >= 0 { if IC_SCOPE_FIRST == 1 { dh[scsl * 6 + 5] = 2 } } 637 } 638 } } 639 let base: i64 = rows * 8 640 rmeta[base + 0] = ic_slice_num(q, fo[0], fl[0]) 641 rmeta[base + 1] = sev 642 rmeta[base + 2] = is_open 643 rmeta[base + 5] = pos 644 rmeta[base + 6] = sc_off 645 rmeta[base + 7] = sc_len 646 if is_open == 1 { st[2] = st[2] + 1 } 647 // ---- tokenize the WHOLE row. Epochs and counts fall out via the pure-digit rule; `open` 648 // and other structural words fall out via the DF cap. No stopword table exists on purpose. 649 var nt: i64 = 0 650 var p: i64 = pos 651 while p < e { 652 if ic_alnum(q[p] as i64) == 1 { 653 // walk forward over token characters; the flag keeps the END POSITION intact 654 // (breaking by assigning the bound would lose it -- and a token whose length is 655 // silently wrong merges two unrelated identifiers) 656 var end2: i64 = p 657 var scan: i64 = 1 658 while scan == 1 { 659 if end2 >= e { scan = 0 } else { 660 if ic_tokchar(q[end2] as i64) == 1 { end2 = end2 + 1 } else { scan = 0 } 661 } 662 } 663 let L: i64 = end2 - p 664 // over-long tokens are SKIPPED, never truncated: a truncated token is a DIFFERENT 665 // token wearing the same bytes, and it would merge families that share a prefix 666 if L >= IC_TOKMIN { if L <= IC_TOKMAX { 667 if ic_all_digits(q, p, L) == 0 { 668 if nt < IC_MAXTOK { 669 let slot: i64 = ic_intern(q, p, L, dh) 670 if slot >= 0 { 671 // per-row dedup: a token counts once per row (document frequency) 672 var dup: i64 = 0 673 var z: i64 = 0 674 while z < nt { if rtok[rows * IC_MAXTOK + z] == slot { dup = 1 } z = z + 1 } 675 if dup == 0 { 676 rtok[rows * IC_MAXTOK + nt] = slot 677 nt = nt + 1 678 if dh[slot * 6 + 3] == 0 { st[3] = st[3] + 1 } 679 dh[slot * 6 + 3] = dh[slot * 6 + 3] + 1 680 } 681 } 682 } 683 } 684 } } 685 p = end2 686 } else { p = p + 1 } 687 } 688 rmeta[base + 3] = nt 689 rows = rows + 1 690 } 691 } 692 pos = e + 1 693 } 694 st[0] = rows 695 return rows 696} 697 698func main(argc: i64, argv: *i64) -> i64 { 699 if argc < 2 { 700 ic_puts("usage: nx_debtcluster run [prefix] [sim_join_permil] | page <outfile> [prefix] [sim] | selftest\n" as *u8) 701 sys_exit(2); return 2 702 } 703 let verb: *u8 = argv[1] as *u8 704 var mode: i64 = IC_MODE_RUN 705 if verb[0] == (115 as u8) { mode = IC_MODE_SELFTEST } 706 if verb[0] == (112 as u8) { mode = IC_MODE_PAGE } 707 if verb[0] == (101 as u8) { mode = IC_MODE_EVAL } 708 var is_self: i64 = 0 709 if mode == IC_MODE_SELFTEST { is_self = 1 } 710 711 // ⚠THE PLANE PREFIX IS A FULL PATH: knowledge/store/debt-. A bare "debt-" resolves to nothing and the 712 // organ correctly returns INSTRUMENT-BLIND rather than "no clusters found" -- keep that default HONEST 713 // (wrong) rather than convenient, so a mis-invocation is loud instead of quietly empty. 714 var prefix: *u8 = "knowledge/store/debt-" as *u8 715 var simjoin: i64 = IC_SIM_JOIN 716 var outfile: *u8 = "sites/nishifamily/code/families" as *u8 717 if mode == IC_MODE_PAGE { 718 if argc < 3 { ic_puts("usage: nx_debtcluster page <outfile> [prefix] [sim_join_permil]\n" as *u8); sys_exit(2); return 2 } 719 outfile = argv[2] as *u8 720 if argc > 3 { prefix = argv[3] as *u8 } 721 if argc > 4 { simjoin = ic_atoi(argv[4] as *u8) } 722 } 723 if mode == IC_MODE_RUN { 724 if argc > 2 { prefix = argv[2] as *u8 } 725 if argc > 3 { simjoin = ic_atoi(argv[3] as *u8) } 726 } 727 if mode == IC_MODE_EVAL { 728 if argc > 2 { prefix = argv[2] as *u8 } 729 } 730 if simjoin <= 0 { simjoin = IC_SIM_JOIN } 731 if simjoin > 1000 { simjoin = IC_SIM_JOIN } 732 733 let q: *u8 = sys_mmap(IC_CAP) 734 let dbf: *i64 = sys_mmap(64) as *i64 735 dbf[0] = 0 736 dbf[1] = 0 737 var n: i64 = 0 738 739 if is_self == 1 { 740 // ── PLANTED FIXTURES. T1/T2 are the pair that matters: near-identical SYMPTOM TEXT, and the only 741 // difference is WHICH ORGAN it happened in. A text-only clusterer merges rows 0/1 with rows 2/3. 742 // This organ must not. If it does, selftest fails and the build does not ship. 743 var o: i64 = 0 744 o = ic_cat(q, o, "1700000001\t9\tmgmt-api\topen\tnx_mgmt_api_core wedged on start, memory grows unbounded on every call and is never freed\n" as *u8) 745 o = ic_cat(q, o, "1700000002\t8\tmgmt-api\topen\tnx_mgmt_api_core leaks on each request, memory grows unbounded on every call and is never freed\n" as *u8) 746 o = ic_cat(q, o, "1700000003\t8\tvault-walk\topen\tnx_vault_walker_svc leaks on each request, memory grows unbounded on every call and is never freed\n" as *u8) 747 o = ic_cat(q, o, "1700000004\t7\tvault-walk\topen\tnx_vault_walker_svc wedged on start, memory grows unbounded on every call and is never freed\n" as *u8) 748 o = ic_cat(q, o, "1700000005\t4\tsolo-lane\topen\tthe carburettor gasket needs replacing before the spring rally weekend arrives\n" as *u8) 749 n = o 750 dbf[0] = 5 751 dbf[1] = 5 752 } else { 753 n = sts_load_honest(prefix, q, IC_CAP, dbf) 754 if n <= 0 { 755 ic_puts("{\"organ\":\"nx_debtcluster\",\"verdict\":\"INSTRUMENT-BLIND\",\"reason\":\"debt plane loaded ZERO bytes -- no rows were examined, so NO clustering claim is made\"}\n" as *u8) 756 sys_exit(2); return 2 757 } 758 } 759 760 let rmeta: *i64 = sys_mmap(8 * 8 * IC_MAXROWS) as *i64 761 let rtok: *i64 = sys_mmap(8 * IC_MAXTOK * IC_MAXROWS) as *i64 762 let dh: *i64 = sys_mmap(8 * 6 * IC_DICT) as *i64 763 let st: *i64 = sys_mmap(128) as *i64 764 var zz: i64 = 0 765 while zz < 16 { st[zz] = 0; zz = zz + 1 } 766 767 let rows: i64 = ic_parse(q, n, rmeta, rtok, dh, st) 768 if rows <= 0 { 769 ic_puts("{\"organ\":\"nx_debtcluster\",\"verdict\":\"INSTRUMENT-BLIND\",\"reason\":\"plane held bytes but ZERO rows parsed -- the row shape is not what this organ reads\"}\n" as *u8) 770 sys_exit(2); return 2 771 } 772 773 // ── IDF. idf = log2(N) - log2(df), in 1/1024 bit units, floored at 0. A token in every row scores 0 and 774 // therefore contributes nothing to any similarity -- stopwords are eliminated BY MEASUREMENT. 775 let logN: i64 = ic_log2q(rows) 776 let dfmax: i64 = (rows * IC_DF_PERMIL) / 1000 777 var s2: i64 = 0 778 while s2 < IC_DICT { 779 let b2: i64 = s2 * 6 780 if dh[b2 + 2] != 0 { 781 let df: i64 = dh[b2 + 3] 782 if df > dfmax { if dfmax > 0 { dh[b2 + 4] = 0 - 1; st[4] = st[4] + 1 } } 783 if dh[b2 + 4] != (0 - 1) { 784 var v: i64 = logN - ic_log2q(df) 785 if v < 0 { v = 0 } 786 dh[b2 + 4] = v 787 } 788 } 789 s2 = s2 + 1 790 } 791 // per-row IDF mass (skipped tokens contribute nothing, so they cannot inflate an overlap ratio) 792 var r2: i64 = 0 793 while r2 < rows { 794 var sum: i64 = 0 795 var t3: i64 = 0 796 while t3 < rmeta[r2 * 8 + 3] { 797 let sl: i64 = rtok[r2 * IC_MAXTOK + t3] 798 let w: i64 = dh[sl * 6 + 4] 799 if w > 0 { sum = sum + w } 800 t3 = t3 + 1 801 } 802 rmeta[r2 * 8 + 4] = sum 803 rmeta[r2 * 8 + 5] = 0 804 r2 = r2 + 1 805 } 806 807 // ── POSTINGS (token -> rows). Turns an O(rows^2) all-pairs scan into a scan over rows that actually 808 // share a rare token. Without this, 2068 rows is 2.1M pairwise set intersections. 809 let pstart: *i64 = sys_mmap(8 * (IC_DICT + 1)) as *i64 810 let pfill: *i64 = sys_mmap(8 * (IC_DICT + 1)) as *i64 811 let prow: *i64 = sys_mmap(8 * IC_POSTCAP) as *i64 812 var s3: i64 = 0 813 var acc2: i64 = 0 814 while s3 < IC_DICT { 815 pstart[s3] = acc2 816 pfill[s3] = 0 817 let b3: i64 = s3 * 6 818 if dh[b3 + 2] != 0 { if dh[b3 + 4] > 0 { acc2 = acc2 + dh[b3 + 3] } } 819 s3 = s3 + 1 820 } 821 var r3: i64 = 0 822 while r3 < rows { 823 var t4: i64 = 0 824 while t4 < rmeta[r3 * 8 + 3] { 825 let sl2: i64 = rtok[r3 * IC_MAXTOK + t4] 826 if dh[sl2 * 6 + 4] > 0 { 827 let idx2: i64 = pstart[sl2] + pfill[sl2] 828 if idx2 < IC_POSTCAP { prow[idx2] = r3; pfill[sl2] = pfill[sl2] + 1 } 829 } 830 t4 = t4 + 1 831 } 832 r3 = r3 + 1 833 } 834 835 // ── ANCHOR PASS. A candidate anchor is an identifier token carried by at least IC_MINFAM rows and by no 836 // more than IC_ANCHOR_DF_PERMIL of them. Its family is EXACTLY the rows that contain it. Membership is a 837 // property each row holds ON ITS OWN, so there is no path through other rows and chaining cannot occur. 838 let acand: *i64 = sys_mmap(8 * IC_MAXFAM) as *i64 839 let apri: *i64 = sys_mmap(8 * IC_MAXFAM) as *i64 840 // aorg[i] = 1 when this anchor NAMES A REAL SOURCE FILE. Organ-named anchors claim their rows FIRST, 841 // so a family gets named after the thing it is about rather than after a phrase its rows happen to 842 // share. This is the higher-precision blocking strategy running ahead of the general one -- which is 843 // what the entity-resolution literature actually prescribes, rather than merging afterwards. 844 let aorg: *i64 = sys_mmap(8 * IC_MAXFAM) as *i64 845 let ascp: *i64 = sys_mmap(8 * IC_MAXFAM) as *i64 846 let orgscratch: *u8 = sys_mmap(IC_MAGIC_1024) 847 var ncand: i64 = 0 848 var anchors_too_broad: i64 = 0 849 // FLOOR THE PROPORTION. (rows * permil)/1000 INTEGER-DIVIDES TO ZERO on a small plane -- at rows=5 the 850 // cap came out 0, every anchor was rejected as too broad, and the organ reported families=0 while 851 // cheerfully declaring coverage_complete=1. Same shape as the minPts error above. 852 // A THRESHOLD DERIVED AS A PROPORTION MUST BE FLOORED AT THE DOMAIN'S SMALLEST REAL UNIT, or it 853 // silently forbids the very thing it is measuring. 854 var anchor_dfmax: i64 = (rows * IC_ANCHOR_DF_PERMIL) / 1000 855 if anchor_dfmax < IC_MINFAM { anchor_dfmax = IC_MINFAM } 856 var sA: i64 = 0 857 while sA < IC_DICT { 858 let bA: i64 = sA * 6 859 if dh[bA + 2] != 0 { if dh[bA + 5] >= 1 { if dh[bA + 4] > 0 { 860 let dfA: i64 = dh[bA + 3] 861 if dfA >= IC_MINFAM { 862 if dfA > anchor_dfmax { anchors_too_broad = anchors_too_broad + 1 } else { 863 if ncand < IC_MAXFAM { 864 var mx: i64 = 0 865 var op: i64 = 0 866 var pi2: i64 = 0 867 while pi2 < pfill[sA] { 868 let rr: i64 = prow[pstart[sA] + pi2] 869 if rmeta[rr * 8 + 1] > mx { mx = rmeta[rr * 8 + 1] } 870 if rmeta[rr * 8 + 2] == 1 { op = op + 1 } 871 pi2 = pi2 + 1 872 } 873 acand[ncand] = sA 874 apri[ncand] = mx * op 875 aorg[ncand] = ic_is_organ(q, dh[bA + 1], dh[bA + 2], orgscratch) 876 ascp[ncand] = 0 877 if dh[bA + 5] >= 2 { ascp[ncand] = 1 } 878 ncand = ncand + 1 879 } 880 } 881 } 882 } } } 883 sA = sA + 1 884 } 885 886 // Claim order: urgent tier first, then highest value. Greedy set-cover -- a row joins EXACTLY ONE family, 887 // so the emitted list is a partition of the clustered rows and no row is ever counted twice across two 888 // recommendations (double-counting would inflate every total on the page). 889 var oa: i64 = 0 890 while oa < ncand { 891 var bi: i64 = oa 892 var ob2: i64 = oa + 1 893 while ob2 < ncand { 894 var better: i64 = 0 895 // CLAIM BY VALUE ALONE (cohesion). The organ tier is deliberately NOT consulted here -- 896 // see IC_ORGAN_CLAIM below for the measured reason. 897 if IC_ORGAN_CLAIM == 1 { 898 if aorg[ob2] > aorg[bi] { better = 1 } 899 if aorg[ob2] == aorg[bi] { if apri[ob2] > apri[bi] { better = 1 } } 900 } 901 if IC_ORGAN_CLAIM == 0 { if apri[ob2] > apri[bi] { better = 1 } } 902 if better == 1 { bi = ob2 } 903 ob2 = ob2 + 1 904 } 905 let t1: i64 = acand[oa] 906 acand[oa] = acand[bi] 907 acand[bi] = t1 908 let t2: i64 = apri[oa] 909 apri[oa] = apri[bi] 910 apri[bi] = t2 911 let t3: i64 = aorg[oa] 912 aorg[oa] = aorg[bi] 913 aorg[bi] = t3 914 let t4: i64 = ascp[oa] 915 ascp[oa] = ascp[bi] 916 ascp[bi] = t4 917 oa = oa + 1 918 } 919 920 // each row's OWN scope token, resolved once (ic_intern is idempotent -- returns the existing slot) 921 let rscope: *i64 = sys_mmap(8 * IC_MAXROWS) as *i64 922 var rs: i64 = 0 923 while rs < rows { 924 rscope[rs] = 0 - 1 925 if rmeta[rs * 8 + 7] >= IC_TOKMIN { if rmeta[rs * 8 + 7] <= IC_TOKMAX { 926 if ic_all_digits(q, rmeta[rs * 8 + 6], rmeta[rs * 8 + 7]) == 0 { 927 rscope[rs] = ic_intern(q, rmeta[rs * 8 + 6], rmeta[rs * 8 + 7], dh) 928 } 929 } } 930 rs = rs + 1 931 } 932 let assign: *i64 = sys_mmap(8 * IC_MAXROWS) as *i64 933 var za: i64 = 0 934 while za < rows { assign[za] = 0 - 1; za = za + 1 } 935 let cmeta: *i64 = sys_mmap(8 * 8 * IC_MAXFAM) as *i64 936 var nc: i64 = 0 937 var phase: i64 = 0 938 while phase < 2 { 939 var ci3: i64 = 0 940 while ci3 < ncand { 941 if nc < IC_MAXFAM { 942 let sl5: i64 = acand[ci3] 943 var freen: i64 = 0 944 var pk: i64 = 0 945 while pk < pfill[sl5] { 946 let cr: i64 = prow[pstart[sl5] + pk] 947 if assign[cr] < 0 { 948 if phase == 1 { freen = freen + 1 } 949 if phase == 0 { if rscope[cr] == sl5 { freen = freen + 1 } } 950 } 951 pk = pk + 1 952 } 953 // A family whose rows have all been claimed by a higher-value anchor is NOT emitted again. 954 if freen >= IC_MINFAM { 955 let cb3: i64 = nc * 8 956 cmeta[cb3 + 0] = sl5 957 cmeta[cb3 + 1] = 0 958 cmeta[cb3 + 2] = 0 959 cmeta[cb3 + 3] = 0 960 cmeta[cb3 + 4] = 0 961 cmeta[cb3 + 5] = 0 962 cmeta[cb3 + 7] = 0 963 var pm: i64 = 0 964 while pm < pfill[sl5] { 965 let rw: i64 = prow[pstart[sl5] + pm] 966 var elig: i64 = 0 967 if assign[rw] < 0 { 968 if phase == 1 { elig = 1 } 969 if phase == 0 { if rscope[rw] == sl5 { elig = 1 } } 970 } 971 if elig == 1 { 972 assign[rw] = nc 973 cmeta[cb3 + 1] = cmeta[cb3 + 1] + 1 974 if rmeta[rw * 8 + 2] == 1 { cmeta[cb3 + 2] = cmeta[cb3 + 2] + 1 } 975 if rmeta[rw * 8 + 1] > cmeta[cb3 + 3] { cmeta[cb3 + 3] = rmeta[rw * 8 + 1] } 976 let ep2: i64 = rmeta[rw * 8 + 0] 977 if ep2 > 0 { 978 if cmeta[cb3 + 4] == 0 { cmeta[cb3 + 4] = ep2 } 979 if ep2 < cmeta[cb3 + 4] { cmeta[cb3 + 4] = ep2 } 980 if ep2 > cmeta[cb3 + 5] { cmeta[cb3 + 5] = ep2 } 981 } 982 } 983 pm = pm + 1 984 } 985 nc = nc + 1 986 } 987 } 988 ci3 = ci3 + 1 989 } 990 phase = phase + 1 991 } 992 993 // SECONDARY (scope) MEMBERSHIP -- the overlap that heals splits. 994 let assign2: *i64 = sys_mmap(8 * IC_MAXROWS) as *i64 995 let sfam: *i64 = sys_mmap(8 * IC_DICT) as *i64 996 var sf2: i64 = 0 997 while sf2 < IC_DICT { sfam[sf2] = 0 - 1; sf2 = sf2 + 1 } 998 var cf2: i64 = 0 999 while cf2 < nc { sfam[cmeta[cf2 * 8 + 0]] = cf2; cf2 = cf2 + 1 } 1000 var overlap_added: i64 = 0 1001 var ov: i64 = 0 1002 while ov < rows { 1003 assign2[ov] = 0 - 1 1004 if IC_OVERLAP == 1 { if rscope[ov] >= 0 { 1005 let sfx: i64 = sfam[rscope[ov]] 1006 if sfx >= 0 { if sfx != assign[ov] { 1007 assign2[ov] = sfx 1008 overlap_added = overlap_added + 1 1009 } } 1010 } } 1011 ov = ov + 1 1012 } 1013 // recount members WITH secondary membership so the published table matches who is actually in it 1014 if overlap_added > 0 { 1015 var oc: i64 = 0 1016 while oc < nc { cmeta[oc * 8 + 1] = 0; cmeta[oc * 8 + 2] = 0; cmeta[oc * 8 + 3] = 0; oc = oc + 1 } 1017 var orr: i64 = 0 1018 while orr < rows { 1019 var oq: i64 = 0 1020 while oq < nc { 1021 if ic_in_fam(assign, assign2, orr, oq) == 1 { 1022 cmeta[oq * 8 + 1] = cmeta[oq * 8 + 1] + 1 1023 if rmeta[orr * 8 + 2] == 1 { cmeta[oq * 8 + 2] = cmeta[oq * 8 + 2] + 1 } 1024 if rmeta[orr * 8 + 1] > cmeta[oq * 8 + 3] { cmeta[oq * 8 + 3] = rmeta[orr * 8 + 1] } 1025 } 1026 oq = oq + 1 1027 } 1028 orr = orr + 1 1029 } 1030 var op: i64 = 0 1031 while op < nc { cmeta[op * 8 + 6] = cmeta[op * 8 + 3] * cmeta[op * 8 + 2]; op = op + 1 } 1032 } 1033 1034 // distinct scopes per family (context, not a score term), then the priority 1035 var cs: i64 = 0 1036 while cs < nc { 1037 var distinct: i64 = 0 1038 var m1: i64 = 0 1039 while m1 < rows { 1040 if assign[m1] == cs { if rmeta[m1 * 8 + 7] > 0 { 1041 var seen: i64 = 0 1042 var m2: i64 = 0 1043 while m2 < m1 { 1044 if assign[m2] == cs { 1045 if ic_slice_eq(q, rmeta[m1 * 8 + 6], rmeta[m1 * 8 + 7], rmeta[m2 * 8 + 6], rmeta[m2 * 8 + 7]) == 1 { seen = 1 } 1046 } 1047 m2 = m2 + 1 1048 } 1049 if seen == 0 { distinct = distinct + 1 } 1050 } } 1051 m1 = m1 + 1 1052 } 1053 cmeta[cs * 8 + 7] = distinct 1054 // ★PRIORITY = max severity x OPEN recurrence. Two measured terms, no invented weights, and already 1055 // CLOSED rows deliberately do not inflate it -- a family that is mostly eaten is mostly done. 1056 // distinct_scopes was REMOVED from the product: under anchor clustering it rewards a GENERIC anchor 1057 // (one touching many lanes) over a specific one, which is backwards for "what should I build". 1058 // A family holding a sev>=IC_URGENT_SEV row is escalated ORDINALLY, never by inflating the number. 1059 var pr: i64 = cmeta[cs * 8 + 3] * cmeta[cs * 8 + 2] 1060 // URGENT-TIER BOOST REMOVED 2026-07-31, ON MEASUREMENT. The intent was that an outage preempts. 1061 // But sev-9 rows are spread across this plane, so 8 of the top 8 families qualified and the tier 1062 // collapsed into 'sort the sev-9 families by size' -- which is what let a 199-row date anchor rank 1063 // first. A tier that almost every candidate reaches is not a tier, it is a constant. 1064 // max_sev x open already weights severity honestly and keeps the ordering readable. 1065 // LAW: AN ESCALATION EVERYTHING QUALIFIES FOR RANKS NOTHING. 1066 cmeta[cs * 8 + 6] = pr 1067 cs = cs + 1 1068 } 1069 // ── RANK + EMIT 1070 // MERGE PASS -- RECALL RECOVERY ON AGGREGATE EVIDENCE. 1071 // The ruler measured precision 840 / recall 635: what anchors group is right, but a true family whose 1072 // members carry DIFFERENT anchors gets split. 2026 entity-resolution practice is explicit about the 1073 // remedy -- RECALL IMPROVES BY RUNNING SEVERAL DIVERSE BLOCKING STRATEGIES IN PARALLEL, precision by 1074 // hard vetoes. Anchors are ONE blocking strategy. 1075 // The v1 linker also recovered recall -- by transitive closure -- and chained catastrophically (44 1076 // members / 15 scopes). The decisive difference here is the EVIDENCE THRESHOLD: two families merge only 1077 // when a MAJORITY of the SMALLER family's rows each independently carry the other family's anchor. A 1078 // single bridging row can never trigger it, which is precisely the property single linkage lacked. 1079 // ONE LEVEL ONLY: a family that has merged is not itself re-merged, so no chain can form. 1080 let slot_fam: *i64 = sys_mmap(8 * IC_DICT) as *i64 1081 var sf: i64 = 0 1082 while sf < IC_DICT { slot_fam[sf] = 0 - 1; sf = sf + 1 } 1083 var cf: i64 = 0 1084 while cf < nc { slot_fam[cmeta[cf * 8 + 0]] = cf; cf = cf + 1 } 1085 let cofam: *i64 = sys_mmap(8 * IC_MAXFAM) as *i64 1086 let cotouch: *i64 = sys_mmap(8 * IC_MAXFAM) as *i64 1087 let fmerge: *i64 = sys_mmap(8 * IC_MAXFAM) as *i64 1088 var zf: i64 = 0 1089 while zf < nc { cofam[zf] = 0; fmerge[zf] = 0 - 1; zf = zf + 1 } 1090 var merged_n: i64 = 0 1091 var fa: i64 = 0 1092 while fa < nc { 1093 var nct: i64 = 0 1094 var rw2: i64 = 0 1095 while rw2 < rows { 1096 if assign[rw2] == fa { 1097 var tk: i64 = 0 1098 while tk < rmeta[rw2 * 8 + 3] { 1099 let sl6: i64 = rtok[rw2 * IC_MAXTOK + tk] 1100 let fb: i64 = slot_fam[sl6] 1101 if fb >= 0 { if fb != fa { 1102 if cofam[fb] == 0 { if nct < IC_MAXFAM { cotouch[nct] = fb; nct = nct + 1 } } 1103 cofam[fb] = cofam[fb] + 1 1104 } } 1105 tk = tk + 1 1106 } 1107 } 1108 rw2 = rw2 + 1 1109 } 1110 var bestb: i64 = 0 - 1 1111 var bestratio: i64 = 0 1112 var ct: i64 = 0 1113 while ct < nct { 1114 let fb2: i64 = cotouch[ct] 1115 var den3: i64 = cmeta[fa * 8 + 1] 1116 if cmeta[fb2 * 8 + 1] < den3 { den3 = cmeta[fb2 * 8 + 1] } 1117 var ratio: i64 = 0 1118 if den3 > 0 { ratio = (cofam[fb2] * 1000) / den3 } 1119 if ratio > bestratio { bestratio = ratio; bestb = fb2 } 1120 cofam[fb2] = 0 1121 ct = ct + 1 1122 } 1123 if IC_MERGE_ENABLED == 1 { if bestb >= 0 { if bestratio >= IC_MERGE_PERMIL { if fmerge[fa] < 0 { if fmerge[bestb] < 0 { 1124 fmerge[fa] = bestb 1125 merged_n = merged_n + 1 1126 } } } } } 1127 fa = fa + 1 1128 } 1129 // apply the merges, then recompute every family stat from the ASSIGNMENT so the table can never 1130 // disagree with who is actually in it 1131 if merged_n > 0 { 1132 var rm2: i64 = 0 1133 while rm2 < rows { 1134 if assign[rm2] >= 0 { if fmerge[assign[rm2]] >= 0 { assign[rm2] = fmerge[assign[rm2]] } } 1135 rm2 = rm2 + 1 1136 } 1137 var rc2: i64 = 0 1138 while rc2 < nc { 1139 cmeta[rc2 * 8 + 1] = 0 1140 cmeta[rc2 * 8 + 2] = 0 1141 cmeta[rc2 * 8 + 3] = 0 1142 cmeta[rc2 * 8 + 4] = 0 1143 cmeta[rc2 * 8 + 5] = 0 1144 rc2 = rc2 + 1 1145 } 1146 var rq: i64 = 0 1147 while rq < rows { 1148 let fq: i64 = assign[rq] 1149 if fq >= 0 { 1150 cmeta[fq * 8 + 1] = cmeta[fq * 8 + 1] + 1 1151 if rmeta[rq * 8 + 2] == 1 { cmeta[fq * 8 + 2] = cmeta[fq * 8 + 2] + 1 } 1152 if rmeta[rq * 8 + 1] > cmeta[fq * 8 + 3] { cmeta[fq * 8 + 3] = rmeta[rq * 8 + 1] } 1153 let eq2: i64 = rmeta[rq * 8 + 0] 1154 if eq2 > 0 { 1155 if cmeta[fq * 8 + 4] == 0 { cmeta[fq * 8 + 4] = eq2 } 1156 if eq2 < cmeta[fq * 8 + 4] { cmeta[fq * 8 + 4] = eq2 } 1157 if eq2 > cmeta[fq * 8 + 5] { cmeta[fq * 8 + 5] = eq2 } 1158 } 1159 } 1160 rq = rq + 1 1161 } 1162 var rp: i64 = 0 1163 while rp < nc { 1164 var pr2: i64 = cmeta[rp * 8 + 3] * cmeta[rp * 8 + 2] 1165 cmeta[rp * 8 + 6] = pr2 1166 rp = rp + 1 1167 } 1168 } 1169 1170 // DENSE PLACEMENT OF NOISE ROWS 1171 var dense_placed: i64 = 0 1172 var dense_model: i64 = 0 1173 if IC_DENSE_ENABLED == 1 { 1174 let g: *i64 = sys_mmap(8 * 128) as *i64 // ppl_load writes g[69..78] (needs >=632B); 512B was the beir-eval arena-overrun crash class 1175 if ppl_load(g, "knowledge/index/semppmi_v1.bin" as *u8) > 0 { dense_model = 1 } 1176 if dense_model == 1 { 1177 let frep: *i64 = sys_mmap(8 * IC_MAXFAM) as *i64 1178 var fz: i64 = 0 1179 while fz < nc { frep[fz] = 0 - 1; fz = fz + 1 } 1180 var rr3: i64 = 0 1181 while rr3 < rows { 1182 if assign[rr3] >= 0 { if frep[assign[rr3]] < 0 { frep[assign[rr3]] = rr3 } } 1183 rr3 = rr3 + 1 1184 } 1185 var topf: i64 = nc 1186 if topf > IC_DENSE_TOPF { topf = IC_DENSE_TOPF } 1187 let ftok: *i64 = sys_mmap(8 * IC_DENSE_TOPF * IC_MAXTOK) as *i64 1188 let ftokn: *i64 = sys_mmap(8 * IC_DENSE_TOPF) as *i64 1189 let ta: *i64 = sys_mmap(8 * IC_MAXTOK) as *i64 1190 var ff: i64 = 0 1191 while ff < topf { 1192 ftokn[ff] = 0 1193 let rp: i64 = frep[ff] 1194 if rp >= 0 { 1195 let sp: i64 = rmeta[rp * 8 + 5] 1196 var ep: i64 = sp 1197 var scn: i64 = 1 1198 while scn == 1 { 1199 if ep >= n { scn = 0 } else { if q[ep] == (IC_NL as u8) { scn = 0 } else { ep = ep + 1 } } 1200 } 1201 ftokn[ff] = ppl_tokenize_ids(g, ((q as i64) + sp) as *u8, ep - sp, ((ftok as i64) + ff * IC_MAXTOK * 8) as *i64, IC_MAXTOK) 1202 } 1203 ff = ff + 1 1204 } 1205 var nz: i64 = 0 1206 while nz < rows { 1207 if assign[nz] < 0 { 1208 let s1: i64 = rmeta[nz * 8 + 5] 1209 var e1: i64 = s1 1210 var sc2: i64 = 1 1211 while sc2 == 1 { 1212 if e1 >= n { sc2 = 0 } else { if q[e1] == (IC_NL as u8) { sc2 = 0 } else { e1 = e1 + 1 } } 1213 } 1214 let na: i64 = ppl_tokenize_ids(g, ((q as i64) + s1) as *u8, e1 - s1, ta, IC_MAXTOK) 1215 if na > 0 { 1216 var bestf: i64 = 0 - 1 1217 var bestsim: i64 = 0 1218 var gf: i64 = 0 1219 while gf < topf { 1220 if ftokn[gf] > 0 { 1221 let sim3: i64 = ppl_maxsim_idf(g, ta, na, ((ftok as i64) + gf * IC_MAXTOK * 8) as *i64, ftokn[gf]) 1222 if sim3 > bestsim { bestsim = sim3; bestf = gf } 1223 } 1224 gf = gf + 1 1225 } 1226 if bestf >= 0 { if bestsim >= IC_DENSE_MIN { 1227 assign[nz] = bestf 1228 dense_placed = dense_placed + 1 1229 } } 1230 } 1231 } 1232 nz = nz + 1 1233 } 1234 if dense_placed > 0 { 1235 var rc3: i64 = 0 1236 while rc3 < nc { 1237 cmeta[rc3 * 8 + 1] = 0 1238 cmeta[rc3 * 8 + 2] = 0 1239 cmeta[rc3 * 8 + 3] = 0 1240 rc3 = rc3 + 1 1241 } 1242 var rq2: i64 = 0 1243 while rq2 < rows { 1244 let fq2: i64 = assign[rq2] 1245 if fq2 >= 0 { 1246 cmeta[fq2 * 8 + 1] = cmeta[fq2 * 8 + 1] + 1 1247 if rmeta[rq2 * 8 + 2] == 1 { cmeta[fq2 * 8 + 2] = cmeta[fq2 * 8 + 2] + 1 } 1248 if rmeta[rq2 * 8 + 1] > cmeta[fq2 * 8 + 3] { cmeta[fq2 * 8 + 3] = rmeta[rq2 * 8 + 1] } 1249 } 1250 rq2 = rq2 + 1 1251 } 1252 var rp2: i64 = 0 1253 while rp2 < nc { 1254 cmeta[rp2 * 8 + 6] = cmeta[rp2 * 8 + 3] * cmeta[rp2 * 8 + 2] 1255 rp2 = rp2 + 1 1256 } 1257 } 1258 } 1259 } 1260 1261 let order: *i64 = sys_mmap(8 * IC_MAXROWS) as *i64 1262 var oi: i64 = 0 1263 while oi < nc { order[oi] = oi; oi = oi + 1 } 1264 var a1: i64 = 0 1265 while a1 < nc { 1266 var best: i64 = a1 1267 var a2: i64 = a1 + 1 1268 while a2 < nc { 1269 if cmeta[order[a2] * 8 + 6] > cmeta[order[best] * 8 + 6] { best = a2 } 1270 a2 = a2 + 1 1271 } 1272 let tmp: i64 = order[a1]; order[a1] = order[best]; order[best] = tmp 1273 a1 = a1 + 1 1274 } 1275 1276 // Counted from the ASSIGNMENT, not from the family table: every row is either in exactly one family or 1277 // it is noise, so clustered + noise must equal rows. Deriving noise from the family list instead (as the 1278 // union-find version did) let the two disagree without anything noticing. 1279 var clustered: i64 = 0 1280 var noise: i64 = 0 1281 let families: i64 = nc 1282 var ci2: i64 = 0 1283 while ci2 < rows { 1284 if assign[ci2] >= 0 { clustered = clustered + 1 } else { noise = noise + 1 } 1285 ci2 = ci2 + 1 1286 } 1287 1288 let ob: *u8 = sys_mmap(IC_OUTCAP) 1289 // The HTML page is rendered from the SAME pass as the JSON, not from a re-read of it. A page generated 1290 // by re-parsing this organ's own output would be a second instrument able to disagree with the first. 1291 let hb: *u8 = sys_mmap(IC_OUTCAP) 1292 let lbuf: *u8 = sys_mmap(IC_LBUF) 1293 var ho: i64 = 0 1294 var o2: i64 = ic_cat(ob, 0, "{\"organ\":\"nx_debtcluster\",\"verb\":\"" as *u8) 1295 if mode == IC_MODE_SELFTEST { o2 = ic_cat(ob, o2, "selftest" as *u8) } 1296 if mode == IC_MODE_RUN { o2 = ic_cat(ob, o2, "run" as *u8) } 1297 if mode == IC_MODE_PAGE { o2 = ic_cat(ob, o2, "page" as *u8) } 1298 if mode == IC_MODE_EVAL { o2 = ic_cat(ob, o2, "eval" as *u8) } 1299 o2 = ic_cat(ob, o2, "\",\"recommendations\":[" as *u8) 1300 1301 var emitted: i64 = 0 1302 // LABEL-FREE QUALITY SIGNAL, added because the 20-row gold set stayed FLAT through a regression that 1303 // was obvious by eye. This one moves: it counts how many emitted families are named after a real 1304 // source file rather than a phrase. It needs no labels, so it scales with the corpus. 1305 let lcnt: *i64 = sys_mmap(8 * IC_DICT) as *i64 1306 let ltouch: *i64 = sys_mmap(8 * IC_POSTCAP) as *i64 1307 var emitted_organ: i64 = 0 1308 var minpri: i64 = 0 - 1 1309 var ei: i64 = 0 1310 while ei < nc { 1311 if emitted < IC_EMIT { 1312 let c2: i64 = order[ei] 1313 if cmeta[c2 * 8 + 1] >= IC_MINFAM { if cmeta[c2 * 8 + 6] > 0 { 1314 var anchsl: i64 = cmeta[c2 * 8 + 0] 1315 // If the claiming anchor does not NAME anything real, relabel the family with the 1316 // organ-named token its members most widely share. Membership is unchanged -- only the 1317 // name. Falls back to the anchor when the family shares no organ name at all. 1318 if ic_is_organ(q, dh[anchsl * 6 + 1], dh[anchsl * 6 + 2], orgscratch) == 0 { 1319 var nlt: i64 = 0 1320 var mm: i64 = 0 1321 while mm < rows { 1322 if ic_in_fam(assign, assign2, mm, c2) == 1 { 1323 var tt: i64 = 0 1324 while tt < rmeta[mm * 8 + 3] { 1325 let slx: i64 = rtok[mm * IC_MAXTOK + tt] 1326 if dh[slx * 6 + 4] > 0 { 1327 if lcnt[slx] == 0 { if nlt < IC_POSTCAP { ltouch[nlt] = slx; nlt = nlt + 1 } } 1328 lcnt[slx] = lcnt[slx] + 1 1329 } 1330 tt = tt + 1 1331 } 1332 } 1333 mm = mm + 1 1334 } 1335 var bestn: i64 = 1 1336 var bestsl: i64 = 0 - 1 1337 var lz: i64 = 0 1338 while lz < nlt { 1339 let sy: i64 = ltouch[lz] 1340 // SHARE x SPECIFICITY, not share alone. Scoring by share alone produced 1341 // 'log' and 'import' (TWICE, so two families carried the SAME name) -- 1342 // both genuinely match a file on disk, so emitted_organ_anchored read 8/8 1343 // while the names got WORSE. I gamed my own metric. 1344 // LAW: A TOKEN CAN NAME A REAL THING AND STILL BE A USELESS NAME -- an 1345 // existence test is necessary and NOT sufficient; weight by how much the 1346 // token actually distinguishes this family from every other. 1347 if lcnt[sy] * dh[sy * 6 + 4] > bestn { 1348 if ic_is_organ(q, dh[sy * 6 + 1], dh[sy * 6 + 2], orgscratch) == 1 { bestn = lcnt[sy] * dh[sy * 6 + 4]; bestsl = sy } 1349 } 1350 lz = lz + 1 1351 } 1352 var lc2: i64 = 0 1353 while lc2 < nlt { lcnt[ltouch[lc2]] = 0; lc2 = lc2 + 1 } 1354 if bestsl >= 0 { anchsl = bestsl } 1355 } 1356 if ic_is_organ(q, dh[anchsl * 6 + 1], dh[anchsl * 6 + 2], orgscratch) == 1 { emitted_organ = emitted_organ + 1 } 1357 if emitted > 0 { o2 = ic_cat(ob, o2, "," as *u8) } 1358 o2 = ic_cat(ob, o2, "{\"rank\":" as *u8) 1359 o2 = ic_catn(ob, o2, emitted + 1) 1360 // ★THE LABEL IS THE ANCHOR ITSELF. Under anchor clustering the family's name is not INFERRED 1361 // from its members -- every member demonstrably CONTAINS this exact token -- so the name can 1362 // never drift from the thing it names. The previous scheme scored the highest-IDF tokens the 1363 // members happened to share and produced "blanket-rebuild|half-work|stale_sweep" for a family 1364 // whose actual subject was nx_stale_sweep: it was naming the WARNING TEXT inside the rows. 1365 var lo: i64 = 0 1366 if dh[anchsl * 6 + 2] < IC_LBUF - 2 { lo = ic_catsl(lbuf, lo, q, dh[anchsl * 6 + 1], dh[anchsl * 6 + 2]) } 1367 lbuf[lo] = 0 as u8 1368 o2 = ic_cat(ob, o2, ",\"label\":\"" as *u8) 1369 o2 = ic_cat(ob, o2, lbuf) 1370 // one HTML row, same numbers, same pass 1371 if ho < IC_OUTCAP - IC_MAGIC_2048 { 1372 ho = ic_cat(hb, ho, "<tr><td class='n'>" as *u8); ho = ic_catn(hb, ho, emitted + 1) 1373 ho = ic_cat(hb, ho, "</td><td><code>" as *u8); ho = ic_cat(hb, ho, lbuf) 1374 ho = ic_cat(hb, ho, "</code></td><td class='n'>" as *u8); ho = ic_catn(hb, ho, cmeta[c2 * 8 + 1]) 1375 ho = ic_cat(hb, ho, "</td><td class='n'>" as *u8); ho = ic_catn(hb, ho, cmeta[c2 * 8 + 2]) 1376 ho = ic_cat(hb, ho, "</td><td class='n'>" as *u8); ho = ic_catn(hb, ho, cmeta[c2 * 8 + 3]) 1377 ho = ic_cat(hb, ho, "</td><td class='n'>" as *u8); ho = ic_catn(hb, ho, cmeta[c2 * 8 + 7]) 1378 var spanh: i64 = 0 1379 if cmeta[c2 * 8 + 5] > cmeta[c2 * 8 + 4] { spanh = (cmeta[c2 * 8 + 5] - cmeta[c2 * 8 + 4]) / IC_DAY } 1380 ho = ic_cat(hb, ho, "</td><td class='n'>" as *u8); ho = ic_catn(hb, ho, spanh) 1381 ho = ic_cat(hb, ho, "</td><td class='n'>" as *u8); ho = ic_catn(hb, ho, cmeta[c2 * 8 + 6]) 1382 ho = ic_cat(hb, ho, "</td><td class='idx'>" as *u8) 1383 var hs: i64 = 0 1384 var hv: i64 = 0 1385 while hv < rows { 1386 if ic_in_fam(assign, assign2, hv, c2) == 1 { if hs < IC_IDS_PER_CLUSTER { 1387 if hs > 0 { ho = ic_cat(hb, ho, ", " as *u8) } 1388 ho = ic_catn(hb, ho, hv) 1389 hs = hs + 1 1390 } } 1391 hv = hv + 1 1392 } 1393 if cmeta[c2 * 8 + 1] > hs { ho = ic_cat(hb, ho, " <span class='muted'>(+" as *u8); ho = ic_catn(hb, ho, cmeta[c2 * 8 + 1] - hs); ho = ic_cat(hb, ho, " more)</span>" as *u8) } 1394 ho = ic_cat(hb, ho, "</td></tr>\n" as *u8) 1395 } 1396 o2 = ic_cat(ob, o2, "\",\"members\":" as *u8); o2 = ic_catn(ob, o2, cmeta[c2 * 8 + 1]) 1397 o2 = ic_cat(ob, o2, ",\"open\":" as *u8); o2 = ic_catn(ob, o2, cmeta[c2 * 8 + 2]) 1398 o2 = ic_cat(ob, o2, ",\"max_sev\":" as *u8); o2 = ic_catn(ob, o2, cmeta[c2 * 8 + 3]) 1399 o2 = ic_cat(ob, o2, ",\"scopes\":" as *u8); o2 = ic_catn(ob, o2, cmeta[c2 * 8 + 7]) 1400 var span: i64 = 0 1401 if cmeta[c2 * 8 + 5] > cmeta[c2 * 8 + 4] { span = (cmeta[c2 * 8 + 5] - cmeta[c2 * 8 + 4]) / IC_DAY } 1402 o2 = ic_cat(ob, o2, ",\"span_days\":" as *u8); o2 = ic_catn(ob, o2, span) 1403 o2 = ic_cat(ob, o2, ",\"priority\":" as *u8); o2 = ic_catn(ob, o2, cmeta[c2 * 8 + 6]) 1404 o2 = ic_cat(ob, o2, ",\"idx\":[" as *u8) 1405 var shown: i64 = 0 1406 var mv: i64 = 0 1407 while mv < rows { 1408 if ic_in_fam(assign, assign2, mv, c2) == 1 { if shown < IC_IDS_PER_CLUSTER { 1409 if shown > 0 { o2 = ic_cat(ob, o2, "," as *u8) } 1410 o2 = ic_catn(ob, o2, mv) 1411 shown = shown + 1 1412 } } 1413 mv = mv + 1 1414 } 1415 o2 = ic_cat(ob, o2, "],\"idx_listed\":" as *u8); o2 = ic_catn(ob, o2, shown) 1416 o2 = ic_cat(ob, o2, "}" as *u8) 1417 if minpri < 0 { minpri = cmeta[c2 * 8 + 6] } 1418 if cmeta[c2 * 8 + 6] < minpri { minpri = cmeta[c2 * 8 + 6] } 1419 emitted = emitted + 1 1420 } } 1421 } 1422 ei = ei + 1 1423 } 1424 if minpri < 0 { minpri = 0 } 1425 1426 o2 = ic_cat(ob, o2, "],\"rows\":" as *u8); o2 = ic_catn(ob, o2, rows) 1427 o2 = ic_cat(ob, o2, ",\"rows_capped\":" as *u8); o2 = ic_catn(ob, o2, st[1]) 1428 o2 = ic_cat(ob, o2, ",\"open_rows\":" as *u8); o2 = ic_catn(ob, o2, st[2]) 1429 o2 = ic_cat(ob, o2, ",\"families\":" as *u8); o2 = ic_catn(ob, o2, families) 1430 o2 = ic_cat(ob, o2, ",\"clustered_rows\":" as *u8); o2 = ic_catn(ob, o2, clustered) 1431 o2 = ic_cat(ob, o2, ",\"noise_rows\":" as *u8); o2 = ic_catn(ob, o2, noise) 1432 o2 = ic_cat(ob, o2, ",\"emitted\":" as *u8); o2 = ic_catn(ob, o2, emitted) 1433 o2 = ic_cat(ob, o2, ",\"headline_min_priority\":" as *u8); o2 = ic_catn(ob, o2, minpri) 1434 o2 = ic_cat(ob, o2, ",\"tokens_distinct\":" as *u8); o2 = ic_catn(ob, o2, st[3]) 1435 o2 = ic_cat(ob, o2, ",\"tokens_df_skipped\":" as *u8); o2 = ic_catn(ob, o2, st[4]) 1436 o2 = ic_cat(ob, o2, ",\"rows_shape_alt\":" as *u8); o2 = ic_catn(ob, o2, st[5]) 1437 o2 = ic_cat(ob, o2, ",\"linker\":\"anchor-scope-then-organ\",\"dense_model_loaded\":" as *u8); o2 = ic_catn(ob, o2, dense_model) 1438 o2 = ic_cat(ob, o2, ",\"overlap_added\":" as *u8); o2 = ic_catn(ob, o2, overlap_added) 1439 o2 = ic_cat(ob, o2, ",\"dense_placed\":" as *u8); o2 = ic_catn(ob, o2, dense_placed) 1440 o2 = ic_cat(ob, o2, ",\"dense_min\":" as *u8); o2 = ic_catn(ob, o2, IC_DENSE_MIN) 1441 o2 = ic_cat(ob, o2, ",\"emitted_organ_anchored\":" as *u8); o2 = ic_catn(ob, o2, emitted_organ) 1442 o2 = ic_cat(ob, o2, ",\"anchors_considered\":" as *u8); o2 = ic_catn(ob, o2, ncand) 1443 // An anchor rejected for breadth is a THEME, not a family. Counted so the exclusion is never silent. 1444 o2 = ic_cat(ob, o2, ",\"anchors_too_broad\":" as *u8); o2 = ic_catn(ob, o2, anchors_too_broad) 1445 o2 = ic_cat(ob, o2, ",\"anchor_df_max\":" as *u8); o2 = ic_catn(ob, o2, anchor_dfmax) 1446 o2 = ic_cat(ob, o2, ",\"sim_join_unused\":" as *u8); o2 = ic_catn(ob, o2, simjoin) 1447 o2 = ic_cat(ob, o2, ",\"minpts\":" as *u8); o2 = ic_catn(ob, o2, IC_MINPTS) 1448 o2 = ic_cat(ob, o2, ",\"pairs_ge250\":" as *u8); o2 = ic_catn(ob, o2, st[6]) 1449 o2 = ic_cat(ob, o2, ",\"pairs_ge_join\":" as *u8); o2 = ic_catn(ob, o2, st[7]) 1450 o2 = ic_cat(ob, o2, ",\"pairs_ge500\":" as *u8); o2 = ic_catn(ob, o2, st[8]) 1451 o2 = ic_cat(ob, o2, ",\"effort_measured\":-1" as *u8) 1452 o2 = ic_cat(ob, o2, ",\"plane_declared\":" as *u8); o2 = ic_catn(ob, o2, dbf[0]) 1453 o2 = ic_cat(ob, o2, ",\"plane_found\":" as *u8); o2 = ic_catn(ob, o2, dbf[1]) 1454 var complete: i64 = 1 1455 if dbf[1] < dbf[0] { complete = 0 } 1456 o2 = ic_cat(ob, o2, ",\"coverage_complete\":" as *u8); o2 = ic_catn(ob, o2, complete) 1457 o2 = ic_cat(ob, o2, ",\"tier\":\"AICL-L2-evidence-capped\",\"tier_note\":\"L1 correlation + L2 derived summary. NOT L3+: no diagnosis is produced and none is claimed. Dense-embedding fusion (nx_recall_dense PPMI / nx_recall_fuse RRF 7/7) is a DECLARED GAP, not wired here.\"" as *u8) 1458 1459 // ── SELFTEST VERDICT. The teeth run against the clusters the real pipeline just produced. 1460 if is_self == 1 { 1461 var pass: i64 = 0 1462 var total: i64 = 0 1463 // T1 rows 0,1 share nx_mgmt_api_core -> MUST be the same family 1464 total = total + 1 1465 var t1: i64 = 0 1466 if assign[0] >= 0 { if assign[0] == assign[1] { t1 = 1; pass = pass + 1 } } 1467 // T2 ★NEGATIVE CONTROL: rows 0/1 vs 2/3 are near-identical SYMPTOM TEXT in DISJOINT organs. 1468 // They MUST NOT merge. Under anchor clustering this is structural rather than threshold-dependent: 1469 // row 2 does not CONTAIN nx_mgmt_api_core, so no setting of any parameter can put it in that 1470 // family. The tooth is kept anyway -- a guard is re-aimed, never deleted, and it now also catches 1471 // an anchor accidentally being built from a non-identifier token. 1472 total = total + 1 1473 var t2: i64 = 0 1474 // NON-VACUOUS BY CONSTRUCTION: it requires row 0 to actually BE in a family first. Written as a 1475 // bare inequality it read -1 != -1 as false when nothing clustered, i.e. the control's result was 1476 // decided by whether the clusterer ran at all rather than by what it decided. 1477 if assign[0] >= 0 { if assign[0] != assign[2] { t2 = 1; pass = pass + 1 } } 1478 // T3 rows 2,3 share nx_vault_walker_svc -> MUST be the same family 1479 total = total + 1 1480 var t3: i64 = 0 1481 if assign[2] >= 0 { if assign[2] == assign[3] { t3 = 1; pass = pass + 1 } } 1482 // T4 the unrelated row MUST be noise: it shares no identifier with anything, so it is UNASSIGNED 1483 total = total + 1 1484 var t4: i64 = 0 1485 if assign[4] < 0 { t4 = 1; pass = pass + 1 } 1486 // T5 the noise class is non-empty -- a clusterer that absorbs everything has no noise class at all 1487 total = total + 1 1488 var t5: i64 = 0 1489 if noise >= 1 { t5 = 1; pass = pass + 1 } 1490 // T6 IDF is monotonic: a token in fewer rows must outrank a token in more 1491 total = total + 1 1492 var t6: i64 = 0 1493 if ic_log2q(2) < ic_log2q(9) { if ic_log2q(9) < ic_log2q(40) { t6 = 1; pass = pass + 1 } } 1494 // T7 exactly two families were formed (mgmt pair + vault pair), not one blob and not four singletons 1495 total = total + 1 1496 var t7: i64 = 0 1497 if families == 2 { t7 = 1; pass = pass + 1 } 1498 o2 = ic_cat(ob, o2, ",\"selftest\":{\"T1_same_organ_joins\":" as *u8); o2 = ic_catn(ob, o2, t1) 1499 o2 = ic_cat(ob, o2, ",\"T2_NEGATIVE_CONTROL_disjoint_organs_stay_apart\":" as *u8); o2 = ic_catn(ob, o2, t2) 1500 o2 = ic_cat(ob, o2, ",\"T3_second_family_joins\":" as *u8); o2 = ic_catn(ob, o2, t3) 1501 o2 = ic_cat(ob, o2, ",\"T4_unrelated_row_is_noise\":" as *u8); o2 = ic_catn(ob, o2, t4) 1502 o2 = ic_cat(ob, o2, ",\"T5_noise_class_nonempty\":" as *u8); o2 = ic_catn(ob, o2, t5) 1503 o2 = ic_cat(ob, o2, ",\"T6_idf_monotonic\":" as *u8); o2 = ic_catn(ob, o2, t6) 1504 o2 = ic_cat(ob, o2, ",\"T7_exactly_two_families\":" as *u8); o2 = ic_catn(ob, o2, t7) 1505 o2 = ic_cat(ob, o2, ",\"pass\":" as *u8); o2 = ic_catn(ob, o2, pass) 1506 o2 = ic_cat(ob, o2, ",\"total\":" as *u8); o2 = ic_catn(ob, o2, total) 1507 o2 = ic_cat(ob, o2, "}" as *u8) 1508 if pass == total { o2 = ic_cat(ob, o2, ",\"verdict\":\"GREEN\"}\n" as *u8) } 1509 else { o2 = ic_cat(ob, o2, ",\"verdict\":\"RED\"}\n" as *u8) } 1510 sys_write(1, ob, o2) 1511 if pass == total { sys_exit(0); return 0 } 1512 sys_exit(1); return 1 1513 } 1514 1515 if complete == 1 { o2 = ic_cat(ob, o2, ",\"verdict\":\"COMPLETE\"" as *u8) } 1516 else { o2 = ic_cat(ob, o2, ",\"verdict\":\"PARTIAL\",\"reason\":\"plane load returned fewer rows than the plane declares -- some rows were never clustered\"" as *u8) } 1517 // EVAL. B-cubed against the hand-verified ground truth, restricted to the labeled rows -- the standard 1518 // way to score a PARTIAL gold set: judge the system's clustering AS RESTRICTED TO the items you labelled. 1519 var b3p: i64 = 0 - 1 1520 var b3r: i64 = 0 - 1 1521 var b3f: i64 = 0 - 1 1522 var gt_n: i64 = 0 1523 var gt_f: i64 = 0 1524 var gt_c: i64 = 0 1525 var do_eval: i64 = 0 1526 if mode == IC_MODE_EVAL { do_eval = 1 } 1527 // The PAGE scores itself too: a dashboard that cannot state its own accuracy is asking to be trusted 1528 // rather than checked, and this one is published where nobody can see the gate. 1529 if mode == IC_MODE_PAGE { do_eval = 1 } 1530 if do_eval == 1 { 1531 let gtid: *i64 = sys_mmap(8 * IC_GTMAX) as *i64 1532 let gtfam: *i64 = sys_mmap(8 * IC_GTMAX) as *i64 1533 let gtrow: *i64 = sys_mmap(8 * IC_GTMAX) as *i64 1534 let gtn: i64 = ic_gt_load(gtid, gtfam) 1535 var gfound: i64 = 0 1536 var gi: i64 = 0 1537 while gi < gtn { 1538 gtrow[gi] = 0 - 1 1539 var rr2: i64 = 0 1540 while rr2 < rows { 1541 if rmeta[rr2 * 8 + 0] == gtid[gi] { gtrow[gi] = rr2; rr2 = rows } else { rr2 = rr2 + 1 } 1542 } 1543 if gtrow[gi] >= 0 { gfound = gfound + 1 } 1544 gi = gi + 1 1545 } 1546 // NON-VACUITY: a ruler that resolved none of its own gold rows must NOT report a score. Without this 1547 // a re-seeded plane would silently yield 0/0 and print a confident-looking zero. 1548 if gfound == 0 { if mode == IC_MODE_EVAL { 1549 ic_puts("{\"organ\":\"nx_debtcluster\",\"verb\":\"eval\",\"verdict\":\"INSTRUMENT-BLIND\",\"reason\":\"ZERO ground-truth rows resolved in the plane -- nothing was compared, so NO quality score is emitted\"}\n" as *u8) 1550 sys_exit(2); return 2 1551 } } 1552 var psum: i64 = 0 1553 var rsum: i64 = 0 1554 var counted: i64 = 0 1555 var ai: i64 = 0 1556 while ai < gtn { 1557 if gtrow[ai] >= 0 { 1558 let ra: i64 = gtrow[ai] 1559 var inter: i64 = 0 1560 var ssize: i64 = 0 1561 var gsize: i64 = 0 1562 var bj: i64 = 0 1563 while bj < gtn { 1564 if gtrow[bj] >= 0 { 1565 let rb: i64 = gtrow[bj] 1566 var same_sys: i64 = 0 1567 if ic_share_fam(assign, assign2, ra, rb) == 1 { same_sys = 1 } 1568 if ra == rb { same_sys = 1 } 1569 var same_gt: i64 = 0 1570 if gtfam[ai] == gtfam[bj] { same_gt = 1 } 1571 if same_sys == 1 { ssize = ssize + 1 } 1572 if same_gt == 1 { gsize = gsize + 1 } 1573 if same_sys == 1 { if same_gt == 1 { inter = inter + 1 } } 1574 } 1575 bj = bj + 1 1576 } 1577 if ssize > 0 { psum = psum + (inter * 1000) / ssize } 1578 if gsize > 0 { rsum = rsum + (inter * 1000) / gsize } 1579 counted = counted + 1 1580 } 1581 ai = ai + 1 1582 } 1583 // b3p/b3r stay -1 when nothing was counted, so an unscored run can never print a confident zero 1584 if counted > 0 { b3p = psum / counted } 1585 if counted > 0 { b3r = rsum / counted } 1586 if b3p + b3r > 0 { b3f = (2 * b3p * b3r) / (b3p + b3r) } 1587 // SPLIT DIAGNOSTIC -- the loss is here, not in the noise pile (dense arm proved that). 1588 // NOTE: rows left unassigned all share assign=-1, so a family fully in noise counts as ONE 1589 // bucket, not N. That UNDERSTATES splitting; it can never overstate it. 1590 var fams_split: i64 = 0 1591 var worst_split: i64 = 0 1592 var gi2: i64 = 0 1593 while gi2 < gtn { 1594 var firstf: i64 = 1 1595 var gk: i64 = 0 1596 while gk < gi2 { if gtfam[gk] == gtfam[gi2] { firstf = 0 } gk = gk + 1 } 1597 if firstf == 1 { 1598 var dsys: i64 = 0 1599 var gm: i64 = 0 1600 while gm < gtn { 1601 if gtfam[gm] == gtfam[gi2] { if gtrow[gm] >= 0 { 1602 let sv: i64 = assign[gtrow[gm]] 1603 var seen2: i64 = 0 1604 var gn2: i64 = 0 1605 while gn2 < gm { 1606 if gtfam[gn2] == gtfam[gi2] { if gtrow[gn2] >= 0 { if assign[gtrow[gn2]] == sv { seen2 = 1 } } } 1607 gn2 = gn2 + 1 1608 } 1609 if seen2 == 0 { dsys = dsys + 1 } 1610 } } 1611 gm = gm + 1 1612 } 1613 if dsys > 1 { fams_split = fams_split + 1 } 1614 if dsys > worst_split { worst_split = dsys } 1615 } 1616 gi2 = gi2 + 1 1617 } 1618 gt_n = gtn 1619 gt_f = gfound 1620 // distinct gold families, COUNTED not declared -- the page states this number publicly and a 1621 // hardcoded one silently goes stale the moment the gold set grows (it did: said 5, was 12). 1622 var dz: i64 = 0 1623 while dz < gtn { 1624 if gtrow[dz] >= 0 { 1625 var seenf: i64 = 0 1626 var dw: i64 = 0 1627 while dw < dz { 1628 if gtrow[dw] >= 0 { if gtfam[dw] == gtfam[dz] { seenf = 1 } } 1629 dw = dw + 1 1630 } 1631 if seenf == 0 { gt_c = gt_c + 1 } 1632 } 1633 dz = dz + 1 1634 } 1635 o2 = ic_cat(ob, o2, ",\"eval\":{\"metric\":\"B-cubed per-element, chosen over ARI/NMI because family sizes span 152 rows down to 2 and pair-counting metrics are dominated by the largest cluster\",\"gt_declared\":" as *u8) 1636 o2 = ic_catn(ob, o2, gtn) 1637 o2 = ic_cat(ob, o2, ",\"gt_found\":" as *u8); o2 = ic_catn(ob, o2, gfound) 1638 o2 = ic_cat(ob, o2, ",\"gt_families\":" as *u8); o2 = ic_catn(ob, o2, gt_c) 1639 o2 = ic_cat(ob, o2, ",\"b3_precision_permil\":" as *u8); o2 = ic_catn(ob, o2, b3p) 1640 o2 = ic_cat(ob, o2, ",\"b3_recall_permil\":" as *u8); o2 = ic_catn(ob, o2, b3r) 1641 o2 = ic_cat(ob, o2, ",\"b3_f1_permil\":" as *u8); o2 = ic_catn(ob, o2, b3f) 1642 o2 = ic_cat(ob, o2, ",\"gold_families_split\":" as *u8); o2 = ic_catn(ob, o2, fams_split) 1643 o2 = ic_cat(ob, o2, ",\"worst_split_buckets\":" as *u8); o2 = ic_catn(ob, o2, worst_split) 1644 o2 = ic_cat(ob, o2, ",\"note\":\"A FLOOR, NOT A CORPUS SCORE -- see gt_declared/gt_found above for the exact denominator, against the whole plane. PRECISION asks whether rows grouped together really belong together; RECALL asks how much of each known family was found. HIGH PRECISION WITH LOW RECALL IS THE EXPECTED SIGNATURE of identifier-anchored grouping and is the EVIDENCE for adding the dense/PPMI layer, not a number to tune away.\"}" as *u8) 1645 } 1646 o2 = ic_cat(ob, o2, ",\"note\":\"A FAMILY IS A CLAIM, NOT A VERDICT. It says these rows describe one problem and that one act may close several. Confirm by reading the members before scheduling. noise_rows is the count that REFUSED to cluster and is never folded into a family. effort_measured=-1 because the plane records file-time, not close-time -- it is NOT zero and is NOT in the priority product.\"}\n" as *u8) 1647 1648 // ── PAGE. Written temp-then-rename so a reader never sees a half-page, and so a failed render leaves 1649 // the previous good page in place rather than truncating it to nothing. 1650 if mode == IC_MODE_PAGE { 1651 let pbuf: *u8 = sys_mmap(IC_MAGIC_16384) 1652 let tmpp: *u8 = sys_mmap(IC_MAGIC_1024) 1653 var tp: i64 = ic_cat(tmpp, 0, outfile) 1654 tp = ic_cat(tmpp, tp, ".tmp" as *u8) 1655 tmpp[tp] = 0 as u8 1656 let fd: i64 = sys_openat_wr(tmpp, IC_WRMODE) 1657 if fd < 0 { 1658 ic_puts("{\"organ\":\"nx_debtcluster\",\"verdict\":\"PAGE-WRITE-FAILED\",\"reason\":\"cannot open the temp page for write -- the PREVIOUS page is left intact, nothing was truncated\",\"path\":\"" as *u8) 1659 ic_puts(tmpp) 1660 ic_puts("\"}\n" as *u8) 1661 sys_exit(3); return 3 1662 } 1663 // A FULL document, not a fragment. The /code pages are complete self-contained HTML with an inline 1664 // design system; a fragment dropped into that tree serves 200 and renders UNSTYLED, which reads as 1665 // broken rather than as new. Variables mirror nx_codewiki's palette (light + dark) so this page is 1666 // native to the wiki rather than bolted onto it. 1667 var po: i64 = 0 1668 po = ic_cat(pbuf, po, "<!doctype html>\n<html lang='en'><head><meta charset='utf-8'><meta name='viewport' content='width=device-width,initial-scale=1'>\n<title>work families - nishi code wiki</title>\n<style>\n" as *u8) 1669 po = ic_cat(pbuf, po, ":root{--bg:#ffffff;--sf:#f6f8fb;--sf2:#eef1f6;--br:#dfe4ec;--ink:#232936;--mut:#68748a;--acc:#1a5dc8;--acc2:#2e8555;--hd:#111726;--hov:#f0f4fa}\n" as *u8) 1670 po = ic_cat(pbuf, po, "@media(prefers-color-scheme:dark){:root{--bg:#0d1017;--sf:#131826;--sf2:#1a2132;--br:#252d40;--ink:#dbe2ec;--mut:#8a94a8;--acc:#5aa7f0;--acc2:#86c793;--hd:#eef2f8;--hov:#141a28}}\n" as *u8) 1671 po = ic_cat(pbuf, po, "*{box-sizing:border-box}body{font-family:system-ui,'Segoe UI',sans-serif;margin:0;background:var(--bg);color:var(--ink);display:grid;grid-template-columns:252px minmax(0,1fr);min-height:100vh}\n" as *u8) 1672 po = ic_cat(pbuf, po, "a{color:var(--acc);text-decoration:none}a:hover{text-decoration:underline}\n" as *u8) 1673 po = ic_cat(pbuf, po, "nav.sb{background:var(--sf);border-right:1px solid var(--br);padding:14px 12px;position:sticky;top:0;height:100vh;overflow-y:auto;font-size:.9em}\n" as *u8) 1674 po = ic_cat(pbuf, po, ".sb .brand{display:block;font-weight:700;color:var(--hd);font-size:1.02em;margin:2px 6px 12px}\n" as *u8) 1675 po = ic_cat(pbuf, po, ".sb a.it{display:block;padding:4px 8px;border-radius:6px;color:var(--ink)}.sb a.it:hover{background:var(--sf2);text-decoration:none;color:var(--hd)}\n" as *u8) 1676 po = ic_cat(pbuf, po, ".sb .sec{margin:14px 6px 4px;font-size:.74em;letter-spacing:.09em;text-transform:uppercase;color:var(--mut)}\n" as *u8) 1677 po = ic_cat(pbuf, po, "main{padding:22px 30px 40px;max-width:1080px;min-width:0}\n" as *u8) 1678 po = ic_cat(pbuf, po, ".crumb{font-size:.85em;color:var(--mut);margin-bottom:10px}.crumb a{color:var(--mut)}\n" as *u8) 1679 po = ic_cat(pbuf, po, "h1{font-size:1.55em;color:var(--hd);margin:2px 0 6px;letter-spacing:-.01em}\n" as *u8) 1680 po = ic_cat(pbuf, po, "code{font-family:ui-monospace,'Cascadia Code',Consolas,monospace;font-size:.86em}\n" as *u8) 1681 po = ic_cat(pbuf, po, "p.muted,.muted{color:var(--mut)}p{line-height:1.5;max-width:78ch}\n" as *u8) 1682 po = ic_cat(pbuf, po, "table{border-collapse:collapse;width:100%;margin:14px 0;font-size:.9em}\n" as *u8) 1683 po = ic_cat(pbuf, po, "th{text-align:left;color:var(--mut);font-weight:600;font-size:.82em;letter-spacing:.04em;text-transform:uppercase;border-bottom:1px solid var(--br);padding:6px 9px}\n" as *u8) 1684 po = ic_cat(pbuf, po, "td{border-bottom:1px solid var(--br);padding:7px 9px;vertical-align:top}tr:hover td{background:var(--hov)}\n" as *u8) 1685 po = ic_cat(pbuf, po, "th.n,td.n{text-align:right;font-variant-numeric:tabular-nums;white-space:nowrap}\n" as *u8) 1686 po = ic_cat(pbuf, po, "td.idx{color:var(--mut);font-family:ui-monospace,Consolas,monospace;font-size:.82em}\n" as *u8) 1687 po = ic_cat(pbuf, po, "tr:first-child td{}\ntbody tr:nth-child(1) td{font-weight:500}\n" as *u8) 1688 po = ic_cat(pbuf, po, "@media(max-width:820px){body{display:block}nav.sb{position:static;height:auto;border-right:0;border-bottom:1px solid var(--br)}}\n" as *u8) 1689 po = ic_cat(pbuf, po, "</style></head><body>\n" as *u8) 1690 po = ic_cat(pbuf, po, "<nav class='sb'><a class='brand' href='/code/'>nishi code wiki</a>\n<div class='sec'>backlog</div>\n" as *u8) 1691 po = ic_cat(pbuf, po, "<a class='it' href='/code/families'>Work families</a>\n<a class='it' href='/code/findings'>Findings</a>\n<a class='it' href='/code/debt'>Debt taxonomy</a>\n<a class='it' href='/code/learnings'>Learnings</a>\n" as *u8) 1692 po = ic_cat(pbuf, po, "<div class='sec'>wiki</div>\n<a class='it' href='/code/'>Overview</a>\n<a class='it' href='/code/search'>Search</a>\n<a class='it' href='/code/ask'>Ask</a>\n</nav>\n<main>\n" as *u8) 1693 po = ic_cat(pbuf, po, "<div class='crumb'><a href='/code/'>code wiki</a> / work families</div>\n" as *u8) 1694 po = ic_cat(pbuf, po, "<h1>work families &mdash; what the debt ledger is actually asking for</h1>\n" as *u8) 1695 po = ic_cat(pbuf, po, "<p class='muted'>The ledger is a flat, append-only list sorted by severity, so it answers <em>what screams loudest</em> and never <em>what should I build</em>. This page groups rows that describe ONE problem, so a seat can pick the act that closes the most rows instead of the loudest single row. A family is the set of rows that all literally name the same thing &mdash; the <em>anchor</em>. Membership is a property each row holds on its own, so unrelated rows can never be chained together through a third, and the family's name is the anchor rather than a guess about it.</p>\n" as *u8) 1696 po = ic_cat(pbuf, po, "<table><tr><th class='n'>#</th><th>family</th><th class='n'>rows</th><th class='n'>open</th><th class='n'>sev</th><th class='n'>scopes</th><th class='n'>days</th><th class='n'>priority</th><th>row idx</th></tr>\n" as *u8) 1697 sys_write(fd, pbuf, po) 1698 sys_write(fd, hb, ho) 1699 po = 0 1700 po = ic_cat(pbuf, po, "</table>\n<p class='muted'>priority = max_sev &times; open_rows. Both terms are measured from the plane. distinct_scopes was removed from the product because under anchor grouping it rewards a GENERIC anchor touching many lanes over a specific one, which is backwards for deciding what to build; it is shown as context instead. <strong>Effort is deliberately absent</strong>: the plane records when a row was FILED, never when it was CLOSED, so an effort term here would be invented rather than measured.</p>\n" as *u8) 1701 po = ic_cat(pbuf, po, "<p class='muted'>Coverage: <strong>" as *u8); po = ic_catn(pbuf, po, rows) 1702 po = ic_cat(pbuf, po, "</strong> rows scanned, <strong>" as *u8); po = ic_catn(pbuf, po, st[2]) 1703 po = ic_cat(pbuf, po, "</strong> open, <strong>" as *u8); po = ic_catn(pbuf, po, families) 1704 po = ic_cat(pbuf, po, "</strong> families found covering <strong>" as *u8); po = ic_catn(pbuf, po, clustered) 1705 po = ic_cat(pbuf, po, "</strong> rows, and <strong>" as *u8); po = ic_catn(pbuf, po, noise) 1706 po = ic_cat(pbuf, po, "</strong> rows that REFUSED to cluster (reported, never folded into a family &mdash; most debt genuinely is singletons). Showing the top " as *u8); po = ic_catn(pbuf, po, emitted) 1707 po = ic_cat(pbuf, po, "; a recommendation list longer than 5&ndash;8 is a list again. Grouped by shared anchor over " as *u8); po = ic_catn(pbuf, po, ncand) 1708 po = ic_cat(pbuf, po, " candidate anchors, of which " as *u8); po = ic_catn(pbuf, po, anchors_too_broad) 1709 po = ic_cat(pbuf, po, " were rejected as too broad to name a work item" as *u8) 1710 po = ic_cat(pbuf, po, ".</p>\n" as *u8) 1711 if b3p >= 0 { 1712 po = ic_cat(pbuf, po, "<p class='muted'><strong>Measured quality of this grouping.</strong> B-cubed against " as *u8); po = ic_catn(pbuf, po, gt_f) 1713 po = ic_cat(pbuf, po, " hand-read rows across " as *u8); po = ic_catn(pbuf, po, gt_c) 1714 po = ic_cat(pbuf, po, " known families: precision <strong>" as *u8); po = ic_catn(pbuf, po, b3p) 1715 po = ic_cat(pbuf, po, "&permil;</strong>, recall <strong>" as *u8); po = ic_catn(pbuf, po, b3r) 1716 po = ic_cat(pbuf, po, "&permil;</strong>, F1 <strong>" as *u8); po = ic_catn(pbuf, po, b3f) 1717 po = ic_cat(pbuf, po, "&permil;</strong>. This is a FLOOR, not a corpus score &mdash; " as *u8); po = ic_catn(pbuf, po, gt_n) 1718 po = ic_cat(pbuf, po, " labelled rows against the whole plane. High precision with lower recall is the expected signature of grouping by shared identifier: what it groups does belong together, but it misses members that describe the same problem in different words.</p>\n" as *u8) 1719 } 1720 if complete == 1 { po = ic_cat(pbuf, po, "<p class='muted'>plane_declared == plane_found, so every row the plane declares was examined.</p>\n" as *u8) } 1721 else { po = ic_cat(pbuf, po, "<p><strong>PARTIAL:</strong> the plane load returned fewer rows than the plane declares &mdash; some rows were never clustered, and this page is a sample, not the set.</p>\n" as *u8) } 1722 po = ic_cat(pbuf, po, "<p class='muted'><strong>A family is a claim, not a verdict.</strong> It says these rows look like one problem and that one act may close several &mdash; confirm by reading the members before scheduling. Generated by <code>nx_debtcluster page</code> (AICL L2, evidence-capped: correlation and summary only, no diagnosis is produced and none is claimed).</p>\n" as *u8) 1723 po = ic_cat(pbuf, po, "</main></body></html>\n" as *u8) 1724 sys_write(fd, pbuf, po) 1725 sys_close(fd) 1726 sys_renameat(tmpp, outfile) 1727 } 1728 1729 sys_write(1, ob, o2) 1730 sys_exit(0); return 0 1731}