nx_debtcluster.nx source

1// nx_debtcluster.nx -- THE LEDGER STOPS BEING A LIST AND BECOMES A RECOMMENDATION. 2// license_tier: ORIGINAL No hw writes (Rule 26). expect_exit: 0 3// 4// ─── THE DEFECT THIS EXISTS TO KILL ──────────────────────────────────────────────────────────────────── 5// MEASURED 2026-07-31: the debt plane holds 2068 rows, 905 open, 281 open at sev>=8, and it is FLAT. Every 6// seat reads it top-down by severity and picks a row. But the rows are not independent -- in the first FORTY 7// sev>=8 rows alone there are at least five families, each one incident wearing many row numbers: 8// mgmt_api availability seq698 crash-loop · seq710 hang · the live sev-9 :18098 outage 9// resource exhaustion seq883 23.6GiB alloc-without-free · seq774 20,440 respawns · seq783 · seq727 · seq975 10// dup-source seq207 · seq701 · seq945 · seq965 · seq974 (965/974 report GREEN divergent=0 11// WHILE THE PARENTS STAY OPEN) 12// law-warden magic nums seq274 L001 (_hdl_build, 2348) · seq315 L006 (core, 3476) -- same rule, same 13// detector, two corpora, filed as two unrelated problems 14// gx-body seq804·805·808·819·820·824·826·828·900 = NINE rows, ONE skull-architecture arc 15// A seat that fixes the top row by severity fixes ONE MEMBER of a family and leaves the family. The ledger 16// is sorted by how loud a row is, never by how much work a single act would close. 17// ★LAW: A LEDGER SORTED BY SEVERITY ANSWERS "WHAT SCREAMS LOUDEST", NEVER "WHAT SHOULD I BUILD". 18// 19// ─── WHAT SOTA SAYS (researched 2026-07-31, and what we deliberately DID NOT copy) ───────────────────── 20// 1. NORMALIZE BEFORE YOU EMBED. LLM-augmented ticket clustering (FLAIRS 2026, ABCD) beats raw-text 21// embedding on silhouette AND human-rated coherence purely by extracting a canonical statement first. 22// GPTrace (ICSE'26, arXiv 2512.01609) does the same for crash dedup: normalize -> embed -> HDBSCAN, 23// scored on silhouette + DBCV + P/R/F1, beating CrashWalk and Igor. 24// 2. DENSITY-BASED, NOT PARTITIONAL. HDBSCAN is chosen in that literature precisely because it DECLINES to 25// cluster outliers. A debt ledger is mostly singletons; a partitional method (k-means, plain 26// agglomerative) has nowhere to put them and pollutes every cluster with noise. We implement the 27// principle, not the library: a core-density test with an EXPLICIT noise class that is always reported. 28// 3. TOPOLOGY BEATS TEXT SIMILARITY FOR CAUSALITY. TopoEvo (arXiv 2605.15611) is unambiguous -- correlation 29// is constrained by the ACTUAL dependency graph, and topology-agnostic baselines manufacture spurious 30// links by treating all relations as equivalent. TWO ROWS BOTH SAYING "LEAK" IN UNCONNECTED ORGANS ARE 31// TWO PROBLEMS. This is the ic_ident_gate below, and it is the single most important line in the organ. 32// 4. RANK BY RISK-REDUCTION OVER EFFORT, and cap the recommendation list at 5-8 (postmortem practice). 33// 5. ★DO NOT BUILD THE AGENTIC RCA TIER. ITBench-AA (Artificial Analysis + IBM Research, launched 34// 2026-05-27, 59 live Kubernetes SRE tasks): EVERY frontier model scores below 50% -- Claude Opus 4.7 35// 47%, GPT-5.5 46%; IBM's own ITBench reports best-agent 13.8% SRE / 25.2% CISO / 0% FinOps. Meanwhile 36// the plain correlate-and-rank layer delivers a reliable 80-85% noise reduction in production. So we 37// build the deterministic layer that demonstrably works and refuse the one that demonstrably does not. 38// 6. GRADE ORDINALLY. Arvo's AI Investigation Capability Ladder (May 2026, upd. July 2026): L0 manual · 39// L1 correlation · L2 summarized timeline · L3 single-shot diagnosis · L4 agentic · L5 closed-loop. 40// THIS ORGAN IS L2, EVIDENCE-CAPPED, AND SAYS SO IN ITS OWN OUTPUT. Claiming L4 would be the banked 41// coverage-gaming defect verbatim. 42// 43// ─── WHY LEXICAL AND NOT THE DENSE MODEL (a deliberate choice, not a shortcut) ────────────────────────── 44// nx_recall_dense runs a real 41MB PPMI model (vocab 102,318 / 2,558,014 triples, integer late-interaction, 45// deterministic). It is available and it is the obvious next rung. It is NOT rung one, because this corpus 46// is identifier-dense: the signal that says two rows are the same problem is the literal token 47// `nx_mgmt_api`, not a distributional neighbourhood. A general-text embedder DILUTES an exact identifier 48// match into a similarity score; IDF-weighted exact matching preserves it. This is the same reason BM25 49// stays competitive on technical corpora and the reason hybrid RRF fusion exists at all. The dense axis is 50// named in the output as a declared GAP (nx_recall_fuse is gate 7/7, nDCG 494->659, not yet in serve) so 51// the tier can never be read as complete. 52// 53// ─── NON-VACUITY: THIS INSTRUMENT CAN FAIL, AND IT PROVES IT ON ITSELF ───────────────────────────────── 54// `selftest` runs the REAL pipeline over planted fixtures. The decisive tooth is T2, a NEGATIVE CONTROL: 55// two rows written to be lexically near-identical ("memory grows unbounded on every call, never freed") 56// but sited in DISJOINT organs. A text-only clusterer merges them -- that is the spurious correlation 57// TopoEvo names. This organ MUST keep them apart, and selftest FAILS if it does not. A clusterer that 58// cannot refuse a merge is not measuring anything. 59// Further teeth: zero rows loaded -> INSTRUMENT-BLIND exit 2 (never "no clusters found") · plane short-read 60// -> coverage_complete=0 and verdict PARTIAL · noise_rows and the row denominator are ALWAYS printed 61// (A SAMPLE FILED WITHOUT ITS DENOMINATOR IS A LIE) · the headline is the MIN emitted priority, never the 62// mean (HEADLINE=MIN law) · effort is reported as -1 UNMEASURED and is NEVER folded into the score, because 63// the plane records file-time and not close-time so effort is genuinely not derivable here (-1=UNMEASURED 64// never the min, per nx_capaxes). 65// 66// nx_debtcluster run [prefix] [sim_join_permil] cluster the live plane -> ranked JSON 67// nx_debtcluster selftest planted fixtures incl. the negative control 68// ONE import only -- nx_store_seed_lib.nx already pulls in nx_seg_store.nx and nx_syscalls.nx, so naming 69// them again is redundant. That redundancy is hygiene, NOT the cause of anything: an earlier note here 70// claimed siblings naming all three would fail to build, and that claim was REFUTED by measurement -- 71// dropping the extra imports moved the duplicate-definition error's line numbers by exactly the number of 72// comment lines added, which proved the duplicate sts_lock was INTRINSIC to nx_store_seed_lib.nx (defined 73// at both line 40 and line 377) rather than an expansion artifact. Fixed at the library; see debt 1785525535. 74import "nx_store_seed_lib.nx" 75import "nx_ppmi_lib.nx" 76const IC_MAGIC_1784947125: i64 = 1784947125 77const IC_MAGIC_1784947134: i64 = 1784947134 78const IC_MAGIC_1784947786: i64 = 1784947786 79const IC_MAGIC_1784951793: i64 = 1784951793 80const IC_MAGIC_1784952157: i64 = 1784952157 81const IC_MAGIC_1784954507: i64 = 1784954507 82const IC_MAGIC_1784955224: i64 = 1784955224 83const IC_MAGIC_1784955875: i64 = 1784955875 84const IC_MAGIC_1784956205: i64 = 1784956205 85const IC_MAGIC_1784995126: i64 = 1784995126 86const IC_MAGIC_1784995721: i64 = 1784995721 87const IC_MAGIC_1784995947: i64 = 1784995947 88const IC_MAGIC_1784996406: i64 = 1784996406 89const IC_MAGIC_1784996558: i64 = 1784996558 90const IC_MAGIC_1785000800: i64 = 1785000800 91const IC_MAGIC_1784925304: i64 = 1784925304 92const IC_MAGIC_1784927702: i64 = 1784927702 93const IC_MAGIC_1784958766: i64 = 1784958766 94const IC_MAGIC_1784978119: i64 = 1784978119 95const IC_MAGIC_1784999221: i64 = 1784999221 96const IC_MAGIC_1785001189: i64 = 1785001189 97const IC_MAGIC_1785002038: i64 = 1785002038 98const IC_MAGIC_1024: i64 = 1024 99const IC_MAGIC_1784856230: i64 = 1784856230 100const IC_MAGIC_1784857155: i64 = 1784857155 101const IC_MAGIC_1784480242: i64 = 1784480242 102const IC_MAGIC_1784856346: i64 = 1784856346 103const IC_MAGIC_1785030820: i64 = 1785030820 104const IC_MAGIC_1785031111: i64 = 1785031111 105const IC_MAGIC_1784995429: i64 = 1784995429 106const IC_MAGIC_1784995511: i64 = 1784995511 107const IC_MAGIC_1784996221: i64 = 1784996221 108const IC_MAGIC_1785002845: i64 = 1785002845 109const IC_MAGIC_1785002909: i64 = 1785002909 110const IC_MAGIC_1785003511: i64 = 1785003511 111const IC_MAGIC_1785003779: i64 = 1785003779 112const IC_MAGIC_1785003935: i64 = 1785003935 113const IC_MAGIC_1785027945: i64 = 1785027945 114const IC_MAGIC_1784661149: i64 = 1784661149 115const IC_MAGIC_1784661346: i64 = 1784661346 116const IC_MAGIC_1784664001: i64 = 1784664001 117const IC_MAGIC_1785027242: i64 = 1785027242 118const IC_MAGIC_1784908046: i64 = 1784908046 119const IC_MAGIC_2048: i64 = 2048 120const IC_MAGIC_16384: i64 = 16384 121 122// Plane read cap. MIRRORS nx_debt.nx's DB_CAP exactly. Reading the same plane with a smaller buffer is how 123// an instrument silently sees a prefix of the truth (the nx_debt_view 512KB blindness, 734 rows of 1835). 124const IC_CAP: i64 = 4194304 125// Row headroom. Plane is ~2068 and grows ~500/wk (nx_debtlive's own measurement), so 8192 is ~12 weeks. 126// Rows beyond this are COUNTED into rows_capped and reported, never silently dropped. 127const IC_MAXROWS: i64 = 8192 128const IC_MAXTOK: i64 = 48 // tokens retained per row; long rows are informative in the head 129const IC_DICT: i64 = 65536 // open-addressed dict slots; ~8x expected distinct tokens keeps probes short 130const IC_POSTCAP: i64 = 393216 // IC_MAXROWS * IC_MAXTOK 131const IC_TOKMIN: i64 = 3 // 1-2 char tokens carry no discriminative signal 132const IC_TOKMAX: i64 = 64 133const IC_HMOD: i64 = 1000003 // keeps the rolling hash bounded and positive (no overflow reliance) 134// A token appearing in more than this share of rows has near-zero IDF and quadratically inflates the 135// postings scan. Skipped tokens are COUNTED and reported (tokens_df_skipped) -- this is why no stopword 136// list exists in this organ: stopwords are DATA-DRIVEN here, not a hardcoded table (rule 11). 137const IC_DF_PERMIL: i64 = 125 138// Default link threshold in permil of IDF-weighted overlap. Overridable as argv[3] and ALWAYS echoed in the 139// envelope, and the pair-count bands below it are reported so the choice is auditable rather than asserted. 140const IC_SIM_JOIN: i64 = 380 141const IC_BAND_LO: i64 = 250 142const IC_BAND_HI: i64 = 500 143// Density: a row must have at least this many GATE-QUALIFYING neighbours (over the similarity threshold AND 144// sharing an identifier) to be a CORE row. Rows that are not core cannot seed a cluster -- this is the 145// HDBSCAN property that makes a noise class possible. 146// ★WHY 1 AND NOT THE LITERATURE'S 2+: minPts must match the DOMAIN'S cluster-size distribution, and a debt 147// ledger's families START AT TWO -- {seq698 mgmt crash-loop, seq710 mgmt hang} is a complete, actionable 148// family. minPts=2 requires >=2 neighbours and therefore >=3 members, so it CANNOT form a pair and silently 149// discards every two-row family as noise. That is not a stricter instrument, it is a blind one: measured 150// here, minPts=2 returned families=0 / noise=5 on a fixture containing two obvious pairs. 151// The noise class does NOT depend on this number -- it is the IDENTIFIER GATE that refuses merges, and the 152// negative control (T2) is what proves the gate still bites at minPts=1. Loosening this without T2 surviving 153// would be tuning until green, which is why T2 is a hard failure and not a warning. 154const IC_MINPTS: i64 = 1 155// ─── ANCHOR CLUSTERING (replaced transitive similarity 2026-07-31, ON MEASUREMENT) ───────────────────── 156// The first implementation linked rows by IDF-weighted text similarity and took the transitive closure. 157// It CHAINED: at sim=250 the top family had 44 members spanning FIFTEEN scopes and the second had 20 across 158// EIGHTEEN, because A~B and B~C merges A,B,C even when A and C share nothing at all. Raising the threshold 159// to stop that just traded the chaining for silence (families 231->104, noise 1373->1747). There is no 160// threshold that fixes single linkage, because the defect is the TRANSITIVITY, not the cutoff. 161// ★THE REPLACEMENT: a family is the set of rows that literally contain ONE shared identifier -- the anchor. 162// Chaining is then impossible BY CONSTRUCTION (membership is a property of each row on its own, never of a 163// path through other rows), and the label stops being guesswork because THE ANCHOR IS THE NAME. 164// An anchor present in more than this share of rows is a THEME, not a family ("buildroot" is not a work 165// item); excluded anchors are COUNTED and reported so the exclusion is never silent. 166const IC_ANCHOR_DF_PERMIL: i64 = 80 167const IC_MINFAM: i64 = 2 168// Two families merge only when this share of the SMALLER family's rows each independently carry the other 169// family's anchor. 500 = a strict majority. This is the guard that separates evidence-based merging from 170// the transitive closure that chained in v1: one bridging row cannot reach a majority, ever. 171// DENSE (SEMANTIC) ARM. The merge pass proved a second LEXICAL signal only adds noise; the remaining 172// recall has to come from a signal that understands WORDS THAT DIFFER. ppl_maxsim_idf is integer 173// late-interaction maxsim over the 43MB PPMI model -- deterministic, no float. 174// SCOPE IS DELIBERATELY NARROW: it only places rows the anchor pass left UNASSIGNED. It never merges, 175// splits or re-labels an existing family, so recall can only rise and the precision risk is bounded to 176// exactly the rows it places. That is the discipline the merge pass lacked. 177// DENSE ARM DISABLED 2026-08-01 -- BUILT, LOADED, MEASURED, REFUTED. Kept as the record. 178// It worked mechanically (dense_model_loaded=1, dense_placed=248 of ~248 noise rows) and was 179// STRICTLY HARMFUL: B-cubed precision 900 -> 828, recall 628 -> 628 (NOT ONE POINT), F1 739 -> 714, 180// and the GATE WENT RED -- T4/T5 failed because it swallowed the deliberately-unrelated fixture row, 181// destroying the noise class outright. At threshold 700 it placed essentially EVERY noise row. 182// THE DIAGNOSIS THAT MATTERS, AND IT KILLS MY SCOPING ASSUMPTION: recall did not move AT ALL, which 183// proves the missing recall is NOT in unassigned rows. Every gold row was already assigned -- the 184// loss is gold-family members SPLIT ACROSS DIFFERENT families. Placing noise could never have fixed 185// it, so this arm was aimed at the wrong defect from the start. 186// LAW: BEFORE ADDING A SIGNAL, PROVE WHERE THE LOSS ACTUALLY IS -- I ASSUMED IT WAS THE NOISE PILE 187// AND SPENT A WHOLE RUNG THERE. The next attempt must target SPLIT families, not unassigned rows. 188// Does the ORGAN tier drive CLAIMING, or only labelling? Organ-first raised emitted_organ_anchored 189// 4->7 but recall sat at 628 and 4 of 5 multi-row gold families stayed split. This flag isolates the 190// grouping half so the two jobs can be measured apart instead of assumed together. 191const IC_OVERLAP: i64 = 1 192const IC_ORGAN_CLAIM: i64 = 0 193const IC_SCOPE_FIRST: i64 = 0 194const IC_DENSE_ENABLED: i64 = 0 195const IC_DENSE_MIN: i64 = 700 196const IC_DENSE_TOPF: i64 = 64 197const IC_MERGE_PERMIL: i64 = 500 198// MERGE PASS DISABLED 2026-07-31 -- BUILT, MEASURED, REFUTED, KEPT AS THE RECORD SO NOBODY REBUILDS IT. 199// Hypothesis: anchors are one blocking strategy, so merging families on majority co-occurrence should 200// recover the recall the ruler measured as missing (P840/R635/F1 723). 201// RESULT: B-cubed came back BIT-IDENTICAL -- 840/635/723, not one point of gain -- while the visible 202// output degraded badly: rank 1 became 'nx_connect_appd' with 161 members swallowing the stale-mirror 203// family, and ranks 2-8 turned into junk phrase anchors (MID-SESSION, OO-CONSOLIDATION, API-pure, 204// build-deploy). Families ballooned 152->161 and 87->115 while naming nothing. 205// TWO LESSONS, both banked: 206// 1. MAJORITY CO-OCCURRENCE IS NOT EVIDENCE OF SAME-SUBJECT. Rows about one organ naturally quote the 207// same handful of ecosystem nouns, so the majority test passes for families that merely share a 208// vocabulary. The anchor was already the right discriminator; a second lexical signal adds noise. 209// 2. THE RULER WAS FLAT WHILE QUALITY VISIBLY FELL. A 20-row gold set over 5 families cannot see a 210// regression that is obvious by eye -- so the gold set is UNDER-POWERED for this class, and that is a 211// finding about the instrument, not a reason to trust the change. 212// LAW: A RULER THAT CANNOT MOVE WHEN THE OUTPUT VISIBLY WORSENS IS NOT YET A RULER FOR THAT DEFECT. 213const IC_MERGE_ENABLED: i64 = 0 // two rows describing one thing is already a family worth naming 214const IC_MAXFAM: i64 = 8192 // <= IC_MAXROWS so the ranking array can always index every family 215// Ordinal escalation, deliberately larger than any achievable max_sev x open product on this plane, so the 216// urgent tier is a TIER and not a thumb on the scale that a big enough pile could still outweigh. 217const IC_URGENT_BOOST: i64 = 1000000 218// A family holding a sev>=9 row is an OUTAGE and preempts on ordinal grounds, exactly as an SRE rota does. 219// Without this, volume dominates severity in any product form and a live sev-9 sinks below a big sev-5 pile. 220const IC_URGENT_SEV: i64 = 9 221// Postmortem practice caps action items at 5-8. Emitting more is how a recommendation becomes a list again. 222const IC_EMIT: i64 = 8 223const IC_LABEL_TOKENS: i64 = 3 224const IC_IDS_PER_CLUSTER: i64 = 12 225const IC_OUTCAP: i64 = 262144 226const IC_LBUF: i64 = 512 // one cluster label ("tok|tok|tok"), reused per emitted family 227const IC_WRMODE: i64 = 420 // 0644 228const IC_TAB: i64 = 9 229const IC_NL: i64 = 10 230const IC_DAY: i64 = 86400 231const IC_MODE_RUN: i64 = 0 232const IC_MODE_SELFTEST: i64 = 1 233const IC_MODE_PAGE: i64 = 2 234const IC_MODE_EVAL: i64 = 3 235const IC_GTMAX: i64 = 64 236 237func ic_len(s: *u8) -> i64 { var i: i64 = 0; while s[i] != (0 as u8) { i = i + 1 } return i } 238func ic_puts(s: *u8) -> i64 { sys_write(1, s, ic_len(s)); return 0 } 239func ic_cat(d: *u8, o: i64, s: *u8) -> i64 { var i: i64 = 0; var o2: i64 = o; while s[i] != (0 as u8) { d[o2] = s[i]; o2 = o2 + 1; i = i + 1 } return o2 } 240func ic_catn(d: *u8, o: i64, v: i64) -> i64 { 241 let t: *u8 = sys_mmap(32) 242 var m: i64 = v 243 var neg: i64 = 0 244 if m < 0 { neg = 1; m = 0 - m } 245 var k: i64 = 0 246 if m == 0 { t[0] = 48 as u8; k = 1 } 247 while m > 0 { t[k] = ((m - (m / 10) * 10) + 48) as u8; m = m / 10; k = k + 1 } 248 var o2: i64 = o 249 if neg == 1 { d[o2] = 45 as u8; o2 = o2 + 1 } 250 var i: i64 = 0 251 while i < k { d[o2] = t[k - 1 - i]; o2 = o2 + 1; i = i + 1 } 252 return o2 253} 254func ic_catsl(d: *u8, o: i64, src: *u8, off: i64, len: i64) -> i64 { 255 var i: i64 = 0 256 var o2: i64 = o 257 while i < len { d[o2] = src[off + i]; o2 = o2 + 1; i = i + 1 } 258 return o2 259} 260func ic_atoi(s: *u8) -> i64 { 261 var v: i64 = 0 262 var i: i64 = 0 263 var go: i64 = 1 264 while go == 1 { 265 let c: i64 = s[i] as i64 266 if c < 48 { go = 0 } else { if c > 57 { go = 0 } else { v = v * 10 + (c - 48); i = i + 1 } } 267 } 268 return v 269} 270// digits of a byte slice -> i64 (the epoch field). Stops at the first non-digit. 271func ic_slice_num(q: *u8, off: i64, len: i64) -> i64 { 272 var v: i64 = 0 273 var i: i64 = 0 274 var go: i64 = 1 275 while go == 1 { 276 if i >= len { go = 0 } else { 277 let c: i64 = q[off + i] as i64 278 if c < 48 { go = 0 } else { if c > 57 { go = 0 } else { v = v * 10 + (c - 48); i = i + 1 } } 279 } 280 } 281 return v 282} 283 284// log2(x) scaled by 1024, integer-only: highest set bit + linear interpolation of the mantissa. 285// Max error ~0.086 bits, monotonic, deterministic, no float anywhere. Used for IDF so that a token in 1 row 286// and a token in 3 rows are actually distinguishable (a bare bit-position would collapse them). 287func ic_log2q(x: i64) -> i64 { 288 if x <= 1 { return 0 } 289 var b: i64 = 0 290 var v: i64 = x 291 while v > 1 { v = v / 2; b = b + 1 } 292 var p: i64 = 1 293 var i: i64 = 0 294 while i < b { p = p * 2; i = i + 1 } 295 let frac: i64 = ((x - p) * IC_MAGIC_1024) / p 296 return b * IC_MAGIC_1024 + frac 297} 298 299// ─── TOKEN CLASSIFICATION ────────────────────────────────────────────────────────────────────────────── 300// An IDENTIFIER token is one that names a THING IN THE SYSTEM rather than describing a symptom: it carries 301// an underscore or a hyphen (`nx_mgmt_api`, `gx-body`, `seg-store`). This is derived from the ecosystem's 302// own naming conventions, not invented. Identifier tokens are what the topology gate keys on -- symptom 303// words ("leak", "hang", "timeout") recur everywhere and must never be sufficient to merge two rows. 304// An IDENTIFIER must carry a separator AND AT LEAST ONE LETTER. The letter rule is not cosmetic: without 305// it '2026-07-31' is an identifier, and dates become the highest-volume anchors on the plane. MEASURED -- 306// the first anchor run ranked '2026-07-31' FIRST with 199 members across 163 scopes, plus '2026-07-25' and 307// '2026-07-23' at ranks 3 and 5. A date is the one thing every row filed that day shares, which makes it a 308// perfect anchor and a worthless family. 309// LAW: A TOKEN THAT EVERY ROW SHARES FOR A REASON UNRELATED TO ITS SUBJECT IS THE BEST-SCORING ANCHOR AND 310// THE WORST POSSIBLE ONE. 311func ic_is_ident(q: *u8, off: i64, len: i64) -> i64 { 312 var sep: i64 = 0 313 var alpha: i64 = 0 314 var i: i64 = 0 315 while i < len { 316 let c: i64 = q[off + i] as i64 317 if c == 95 { sep = 1 } 318 if c == 45 { sep = 1 } 319 if c >= 97 { if c <= 122 { alpha = 1 } } 320 if c >= 65 { if c <= 90 { alpha = 1 } } 321 i = i + 1 322 } 323 if sep == 1 { if alpha == 1 { return 1 } } 324 return 0 325} 326// IS THIS TOKEN THE NAME OF SOMETHING THAT ACTUALLY EXISTS? 327// The junk anchors all shared one property: MID-SESSION, API-pure, byte-identical and build-deploy are 328// phrases lifted from row prose, and NOTHING IN THE SYSTEM IS CALLED THAT. nx_mgmt_api.nx is a real file. 329// So the test is not a heuristic about shape -- it is a lookup against the tree. This is the topology 330// grounding TopoEvo actually prescribes (constrain correlation by the real system), where the earlier 331// separator-and-a-letter rule was only a shape approximation of it. 332// Cost is ~2 open() calls per candidate anchor (~1900), which is nothing. 333func ic_file_exists(path: *u8) -> i64 { 334 let fd: i64 = sys_openat_rd(path) 335 if fd < 0 { return 0 } 336 sys_close(fd) 337 return 1 338} 339func ic_try(pre: *u8, q: *u8, off: i64, len: i64, suf: *u8, scratch: *u8) -> i64 { 340 var o: i64 = ic_cat(scratch, 0, pre) 341 var i: i64 = 0 342 while i < len { scratch[o] = q[off + i]; o = o + 1; i = i + 1 } 343 o = ic_cat(scratch, o, suf) 344 scratch[o] = 0 as u8 345 return ic_file_exists(scratch) 346} 347// WIDENED 2026-08-01 ON MEASUREMENT. Organ-first raised emitted_organ_anchored 4/8 -> 6/8, but the 117-row 348// stale-mirror family KEPT the phrase anchor 'auto-filed' -- because its actual subject, nx_stale_sweep, 349// is a .cron.sh and NOT a .nx organ. The lookup correctly answered 'not a source file' and I had wrongly 350// equated that with 'not a real thing'. The system's real subjects are .nx SOURCES, .elf BINARIES and .sh 351// SCRIPTS alike. 352// LAW: A NAME TEST THAT KNOWS ONLY ONE FILE TYPE MISSES EVERY SUBJECT THAT LIVES IN ANOTHER. 353func ic_is_organ(q: *u8, off: i64, len: i64, scratch: *u8) -> i64 { 354 if ic_try("buildroot/runtime/" as *u8, q, off, len, ".nx" as *u8, scratch) == 1 { return 1 } 355 if ic_try("buildroot/runtime/_hdl_build/" as *u8, q, off, len, ".nx" as *u8, scratch) == 1 { return 1 } 356 if ic_try("" as *u8, q, off, len, ".elf" as *u8, scratch) == 1 { return 1 } 357 if ic_try("" as *u8, q, off, len, ".sh" as *u8, scratch) == 1 { return 1 } 358 if ic_try("" as *u8, q, off, len, ".cron.sh" as *u8, scratch) == 1 { return 1 } 359 return 0 360} 361 362// Pure-digit tokens are epochs, counts and dates. They are high-cardinality noise that would link unrelated 363// rows filed in the same second, so they are dropped at the tokenizer. 364func ic_all_digits(q: *u8, off: i64, len: i64) -> i64 { 365 var i: i64 = 0 366 while i < len { 367 let c: i64 = q[off + i] as i64 368 if c < 48 { return 0 } 369 if c > 57 { return 0 } 370 i = i + 1 371 } 372 return 1 373} 374func ic_tokchar(c: i64) -> i64 { 375 if c >= 97 { if c <= 122 { return 1 } } 376 if c >= 65 { if c <= 90 { return 1 } } 377 if c >= 48 { if c <= 57 { return 1 } } 378 if c == 95 { return 1 } 379 if c == 45 { return 1 } 380 return 0 381} 382func ic_alnum(c: i64) -> i64 { 383 if c >= 97 { if c <= 122 { return 1 } } 384 if c >= 65 { if c <= 90 { return 1 } } 385 if c >= 48 { if c <= 57 { return 1 } } 386 return 0 387} 388// case-insensitive byte compare of two slices in the same buffer 389func ic_slice_eq(q: *u8, a: i64, al: i64, b: i64, bl: i64) -> i64 { 390 if al != bl { return 0 } 391 var i: i64 = 0 392 while i < al { 393 var x: i64 = q[a + i] as i64 394 var y: i64 = q[b + i] as i64 395 if x >= 65 { if x <= 90 { x = x + 32 } } 396 if y >= 65 { if y <= 90 { y = y + 32 } } 397 if x != y { return 0 } 398 i = i + 1 399 } 400 return 1 401} 402// rolling hash, case-folded, kept inside IC_HMOD so it is always positive and never relies on overflow 403func ic_hash(q: *u8, off: i64, len: i64) -> i64 { 404 var h: i64 = 0 405 var i: i64 = 0 406 while i < len { 407 var c: i64 = q[off + i] as i64 408 if c >= 65 { if c <= 90 { c = c + 32 } } 409 h = h * 131 + c 410 h = h - (h / IC_HMOD) * IC_HMOD 411 i = i + 1 412 } 413 return h 414} 415// Intern a token. dh: [0]=hash [1]=off [2]=len [3]=df [4]=idf [5]=ident, stride 6. Returns slot, or -1 full. 416func ic_intern(q: *u8, off: i64, len: i64, dh: *i64) -> i64 { 417 let h: i64 = ic_hash(q, off, len) 418 var s: i64 = h - (h / IC_DICT) * IC_DICT 419 var probes: i64 = 0 420 var res: i64 = 0 - 1 421 var go: i64 = 1 422 while go == 1 { 423 if probes >= IC_DICT { go = 0 } else { 424 let base: i64 = s * 6 425 if dh[base + 2] == 0 { 426 dh[base + 0] = h 427 dh[base + 1] = off 428 dh[base + 2] = len 429 dh[base + 3] = 0 430 dh[base + 4] = 0 431 dh[base + 5] = ic_is_ident(q, off, len) 432 res = s 433 go = 0 434 } else { 435 if dh[base + 0] == h { 436 if ic_slice_eq(q, dh[base + 1], dh[base + 2], off, len) == 1 { res = s; go = 0 } 437 } 438 if go == 1 { 439 s = s + 1 440 if s >= IC_DICT { s = 0 } 441 probes = probes + 1 442 } 443 } 444 } 445 } 446 return res 447} 448 449// ─── UNION-FIND ──────────────────────────────────────────────────────────────────────────────────────── 450// THE RULER: HAND-VERIFIED GROUND TRUTH. 451// Every tuning decision in this organ up to 2026-07-31 was made BY INSPECTION -- I read the output, judged 452// the junk, changed a parameter. That is taste, not measurement, and it cannot tell a real improvement from 453// one that merely moved the junk somewhere I did not look. This table is the correction. 454// Each row below is a debt row I PERSONALLY READ IN FULL and assigned to a family by its subject. Keyed on 455// the EPOCH ID, never the row index: indices are positional and shift as the plane grows, so an index-keyed 456// ruler silently re-labels itself between runs and reports drift as quality change. 457// It is a SMALL SAMPLE (5 families / 20 rows of ~2259) and the output ALWAYS prints gt_found against 458// gt_declared so nobody can mistake this floor for a corpus-wide score. 459// Scored with B-CUBED, chosen over ARI/NMI on purpose: family sizes run 152 rows down to 2, and 460// pair-counting metrics are dominated by the largest cluster. B-cubed is per-ELEMENT, so a 2-row family 461// swallowed by a 152-row one costs exactly what it should. 462// families: 1=mgmt-api availability 2=dup-source 3=gx-body 4=untrusted-input security 5=memory leak 463func ic_gt_load(gtid: *i64, gtfam: *i64) -> i64 { 464 var n: i64 = 0 465 gtid[n] = IC_MAGIC_1784856230; gtfam[n] = 1; n = n + 1 466 gtid[n] = IC_MAGIC_1784857155; gtfam[n] = 1; n = n + 1 467 gtid[n] = IC_MAGIC_1784480242; gtfam[n] = 2; n = n + 1 468 gtid[n] = IC_MAGIC_1784856346; gtfam[n] = 2; n = n + 1 469 gtid[n] = IC_MAGIC_1785030820; gtfam[n] = 2; n = n + 1 470 gtid[n] = IC_MAGIC_1785031111; gtfam[n] = 2; n = n + 1 471 gtid[n] = IC_MAGIC_1784995429; gtfam[n] = 3; n = n + 1 472 gtid[n] = IC_MAGIC_1784995511; gtfam[n] = 3; n = n + 1 473 gtid[n] = IC_MAGIC_1784996221; gtfam[n] = 3; n = n + 1 474 gtid[n] = IC_MAGIC_1785002845; gtfam[n] = 3; n = n + 1 475 gtid[n] = IC_MAGIC_1785002909; gtfam[n] = 3; n = n + 1 476 gtid[n] = IC_MAGIC_1785003511; gtfam[n] = 3; n = n + 1 477 gtid[n] = IC_MAGIC_1785003779; gtfam[n] = 3; n = n + 1 478 gtid[n] = IC_MAGIC_1785003935; gtfam[n] = 3; n = n + 1 479 gtid[n] = IC_MAGIC_1785027945; gtfam[n] = 3; n = n + 1 480 gtid[n] = IC_MAGIC_1784661149; gtfam[n] = 4; n = n + 1 481 gtid[n] = IC_MAGIC_1784661346; gtfam[n] = 4; n = n + 1 482 gtid[n] = IC_MAGIC_1784664001; gtfam[n] = 4; n = n + 1 483 gtid[n] = IC_MAGIC_1785027242; gtfam[n] = 5; n = n + 1 484 gtid[n] = IC_MAGIC_1784908046; gtfam[n] = 5; n = n + 1 485 gtid[n] = IC_MAGIC_1784947125; gtfam[n] = 3; n = n + 1 486 gtid[n] = IC_MAGIC_1784947134; gtfam[n] = 3; n = n + 1 487 gtid[n] = IC_MAGIC_1784947786; gtfam[n] = 3; n = n + 1 488 gtid[n] = IC_MAGIC_1784951793; gtfam[n] = 3; n = n + 1 489 gtid[n] = IC_MAGIC_1784952157; gtfam[n] = 3; n = n + 1 490 gtid[n] = IC_MAGIC_1784954507; gtfam[n] = 3; n = n + 1 491 gtid[n] = IC_MAGIC_1784955224; gtfam[n] = 3; n = n + 1 492 gtid[n] = IC_MAGIC_1784955875; gtfam[n] = 3; n = n + 1 493 gtid[n] = IC_MAGIC_1784956205; gtfam[n] = 3; n = n + 1 494 gtid[n] = IC_MAGIC_1784995126; gtfam[n] = 3; n = n + 1 495 gtid[n] = IC_MAGIC_1784995721; gtfam[n] = 3; n = n + 1 496 gtid[n] = IC_MAGIC_1784995947; gtfam[n] = 3; n = n + 1 497 gtid[n] = IC_MAGIC_1784996406; gtfam[n] = 3; n = n + 1 498 gtid[n] = IC_MAGIC_1784996558; gtfam[n] = 3; n = n + 1 499 gtid[n] = IC_MAGIC_1785000800; gtfam[n] = 3; n = n + 1 500 gtid[n] = IC_MAGIC_1784925304; gtfam[n] = 6; n = n + 1 501 gtid[n] = IC_MAGIC_1784927702; gtfam[n] = 7; n = n + 1 502 gtid[n] = IC_MAGIC_1784958766; gtfam[n] = 8; n = n + 1 503 gtid[n] = IC_MAGIC_1784978119; gtfam[n] = 9; n = n + 1 504 gtid[n] = IC_MAGIC_1784999221; gtfam[n] = 10; n = n + 1 505 gtid[n] = IC_MAGIC_1785001189; gtfam[n] = 11; n = n + 1 506 gtid[n] = IC_MAGIC_1785002038; gtfam[n] = 12; n = n + 1 507 return n 508} 509 510// BOUNDED OVERLAP: a row belongs to its primary family and, at most, ALSO to its scope family. 511// Splits survived every single-membership design (gold_families_split stuck at 4-5) because one row 512// can only sit in one bucket, so a family whose members carry different anchors CANNOT be whole. 513// Cap is TWO deliberately: unbounded overlap would put a row in every family it name-drops and 514// precision would collapse. B-cubed scores overlapping clusters natively, so the ruler can judge this. 515func ic_in_fam(as1: *i64, as2: *i64, r: i64, c: i64) -> i64 { 516 if as1[r] == c { return 1 } 517 if as2[r] == c { return 1 } 518 return 0 519} 520func ic_share_fam(as1: *i64, as2: *i64, a: i64, b: i64) -> i64 { 521 if as1[a] >= 0 { if ic_in_fam(as1, as2, b, as1[a]) == 1 { return 1 } } 522 if as2[a] >= 0 { if ic_in_fam(as1, as2, b, as2[a]) == 1 { return 1 } } 523 return 0 524} 525func ic_find(par: *i64, x: i64) -> i64 { 526 var r: i64 = x 527 while par[r] != r { r = par[r] } 528 var c: i64 = x 529 while par[c] != c { let nxt: i64 = par[c]; par[c] = r; c = nxt } 530 return r 531} 532func ic_union(par: *i64, a: i64, b: i64) -> i64 { 533 let ra: i64 = ic_find(par, a) 534 let rb: i64 = ic_find(par, b) 535 if ra == rb { return 0 } 536 if ra < rb { par[rb] = ra } else { par[ra] = rb } 537 return 1 538} 539 540// ─── THE PIPELINE ────────────────────────────────────────────────────────────────────────────────────── 541// Runs over an in-memory TSV plane so that `run` and `selftest` execute the IDENTICAL code path. A selftest 542// that exercises a reimplementation of the thing it certifies proves nothing (A DELEGATED CHECK IS NOT A 543// CHECK UNTIL YOU WATCH THE DELEGATE PERFORM IT). 544// 545// arrays, all preallocated once (sys_mmap is page-granular with no free -- allocating inside a loop is the 546// banked unfreed-mmap class): 547// rmeta stride 8: [0]=epoch [1]=sev [2]=open [3]=ntok [4]=idfsum [5]=core [6]=scope_off [7]=scope_len 548// rtok: IC_MAXROWS * IC_MAXTOK slot ids 549// st: [0]=rows [1]=rows_capped [2]=open_rows [3]=distinct_tokens [4]=df_skipped [5]=shape_alt 550// [6]=pairs_lo [7]=pairs_join [8]=pairs_hi 551func ic_parse(q: *u8, n: i64, rmeta: *i64, rtok: *i64, dh: *i64, st: *i64) -> i64 { 552 var rows: i64 = 0 553 var pos: i64 = 0 554 // Field-bound scratch, allocated ONCE. sys_mmap is raw and page-granular with no free, so allocating 555 // these inside the per-row loop would leak 8KB x rows -- the exact unfreed-mmap class this ecosystem 556 // banked as a law. A leak in the organ that ranks leaks would be its own best finding. 557 let fo: *i64 = sys_mmap(64) as *i64 558 let fl: *i64 = sys_mmap(64) as *i64 559 while pos < n { 560 var e: i64 = pos 561 var fin: i64 = 0 562 while fin == 0 { 563 if e >= n { fin = 1 } else { if q[e] == (IC_NL as u8) { fin = 1 } else { e = e + 1 } } 564 } 565 if e > pos { 566 if rows >= IC_MAXROWS { st[1] = st[1] + 1 } else { 567 // ---- field split (bounded scan; we only need the first five field bounds) ---- 568 var f: i64 = 0 569 var fs: i64 = pos 570 var k: i64 = pos 571 while k <= e { 572 var sep: i64 = 0 573 if k == e { sep = 1 } else { if q[k] == (IC_TAB as u8) { sep = 1 } } 574 if sep == 1 { 575 if f < 7 { fo[f] = fs; fl[f] = k - fs } 576 f = f + 1 577 fs = k + 1 578 } 579 k = k + 1 580 } 581 // ---- severity: DEFENSIVE AT THE BOUNDARY (rule 12). The plane carries two row shapes -- 582 // the common `epoch/sev/scope/status/desc` and a rarer `id/title/sev/status/...` (D001). 583 // Reading sev from a fixed index gives the second shape a severity derived from the first 584 // letter of its title. So: accept the first of fields 1..2 that is a LONE digit 1-9. 585 var sev: i64 = 0 586 var sev_at: i64 = 0 - 1 587 var ti: i64 = 1 588 while ti <= 2 { 589 if sev_at < 0 { if f > ti { if fl[ti] == 1 { 590 let c: i64 = q[fo[ti]] as i64 591 if c >= 49 { if c <= 57 { sev = c - 48; sev_at = ti } } 592 } } } 593 ti = ti + 1 594 } 595 if sev_at != 1 { st[5] = st[5] + 1 } 596 // ---- status: an `open` field anywhere in the first five (both shapes put it at 3) ---- 597 var is_open: i64 = 0 598 var si: i64 = 2 599 while si <= 4 { 600 if f > si { if fl[si] == 4 { 601 let o0: i64 = fo[si] 602 if q[o0] == (111 as u8) { if q[o0+1] == (112 as u8) { if q[o0+2] == (101 as u8) { if q[o0+3] == (110 as u8) { is_open = 1 } } } } 603 } } 604 si = si + 1 605 } 606 // ---- scope: only trustworthy when the row is the common shape ---- 607 var sc_off: i64 = 0 608 var sc_len: i64 = 0 609 if sev_at == 1 { if f > 2 { sc_off = fo[2]; sc_len = fl[2] } } 610 // THE SCOPE IS THE FILER'S OWN STATEMENT OF WHAT THE ROW IS ABOUT -- the most 611 // authoritative family signal on the row, and I had been treating it as just another 612 // token. MEASURED: organ-first ordering split 4 of 5 multi-row gold families (worst 613 // into 5 buckets) because a row gets claimed by an organ it merely MENTIONS instead of 614 // by its subject. Marking the scope token outranks that. 615 if sc_len >= IC_TOKMIN { if sc_len <= IC_TOKMAX { 616 if ic_all_digits(q, sc_off, sc_len) == 0 { 617 let scsl: i64 = ic_intern(q, sc_off, sc_len, dh) 618 // SCOPE-FIRST DISABLED 2026-08-01 -- BUILT, MEASURED, REFUTED BY THE SECOND RULER. 619 // B-cubed said BETTER (precision 900->923, F1 739->747) but the label-free 620 // metric said WORSE: emitted_organ_anchored 7 -> 3, and the emitted labels 621 // degenerated to generic buckets -- gates(129 members across 108 SCOPES), 622 // sites, evidence, plane, CORRECTION. It also did NOT do what it was built 623 // for: gold_families_split stayed 4 and worst_split stayed 5. 624 // ROOT CAUSE: a slot rank is GLOBAL, so a scope word that is also common 625 // prose ('gates') outranks everything EVERYWHERE it appears, not only on the 626 // rows it is actually the scope OF. Fixing that needs PER-ROW anchor 627 // eligibility, which this design cannot express. 628 // LAW: WHEN TWO RULERS DISAGREE, THE ONE MEASURING THE THING THAT VISIBLY 629 // CHANGED WINS -- a +23 precision point does not buy 'CORRECTION' as the name 630 // of a work item. 631 if scsl >= 0 { if IC_SCOPE_FIRST == 1 { dh[scsl * 6 + 5] = 2 } } 632 } 633 } } 634 let base: i64 = rows * 8 635 rmeta[base + 0] = ic_slice_num(q, fo[0], fl[0]) 636 rmeta[base + 1] = sev 637 rmeta[base + 2] = is_open 638 rmeta[base + 5] = pos 639 rmeta[base + 6] = sc_off 640 rmeta[base + 7] = sc_len 641 if is_open == 1 { st[2] = st[2] + 1 } 642 // ---- tokenize the WHOLE row. Epochs and counts fall out via the pure-digit rule; `open` 643 // and other structural words fall out via the DF cap. No stopword table exists on purpose. 644 var nt: i64 = 0 645 var p: i64 = pos 646 while p < e { 647 if ic_alnum(q[p] as i64) == 1 { 648 // walk forward over token characters; the flag keeps the END POSITION intact 649 // (breaking by assigning the bound would lose it -- and a token whose length is 650 // silently wrong merges two unrelated identifiers) 651 var end2: i64 = p 652 var scan: i64 = 1 653 while scan == 1 { 654 if end2 >= e { scan = 0 } else { 655 if ic_tokchar(q[end2] as i64) == 1 { end2 = end2 + 1 } else { scan = 0 } 656 } 657 } 658 let L: i64 = end2 - p 659 // over-long tokens are SKIPPED, never truncated: a truncated token is a DIFFERENT 660 // token wearing the same bytes, and it would merge families that share a prefix 661 if L >= IC_TOKMIN { if L <= IC_TOKMAX { 662 if ic_all_digits(q, p, L) == 0 { 663 if nt < IC_MAXTOK { 664 let slot: i64 = ic_intern(q, p, L, dh) 665 if slot >= 0 { 666 // per-row dedup: a token counts once per row (document frequency) 667 var dup: i64 = 0 668 var z: i64 = 0 669 while z < nt { if rtok[rows * IC_MAXTOK + z] == slot { dup = 1 } z = z + 1 } 670 if dup == 0 { 671 rtok[rows * IC_MAXTOK + nt] = slot 672 nt = nt + 1 673 if dh[slot * 6 + 3] == 0 { st[3] = st[3] + 1 } 674 dh[slot * 6 + 3] = dh[slot * 6 + 3] + 1 675 } 676 } 677 } 678 } 679 } } 680 p = end2 681 } else { p = p + 1 } 682 } 683 rmeta[base + 3] = nt 684 rows = rows + 1 685 } 686 } 687 pos = e + 1 688 } 689 st[0] = rows 690 return rows 691} 692 693func main(argc: i64, argv: *i64) -> i64 { 694 if argc < 2 { 695 ic_puts("usage: nx_debtcluster run [prefix] [sim_join_permil] | page <outfile> [prefix] [sim] | selftest\n" as *u8) 696 sys_exit(2); return 2 697 } 698 let verb: *u8 = argv[1] as *u8 699 var mode: i64 = IC_MODE_RUN 700 if verb[0] == (115 as u8) { mode = IC_MODE_SELFTEST } 701 if verb[0] == (112 as u8) { mode = IC_MODE_PAGE } 702 if verb[0] == (101 as u8) { mode = IC_MODE_EVAL } 703 var is_self: i64 = 0 704 if mode == IC_MODE_SELFTEST { is_self = 1 } 705 706 // ⚠THE PLANE PREFIX IS A FULL PATH: knowledge/store/debt-. A bare "debt-" resolves to nothing and the 707 // organ correctly returns INSTRUMENT-BLIND rather than "no clusters found" -- keep that default HONEST 708 // (wrong) rather than convenient, so a mis-invocation is loud instead of quietly empty. 709 var prefix: *u8 = "knowledge/store/debt-" as *u8 710 var simjoin: i64 = IC_SIM_JOIN 711 var outfile: *u8 = "sites/nishifamily/code/families" as *u8 712 if mode == IC_MODE_PAGE { 713 if argc < 3 { ic_puts("usage: nx_debtcluster page <outfile> [prefix] [sim_join_permil]\n" as *u8); sys_exit(2); return 2 } 714 outfile = argv[2] as *u8 715 if argc > 3 { prefix = argv[3] as *u8 } 716 if argc > 4 { simjoin = ic_atoi(argv[4] as *u8) } 717 } 718 if mode == IC_MODE_RUN { 719 if argc > 2 { prefix = argv[2] as *u8 } 720 if argc > 3 { simjoin = ic_atoi(argv[3] as *u8) } 721 } 722 if mode == IC_MODE_EVAL { 723 if argc > 2 { prefix = argv[2] as *u8 } 724 } 725 if simjoin <= 0 { simjoin = IC_SIM_JOIN } 726 if simjoin > 1000 { simjoin = IC_SIM_JOIN } 727 728 let q: *u8 = sys_mmap(IC_CAP) 729 let dbf: *i64 = sys_mmap(64) as *i64 730 dbf[0] = 0 731 dbf[1] = 0 732 var n: i64 = 0 733 734 if is_self == 1 { 735 // ── PLANTED FIXTURES. T1/T2 are the pair that matters: near-identical SYMPTOM TEXT, and the only 736 // difference is WHICH ORGAN it happened in. A text-only clusterer merges rows 0/1 with rows 2/3. 737 // This organ must not. If it does, selftest fails and the build does not ship. 738 var o: i64 = 0 739 o = ic_cat(q, o, "1700000001\t9\tmgmt-api\topen\tnx_mgmt_api_core wedged on start, memory grows unbounded on every call and is never freed\n" as *u8) 740 o = ic_cat(q, o, "1700000002\t8\tmgmt-api\topen\tnx_mgmt_api_core leaks on each request, memory grows unbounded on every call and is never freed\n" as *u8) 741 o = ic_cat(q, o, "1700000003\t8\tvault-walk\topen\tnx_vault_walker_svc leaks on each request, memory grows unbounded on every call and is never freed\n" as *u8) 742 o = ic_cat(q, o, "1700000004\t7\tvault-walk\topen\tnx_vault_walker_svc wedged on start, memory grows unbounded on every call and is never freed\n" as *u8) 743 o = ic_cat(q, o, "1700000005\t4\tsolo-lane\topen\tthe carburettor gasket needs replacing before the spring rally weekend arrives\n" as *u8) 744 n = o 745 dbf[0] = 5 746 dbf[1] = 5 747 } else { 748 n = sts_load_honest(prefix, q, IC_CAP, dbf) 749 if n <= 0 { 750 ic_puts("{\"organ\":\"nx_debtcluster\",\"verdict\":\"INSTRUMENT-BLIND\",\"reason\":\"debt plane loaded ZERO bytes -- no rows were examined, so NO clustering claim is made\"}\n" as *u8) 751 sys_exit(2); return 2 752 } 753 } 754 755 let rmeta: *i64 = sys_mmap(8 * 8 * IC_MAXROWS) as *i64 756 let rtok: *i64 = sys_mmap(8 * IC_MAXTOK * IC_MAXROWS) as *i64 757 let dh: *i64 = sys_mmap(8 * 6 * IC_DICT) as *i64 758 let st: *i64 = sys_mmap(128) as *i64 759 var zz: i64 = 0 760 while zz < 16 { st[zz] = 0; zz = zz + 1 } 761 762 let rows: i64 = ic_parse(q, n, rmeta, rtok, dh, st) 763 if rows <= 0 { 764 ic_puts("{\"organ\":\"nx_debtcluster\",\"verdict\":\"INSTRUMENT-BLIND\",\"reason\":\"plane held bytes but ZERO rows parsed -- the row shape is not what this organ reads\"}\n" as *u8) 765 sys_exit(2); return 2 766 } 767 768 // ── IDF. idf = log2(N) - log2(df), in 1/1024 bit units, floored at 0. A token in every row scores 0 and 769 // therefore contributes nothing to any similarity -- stopwords are eliminated BY MEASUREMENT. 770 let logN: i64 = ic_log2q(rows) 771 let dfmax: i64 = (rows * IC_DF_PERMIL) / 1000 772 var s2: i64 = 0 773 while s2 < IC_DICT { 774 let b2: i64 = s2 * 6 775 if dh[b2 + 2] != 0 { 776 let df: i64 = dh[b2 + 3] 777 if df > dfmax { if dfmax > 0 { dh[b2 + 4] = 0 - 1; st[4] = st[4] + 1 } } 778 if dh[b2 + 4] != (0 - 1) { 779 var v: i64 = logN - ic_log2q(df) 780 if v < 0 { v = 0 } 781 dh[b2 + 4] = v 782 } 783 } 784 s2 = s2 + 1 785 } 786 // per-row IDF mass (skipped tokens contribute nothing, so they cannot inflate an overlap ratio) 787 var r2: i64 = 0 788 while r2 < rows { 789 var sum: i64 = 0 790 var t3: i64 = 0 791 while t3 < rmeta[r2 * 8 + 3] { 792 let sl: i64 = rtok[r2 * IC_MAXTOK + t3] 793 let w: i64 = dh[sl * 6 + 4] 794 if w > 0 { sum = sum + w } 795 t3 = t3 + 1 796 } 797 rmeta[r2 * 8 + 4] = sum 798 rmeta[r2 * 8 + 5] = 0 799 r2 = r2 + 1 800 } 801 802 // ── POSTINGS (token -> rows). Turns an O(rows^2) all-pairs scan into a scan over rows that actually 803 // share a rare token. Without this, 2068 rows is 2.1M pairwise set intersections. 804 let pstart: *i64 = sys_mmap(8 * (IC_DICT + 1)) as *i64 805 let pfill: *i64 = sys_mmap(8 * (IC_DICT + 1)) as *i64 806 let prow: *i64 = sys_mmap(8 * IC_POSTCAP) as *i64 807 var s3: i64 = 0 808 var acc2: i64 = 0 809 while s3 < IC_DICT { 810 pstart[s3] = acc2 811 pfill[s3] = 0 812 let b3: i64 = s3 * 6 813 if dh[b3 + 2] != 0 { if dh[b3 + 4] > 0 { acc2 = acc2 + dh[b3 + 3] } } 814 s3 = s3 + 1 815 } 816 var r3: i64 = 0 817 while r3 < rows { 818 var t4: i64 = 0 819 while t4 < rmeta[r3 * 8 + 3] { 820 let sl2: i64 = rtok[r3 * IC_MAXTOK + t4] 821 if dh[sl2 * 6 + 4] > 0 { 822 let idx2: i64 = pstart[sl2] + pfill[sl2] 823 if idx2 < IC_POSTCAP { prow[idx2] = r3; pfill[sl2] = pfill[sl2] + 1 } 824 } 825 t4 = t4 + 1 826 } 827 r3 = r3 + 1 828 } 829 830 // ── ANCHOR PASS. A candidate anchor is an identifier token carried by at least IC_MINFAM rows and by no 831 // more than IC_ANCHOR_DF_PERMIL of them. Its family is EXACTLY the rows that contain it. Membership is a 832 // property each row holds ON ITS OWN, so there is no path through other rows and chaining cannot occur. 833 let acand: *i64 = sys_mmap(8 * IC_MAXFAM) as *i64 834 let apri: *i64 = sys_mmap(8 * IC_MAXFAM) as *i64 835 // aorg[i] = 1 when this anchor NAMES A REAL SOURCE FILE. Organ-named anchors claim their rows FIRST, 836 // so a family gets named after the thing it is about rather than after a phrase its rows happen to 837 // share. This is the higher-precision blocking strategy running ahead of the general one -- which is 838 // what the entity-resolution literature actually prescribes, rather than merging afterwards. 839 let aorg: *i64 = sys_mmap(8 * IC_MAXFAM) as *i64 840 let ascp: *i64 = sys_mmap(8 * IC_MAXFAM) as *i64 841 let orgscratch: *u8 = sys_mmap(IC_MAGIC_1024) 842 var ncand: i64 = 0 843 var anchors_too_broad: i64 = 0 844 // FLOOR THE PROPORTION. (rows * permil)/1000 INTEGER-DIVIDES TO ZERO on a small plane -- at rows=5 the 845 // cap came out 0, every anchor was rejected as too broad, and the organ reported families=0 while 846 // cheerfully declaring coverage_complete=1. Same shape as the minPts error above. 847 // A THRESHOLD DERIVED AS A PROPORTION MUST BE FLOORED AT THE DOMAIN'S SMALLEST REAL UNIT, or it 848 // silently forbids the very thing it is measuring. 849 var anchor_dfmax: i64 = (rows * IC_ANCHOR_DF_PERMIL) / 1000 850 if anchor_dfmax < IC_MINFAM { anchor_dfmax = IC_MINFAM } 851 var sA: i64 = 0 852 while sA < IC_DICT { 853 let bA: i64 = sA * 6 854 if dh[bA + 2] != 0 { if dh[bA + 5] >= 1 { if dh[bA + 4] > 0 { 855 let dfA: i64 = dh[bA + 3] 856 if dfA >= IC_MINFAM { 857 if dfA > anchor_dfmax { anchors_too_broad = anchors_too_broad + 1 } else { 858 if ncand < IC_MAXFAM { 859 var mx: i64 = 0 860 var op: i64 = 0 861 var pi2: i64 = 0 862 while pi2 < pfill[sA] { 863 let rr: i64 = prow[pstart[sA] + pi2] 864 if rmeta[rr * 8 + 1] > mx { mx = rmeta[rr * 8 + 1] } 865 if rmeta[rr * 8 + 2] == 1 { op = op + 1 } 866 pi2 = pi2 + 1 867 } 868 acand[ncand] = sA 869 apri[ncand] = mx * op 870 aorg[ncand] = ic_is_organ(q, dh[bA + 1], dh[bA + 2], orgscratch) 871 ascp[ncand] = 0 872 if dh[bA + 5] >= 2 { ascp[ncand] = 1 } 873 ncand = ncand + 1 874 } 875 } 876 } 877 } } } 878 sA = sA + 1 879 } 880 881 // Claim order: urgent tier first, then highest value. Greedy set-cover -- a row joins EXACTLY ONE family, 882 // so the emitted list is a partition of the clustered rows and no row is ever counted twice across two 883 // recommendations (double-counting would inflate every total on the page). 884 var oa: i64 = 0 885 while oa < ncand { 886 var bi: i64 = oa 887 var ob2: i64 = oa + 1 888 while ob2 < ncand { 889 var better: i64 = 0 890 // CLAIM BY VALUE ALONE (cohesion). The organ tier is deliberately NOT consulted here -- 891 // see IC_ORGAN_CLAIM below for the measured reason. 892 if IC_ORGAN_CLAIM == 1 { 893 if aorg[ob2] > aorg[bi] { better = 1 } 894 if aorg[ob2] == aorg[bi] { if apri[ob2] > apri[bi] { better = 1 } } 895 } 896 if IC_ORGAN_CLAIM == 0 { if apri[ob2] > apri[bi] { better = 1 } } 897 if better == 1 { bi = ob2 } 898 ob2 = ob2 + 1 899 } 900 let t1: i64 = acand[oa] 901 acand[oa] = acand[bi] 902 acand[bi] = t1 903 let t2: i64 = apri[oa] 904 apri[oa] = apri[bi] 905 apri[bi] = t2 906 let t3: i64 = aorg[oa] 907 aorg[oa] = aorg[bi] 908 aorg[bi] = t3 909 let t4: i64 = ascp[oa] 910 ascp[oa] = ascp[bi] 911 ascp[bi] = t4 912 oa = oa + 1 913 } 914 915 // each row's OWN scope token, resolved once (ic_intern is idempotent -- returns the existing slot) 916 let rscope: *i64 = sys_mmap(8 * IC_MAXROWS) as *i64 917 var rs: i64 = 0 918 while rs < rows { 919 rscope[rs] = 0 - 1 920 if rmeta[rs * 8 + 7] >= IC_TOKMIN { if rmeta[rs * 8 + 7] <= IC_TOKMAX { 921 if ic_all_digits(q, rmeta[rs * 8 + 6], rmeta[rs * 8 + 7]) == 0 { 922 rscope[rs] = ic_intern(q, rmeta[rs * 8 + 6], rmeta[rs * 8 + 7], dh) 923 } 924 } } 925 rs = rs + 1 926 } 927 let assign: *i64 = sys_mmap(8 * IC_MAXROWS) as *i64 928 var za: i64 = 0 929 while za < rows { assign[za] = 0 - 1; za = za + 1 } 930 let cmeta: *i64 = sys_mmap(8 * 8 * IC_MAXFAM) as *i64 931 var nc: i64 = 0 932 var phase: i64 = 0 933 while phase < 2 { 934 var ci3: i64 = 0 935 while ci3 < ncand { 936 if nc < IC_MAXFAM { 937 let sl5: i64 = acand[ci3] 938 var freen: i64 = 0 939 var pk: i64 = 0 940 while pk < pfill[sl5] { 941 let cr: i64 = prow[pstart[sl5] + pk] 942 if assign[cr] < 0 { 943 if phase == 1 { freen = freen + 1 } 944 if phase == 0 { if rscope[cr] == sl5 { freen = freen + 1 } } 945 } 946 pk = pk + 1 947 } 948 // A family whose rows have all been claimed by a higher-value anchor is NOT emitted again. 949 if freen >= IC_MINFAM { 950 let cb3: i64 = nc * 8 951 cmeta[cb3 + 0] = sl5 952 cmeta[cb3 + 1] = 0 953 cmeta[cb3 + 2] = 0 954 cmeta[cb3 + 3] = 0 955 cmeta[cb3 + 4] = 0 956 cmeta[cb3 + 5] = 0 957 cmeta[cb3 + 7] = 0 958 var pm: i64 = 0 959 while pm < pfill[sl5] { 960 let rw: i64 = prow[pstart[sl5] + pm] 961 var elig: i64 = 0 962 if assign[rw] < 0 { 963 if phase == 1 { elig = 1 } 964 if phase == 0 { if rscope[rw] == sl5 { elig = 1 } } 965 } 966 if elig == 1 { 967 assign[rw] = nc 968 cmeta[cb3 + 1] = cmeta[cb3 + 1] + 1 969 if rmeta[rw * 8 + 2] == 1 { cmeta[cb3 + 2] = cmeta[cb3 + 2] + 1 } 970 if rmeta[rw * 8 + 1] > cmeta[cb3 + 3] { cmeta[cb3 + 3] = rmeta[rw * 8 + 1] } 971 let ep2: i64 = rmeta[rw * 8 + 0] 972 if ep2 > 0 { 973 if cmeta[cb3 + 4] == 0 { cmeta[cb3 + 4] = ep2 } 974 if ep2 < cmeta[cb3 + 4] { cmeta[cb3 + 4] = ep2 } 975 if ep2 > cmeta[cb3 + 5] { cmeta[cb3 + 5] = ep2 } 976 } 977 } 978 pm = pm + 1 979 } 980 nc = nc + 1 981 } 982 } 983 ci3 = ci3 + 1 984 } 985 phase = phase + 1 986 } 987 988 // SECONDARY (scope) MEMBERSHIP -- the overlap that heals splits. 989 let assign2: *i64 = sys_mmap(8 * IC_MAXROWS) as *i64 990 let sfam: *i64 = sys_mmap(8 * IC_DICT) as *i64 991 var sf2: i64 = 0 992 while sf2 < IC_DICT { sfam[sf2] = 0 - 1; sf2 = sf2 + 1 } 993 var cf2: i64 = 0 994 while cf2 < nc { sfam[cmeta[cf2 * 8 + 0]] = cf2; cf2 = cf2 + 1 } 995 var overlap_added: i64 = 0 996 var ov: i64 = 0 997 while ov < rows { 998 assign2[ov] = 0 - 1 999 if IC_OVERLAP == 1 { if rscope[ov] >= 0 { 1000 let sfx: i64 = sfam[rscope[ov]] 1001 if sfx >= 0 { if sfx != assign[ov] { 1002 assign2[ov] = sfx 1003 overlap_added = overlap_added + 1 1004 } } 1005 } } 1006 ov = ov + 1 1007 } 1008 // recount members WITH secondary membership so the published table matches who is actually in it 1009 if overlap_added > 0 { 1010 var oc: i64 = 0 1011 while oc < nc { cmeta[oc * 8 + 1] = 0; cmeta[oc * 8 + 2] = 0; cmeta[oc * 8 + 3] = 0; oc = oc + 1 } 1012 var orr: i64 = 0 1013 while orr < rows { 1014 var oq: i64 = 0 1015 while oq < nc { 1016 if ic_in_fam(assign, assign2, orr, oq) == 1 { 1017 cmeta[oq * 8 + 1] = cmeta[oq * 8 + 1] + 1 1018 if rmeta[orr * 8 + 2] == 1 { cmeta[oq * 8 + 2] = cmeta[oq * 8 + 2] + 1 } 1019 if rmeta[orr * 8 + 1] > cmeta[oq * 8 + 3] { cmeta[oq * 8 + 3] = rmeta[orr * 8 + 1] } 1020 } 1021 oq = oq + 1 1022 } 1023 orr = orr + 1 1024 } 1025 var op: i64 = 0 1026 while op < nc { cmeta[op * 8 + 6] = cmeta[op * 8 + 3] * cmeta[op * 8 + 2]; op = op + 1 } 1027 } 1028 1029 // distinct scopes per family (context, not a score term), then the priority 1030 var cs: i64 = 0 1031 while cs < nc { 1032 var distinct: i64 = 0 1033 var m1: i64 = 0 1034 while m1 < rows { 1035 if assign[m1] == cs { if rmeta[m1 * 8 + 7] > 0 { 1036 var seen: i64 = 0 1037 var m2: i64 = 0 1038 while m2 < m1 { 1039 if assign[m2] == cs { 1040 if ic_slice_eq(q, rmeta[m1 * 8 + 6], rmeta[m1 * 8 + 7], rmeta[m2 * 8 + 6], rmeta[m2 * 8 + 7]) == 1 { seen = 1 } 1041 } 1042 m2 = m2 + 1 1043 } 1044 if seen == 0 { distinct = distinct + 1 } 1045 } } 1046 m1 = m1 + 1 1047 } 1048 cmeta[cs * 8 + 7] = distinct 1049 // ★PRIORITY = max severity x OPEN recurrence. Two measured terms, no invented weights, and already 1050 // CLOSED rows deliberately do not inflate it -- a family that is mostly eaten is mostly done. 1051 // distinct_scopes was REMOVED from the product: under anchor clustering it rewards a GENERIC anchor 1052 // (one touching many lanes) over a specific one, which is backwards for "what should I build". 1053 // A family holding a sev>=IC_URGENT_SEV row is escalated ORDINALLY, never by inflating the number. 1054 var pr: i64 = cmeta[cs * 8 + 3] * cmeta[cs * 8 + 2] 1055 // URGENT-TIER BOOST REMOVED 2026-07-31, ON MEASUREMENT. The intent was that an outage preempts. 1056 // But sev-9 rows are spread across this plane, so 8 of the top 8 families qualified and the tier 1057 // collapsed into 'sort the sev-9 families by size' -- which is what let a 199-row date anchor rank 1058 // first. A tier that almost every candidate reaches is not a tier, it is a constant. 1059 // max_sev x open already weights severity honestly and keeps the ordering readable. 1060 // LAW: AN ESCALATION EVERYTHING QUALIFIES FOR RANKS NOTHING. 1061 cmeta[cs * 8 + 6] = pr 1062 cs = cs + 1 1063 } 1064 // ── RANK + EMIT 1065 // MERGE PASS -- RECALL RECOVERY ON AGGREGATE EVIDENCE. 1066 // The ruler measured precision 840 / recall 635: what anchors group is right, but a true family whose 1067 // members carry DIFFERENT anchors gets split. 2026 entity-resolution practice is explicit about the 1068 // remedy -- RECALL IMPROVES BY RUNNING SEVERAL DIVERSE BLOCKING STRATEGIES IN PARALLEL, precision by 1069 // hard vetoes. Anchors are ONE blocking strategy. 1070 // The v1 linker also recovered recall -- by transitive closure -- and chained catastrophically (44 1071 // members / 15 scopes). The decisive difference here is the EVIDENCE THRESHOLD: two families merge only 1072 // when a MAJORITY of the SMALLER family's rows each independently carry the other family's anchor. A 1073 // single bridging row can never trigger it, which is precisely the property single linkage lacked. 1074 // ONE LEVEL ONLY: a family that has merged is not itself re-merged, so no chain can form. 1075 let slot_fam: *i64 = sys_mmap(8 * IC_DICT) as *i64 1076 var sf: i64 = 0 1077 while sf < IC_DICT { slot_fam[sf] = 0 - 1; sf = sf + 1 } 1078 var cf: i64 = 0 1079 while cf < nc { slot_fam[cmeta[cf * 8 + 0]] = cf; cf = cf + 1 } 1080 let cofam: *i64 = sys_mmap(8 * IC_MAXFAM) as *i64 1081 let cotouch: *i64 = sys_mmap(8 * IC_MAXFAM) as *i64 1082 let fmerge: *i64 = sys_mmap(8 * IC_MAXFAM) as *i64 1083 var zf: i64 = 0 1084 while zf < nc { cofam[zf] = 0; fmerge[zf] = 0 - 1; zf = zf + 1 } 1085 var merged_n: i64 = 0 1086 var fa: i64 = 0 1087 while fa < nc { 1088 var nct: i64 = 0 1089 var rw2: i64 = 0 1090 while rw2 < rows { 1091 if assign[rw2] == fa { 1092 var tk: i64 = 0 1093 while tk < rmeta[rw2 * 8 + 3] { 1094 let sl6: i64 = rtok[rw2 * IC_MAXTOK + tk] 1095 let fb: i64 = slot_fam[sl6] 1096 if fb >= 0 { if fb != fa { 1097 if cofam[fb] == 0 { if nct < IC_MAXFAM { cotouch[nct] = fb; nct = nct + 1 } } 1098 cofam[fb] = cofam[fb] + 1 1099 } } 1100 tk = tk + 1 1101 } 1102 } 1103 rw2 = rw2 + 1 1104 } 1105 var bestb: i64 = 0 - 1 1106 var bestratio: i64 = 0 1107 var ct: i64 = 0 1108 while ct < nct { 1109 let fb2: i64 = cotouch[ct] 1110 var den3: i64 = cmeta[fa * 8 + 1] 1111 if cmeta[fb2 * 8 + 1] < den3 { den3 = cmeta[fb2 * 8 + 1] } 1112 var ratio: i64 = 0 1113 if den3 > 0 { ratio = (cofam[fb2] * 1000) / den3 } 1114 if ratio > bestratio { bestratio = ratio; bestb = fb2 } 1115 cofam[fb2] = 0 1116 ct = ct + 1 1117 } 1118 if IC_MERGE_ENABLED == 1 { if bestb >= 0 { if bestratio >= IC_MERGE_PERMIL { if fmerge[fa] < 0 { if fmerge[bestb] < 0 { 1119 fmerge[fa] = bestb 1120 merged_n = merged_n + 1 1121 } } } } } 1122 fa = fa + 1 1123 } 1124 // apply the merges, then recompute every family stat from the ASSIGNMENT so the table can never 1125 // disagree with who is actually in it 1126 if merged_n > 0 { 1127 var rm2: i64 = 0 1128 while rm2 < rows { 1129 if assign[rm2] >= 0 { if fmerge[assign[rm2]] >= 0 { assign[rm2] = fmerge[assign[rm2]] } } 1130 rm2 = rm2 + 1 1131 } 1132 var rc2: i64 = 0 1133 while rc2 < nc { 1134 cmeta[rc2 * 8 + 1] = 0 1135 cmeta[rc2 * 8 + 2] = 0 1136 cmeta[rc2 * 8 + 3] = 0 1137 cmeta[rc2 * 8 + 4] = 0 1138 cmeta[rc2 * 8 + 5] = 0 1139 rc2 = rc2 + 1 1140 } 1141 var rq: i64 = 0 1142 while rq < rows { 1143 let fq: i64 = assign[rq] 1144 if fq >= 0 { 1145 cmeta[fq * 8 + 1] = cmeta[fq * 8 + 1] + 1 1146 if rmeta[rq * 8 + 2] == 1 { cmeta[fq * 8 + 2] = cmeta[fq * 8 + 2] + 1 } 1147 if rmeta[rq * 8 + 1] > cmeta[fq * 8 + 3] { cmeta[fq * 8 + 3] = rmeta[rq * 8 + 1] } 1148 let eq2: i64 = rmeta[rq * 8 + 0] 1149 if eq2 > 0 { 1150 if cmeta[fq * 8 + 4] == 0 { cmeta[fq * 8 + 4] = eq2 } 1151 if eq2 < cmeta[fq * 8 + 4] { cmeta[fq * 8 + 4] = eq2 } 1152 if eq2 > cmeta[fq * 8 + 5] { cmeta[fq * 8 + 5] = eq2 } 1153 } 1154 } 1155 rq = rq + 1 1156 } 1157 var rp: i64 = 0 1158 while rp < nc { 1159 var pr2: i64 = cmeta[rp * 8 + 3] * cmeta[rp * 8 + 2] 1160 cmeta[rp * 8 + 6] = pr2 1161 rp = rp + 1 1162 } 1163 } 1164 1165 // DENSE PLACEMENT OF NOISE ROWS 1166 var dense_placed: i64 = 0 1167 var dense_model: i64 = 0 1168 if IC_DENSE_ENABLED == 1 { 1169 let g: *i64 = sys_mmap(8 * 64) as *i64 1170 if ppl_load(g, "knowledge/index/semppmi_v1.bin" as *u8) > 0 { dense_model = 1 } 1171 if dense_model == 1 { 1172 let frep: *i64 = sys_mmap(8 * IC_MAXFAM) as *i64 1173 var fz: i64 = 0 1174 while fz < nc { frep[fz] = 0 - 1; fz = fz + 1 } 1175 var rr3: i64 = 0 1176 while rr3 < rows { 1177 if assign[rr3] >= 0 { if frep[assign[rr3]] < 0 { frep[assign[rr3]] = rr3 } } 1178 rr3 = rr3 + 1 1179 } 1180 var topf: i64 = nc 1181 if topf > IC_DENSE_TOPF { topf = IC_DENSE_TOPF } 1182 let ftok: *i64 = sys_mmap(8 * IC_DENSE_TOPF * IC_MAXTOK) as *i64 1183 let ftokn: *i64 = sys_mmap(8 * IC_DENSE_TOPF) as *i64 1184 let ta: *i64 = sys_mmap(8 * IC_MAXTOK) as *i64 1185 var ff: i64 = 0 1186 while ff < topf { 1187 ftokn[ff] = 0 1188 let rp: i64 = frep[ff] 1189 if rp >= 0 { 1190 let sp: i64 = rmeta[rp * 8 + 5] 1191 var ep: i64 = sp 1192 var scn: i64 = 1 1193 while scn == 1 { 1194 if ep >= n { scn = 0 } else { if q[ep] == (IC_NL as u8) { scn = 0 } else { ep = ep + 1 } } 1195 } 1196 ftokn[ff] = ppl_tokenize_ids(g, ((q as i64) + sp) as *u8, ep - sp, ((ftok as i64) + ff * IC_MAXTOK * 8) as *i64, IC_MAXTOK) 1197 } 1198 ff = ff + 1 1199 } 1200 var nz: i64 = 0 1201 while nz < rows { 1202 if assign[nz] < 0 { 1203 let s1: i64 = rmeta[nz * 8 + 5] 1204 var e1: i64 = s1 1205 var sc2: i64 = 1 1206 while sc2 == 1 { 1207 if e1 >= n { sc2 = 0 } else { if q[e1] == (IC_NL as u8) { sc2 = 0 } else { e1 = e1 + 1 } } 1208 } 1209 let na: i64 = ppl_tokenize_ids(g, ((q as i64) + s1) as *u8, e1 - s1, ta, IC_MAXTOK) 1210 if na > 0 { 1211 var bestf: i64 = 0 - 1 1212 var bestsim: i64 = 0 1213 var gf: i64 = 0 1214 while gf < topf { 1215 if ftokn[gf] > 0 { 1216 let sim3: i64 = ppl_maxsim_idf(g, ta, na, ((ftok as i64) + gf * IC_MAXTOK * 8) as *i64, ftokn[gf]) 1217 if sim3 > bestsim { bestsim = sim3; bestf = gf } 1218 } 1219 gf = gf + 1 1220 } 1221 if bestf >= 0 { if bestsim >= IC_DENSE_MIN { 1222 assign[nz] = bestf 1223 dense_placed = dense_placed + 1 1224 } } 1225 } 1226 } 1227 nz = nz + 1 1228 } 1229 if dense_placed > 0 { 1230 var rc3: i64 = 0 1231 while rc3 < nc { 1232 cmeta[rc3 * 8 + 1] = 0 1233 cmeta[rc3 * 8 + 2] = 0 1234 cmeta[rc3 * 8 + 3] = 0 1235 rc3 = rc3 + 1 1236 } 1237 var rq2: i64 = 0 1238 while rq2 < rows { 1239 let fq2: i64 = assign[rq2] 1240 if fq2 >= 0 { 1241 cmeta[fq2 * 8 + 1] = cmeta[fq2 * 8 + 1] + 1 1242 if rmeta[rq2 * 8 + 2] == 1 { cmeta[fq2 * 8 + 2] = cmeta[fq2 * 8 + 2] + 1 } 1243 if rmeta[rq2 * 8 + 1] > cmeta[fq2 * 8 + 3] { cmeta[fq2 * 8 + 3] = rmeta[rq2 * 8 + 1] } 1244 } 1245 rq2 = rq2 + 1 1246 } 1247 var rp2: i64 = 0 1248 while rp2 < nc { 1249 cmeta[rp2 * 8 + 6] = cmeta[rp2 * 8 + 3] * cmeta[rp2 * 8 + 2] 1250 rp2 = rp2 + 1 1251 } 1252 } 1253 } 1254 } 1255 1256 let order: *i64 = sys_mmap(8 * IC_MAXROWS) as *i64 1257 var oi: i64 = 0 1258 while oi < nc { order[oi] = oi; oi = oi + 1 } 1259 var a1: i64 = 0 1260 while a1 < nc { 1261 var best: i64 = a1 1262 var a2: i64 = a1 + 1 1263 while a2 < nc { 1264 if cmeta[order[a2] * 8 + 6] > cmeta[order[best] * 8 + 6] { best = a2 } 1265 a2 = a2 + 1 1266 } 1267 let tmp: i64 = order[a1]; order[a1] = order[best]; order[best] = tmp 1268 a1 = a1 + 1 1269 } 1270 1271 // Counted from the ASSIGNMENT, not from the family table: every row is either in exactly one family or 1272 // it is noise, so clustered + noise must equal rows. Deriving noise from the family list instead (as the 1273 // union-find version did) let the two disagree without anything noticing. 1274 var clustered: i64 = 0 1275 var noise: i64 = 0 1276 let families: i64 = nc 1277 var ci2: i64 = 0 1278 while ci2 < rows { 1279 if assign[ci2] >= 0 { clustered = clustered + 1 } else { noise = noise + 1 } 1280 ci2 = ci2 + 1 1281 } 1282 1283 let ob: *u8 = sys_mmap(IC_OUTCAP) 1284 // The HTML page is rendered from the SAME pass as the JSON, not from a re-read of it. A page generated 1285 // by re-parsing this organ's own output would be a second instrument able to disagree with the first. 1286 let hb: *u8 = sys_mmap(IC_OUTCAP) 1287 let lbuf: *u8 = sys_mmap(IC_LBUF) 1288 var ho: i64 = 0 1289 var o2: i64 = ic_cat(ob, 0, "{\"organ\":\"nx_debtcluster\",\"verb\":\"" as *u8) 1290 if mode == IC_MODE_SELFTEST { o2 = ic_cat(ob, o2, "selftest" as *u8) } 1291 if mode == IC_MODE_RUN { o2 = ic_cat(ob, o2, "run" as *u8) } 1292 if mode == IC_MODE_PAGE { o2 = ic_cat(ob, o2, "page" as *u8) } 1293 if mode == IC_MODE_EVAL { o2 = ic_cat(ob, o2, "eval" as *u8) } 1294 o2 = ic_cat(ob, o2, "\",\"recommendations\":[" as *u8) 1295 1296 var emitted: i64 = 0 1297 // LABEL-FREE QUALITY SIGNAL, added because the 20-row gold set stayed FLAT through a regression that 1298 // was obvious by eye. This one moves: it counts how many emitted families are named after a real 1299 // source file rather than a phrase. It needs no labels, so it scales with the corpus. 1300 let lcnt: *i64 = sys_mmap(8 * IC_DICT) as *i64 1301 let ltouch: *i64 = sys_mmap(8 * IC_POSTCAP) as *i64 1302 var emitted_organ: i64 = 0 1303 var minpri: i64 = 0 - 1 1304 var ei: i64 = 0 1305 while ei < nc { 1306 if emitted < IC_EMIT { 1307 let c2: i64 = order[ei] 1308 if cmeta[c2 * 8 + 1] >= IC_MINFAM { if cmeta[c2 * 8 + 6] > 0 { 1309 var anchsl: i64 = cmeta[c2 * 8 + 0] 1310 // If the claiming anchor does not NAME anything real, relabel the family with the 1311 // organ-named token its members most widely share. Membership is unchanged -- only the 1312 // name. Falls back to the anchor when the family shares no organ name at all. 1313 if ic_is_organ(q, dh[anchsl * 6 + 1], dh[anchsl * 6 + 2], orgscratch) == 0 { 1314 var nlt: i64 = 0 1315 var mm: i64 = 0 1316 while mm < rows { 1317 if ic_in_fam(assign, assign2, mm, c2) == 1 { 1318 var tt: i64 = 0 1319 while tt < rmeta[mm * 8 + 3] { 1320 let slx: i64 = rtok[mm * IC_MAXTOK + tt] 1321 if dh[slx * 6 + 4] > 0 { 1322 if lcnt[slx] == 0 { if nlt < IC_POSTCAP { ltouch[nlt] = slx; nlt = nlt + 1 } } 1323 lcnt[slx] = lcnt[slx] + 1 1324 } 1325 tt = tt + 1 1326 } 1327 } 1328 mm = mm + 1 1329 } 1330 var bestn: i64 = 1 1331 var bestsl: i64 = 0 - 1 1332 var lz: i64 = 0 1333 while lz < nlt { 1334 let sy: i64 = ltouch[lz] 1335 // SHARE x SPECIFICITY, not share alone. Scoring by share alone produced 1336 // 'log' and 'import' (TWICE, so two families carried the SAME name) -- 1337 // both genuinely match a file on disk, so emitted_organ_anchored read 8/8 1338 // while the names got WORSE. I gamed my own metric. 1339 // LAW: A TOKEN CAN NAME A REAL THING AND STILL BE A USELESS NAME -- an 1340 // existence test is necessary and NOT sufficient; weight by how much the 1341 // token actually distinguishes this family from every other. 1342 if lcnt[sy] * dh[sy * 6 + 4] > bestn { 1343 if ic_is_organ(q, dh[sy * 6 + 1], dh[sy * 6 + 2], orgscratch) == 1 { bestn = lcnt[sy] * dh[sy * 6 + 4]; bestsl = sy } 1344 } 1345 lz = lz + 1 1346 } 1347 var lc2: i64 = 0 1348 while lc2 < nlt { lcnt[ltouch[lc2]] = 0; lc2 = lc2 + 1 } 1349 if bestsl >= 0 { anchsl = bestsl } 1350 } 1351 if ic_is_organ(q, dh[anchsl * 6 + 1], dh[anchsl * 6 + 2], orgscratch) == 1 { emitted_organ = emitted_organ + 1 } 1352 if emitted > 0 { o2 = ic_cat(ob, o2, "," as *u8) } 1353 o2 = ic_cat(ob, o2, "{\"rank\":" as *u8) 1354 o2 = ic_catn(ob, o2, emitted + 1) 1355 // ★THE LABEL IS THE ANCHOR ITSELF. Under anchor clustering the family's name is not INFERRED 1356 // from its members -- every member demonstrably CONTAINS this exact token -- so the name can 1357 // never drift from the thing it names. The previous scheme scored the highest-IDF tokens the 1358 // members happened to share and produced "blanket-rebuild|half-work|stale_sweep" for a family 1359 // whose actual subject was nx_stale_sweep: it was naming the WARNING TEXT inside the rows. 1360 var lo: i64 = 0 1361 if dh[anchsl * 6 + 2] < IC_LBUF - 2 { lo = ic_catsl(lbuf, lo, q, dh[anchsl * 6 + 1], dh[anchsl * 6 + 2]) } 1362 lbuf[lo] = 0 as u8 1363 o2 = ic_cat(ob, o2, ",\"label\":\"" as *u8) 1364 o2 = ic_cat(ob, o2, lbuf) 1365 // one HTML row, same numbers, same pass 1366 if ho < IC_OUTCAP - IC_MAGIC_2048 { 1367 ho = ic_cat(hb, ho, "<tr><td class='n'>" as *u8); ho = ic_catn(hb, ho, emitted + 1) 1368 ho = ic_cat(hb, ho, "</td><td><code>" as *u8); ho = ic_cat(hb, ho, lbuf) 1369 ho = ic_cat(hb, ho, "</code></td><td class='n'>" as *u8); ho = ic_catn(hb, ho, cmeta[c2 * 8 + 1]) 1370 ho = ic_cat(hb, ho, "</td><td class='n'>" as *u8); ho = ic_catn(hb, ho, cmeta[c2 * 8 + 2]) 1371 ho = ic_cat(hb, ho, "</td><td class='n'>" as *u8); ho = ic_catn(hb, ho, cmeta[c2 * 8 + 3]) 1372 ho = ic_cat(hb, ho, "</td><td class='n'>" as *u8); ho = ic_catn(hb, ho, cmeta[c2 * 8 + 7]) 1373 var spanh: i64 = 0 1374 if cmeta[c2 * 8 + 5] > cmeta[c2 * 8 + 4] { spanh = (cmeta[c2 * 8 + 5] - cmeta[c2 * 8 + 4]) / IC_DAY } 1375 ho = ic_cat(hb, ho, "</td><td class='n'>" as *u8); ho = ic_catn(hb, ho, spanh) 1376 ho = ic_cat(hb, ho, "</td><td class='n'>" as *u8); ho = ic_catn(hb, ho, cmeta[c2 * 8 + 6]) 1377 ho = ic_cat(hb, ho, "</td><td class='idx'>" as *u8) 1378 var hs: i64 = 0 1379 var hv: i64 = 0 1380 while hv < rows { 1381 if ic_in_fam(assign, assign2, hv, c2) == 1 { if hs < IC_IDS_PER_CLUSTER { 1382 if hs > 0 { ho = ic_cat(hb, ho, ", " as *u8) } 1383 ho = ic_catn(hb, ho, hv) 1384 hs = hs + 1 1385 } } 1386 hv = hv + 1 1387 } 1388 if cmeta[c2 * 8 + 1] > hs { ho = ic_cat(hb, ho, " <span class='muted'>(+" as *u8); ho = ic_catn(hb, ho, cmeta[c2 * 8 + 1] - hs); ho = ic_cat(hb, ho, " more)</span>" as *u8) } 1389 ho = ic_cat(hb, ho, "</td></tr>\n" as *u8) 1390 } 1391 o2 = ic_cat(ob, o2, "\",\"members\":" as *u8); o2 = ic_catn(ob, o2, cmeta[c2 * 8 + 1]) 1392 o2 = ic_cat(ob, o2, ",\"open\":" as *u8); o2 = ic_catn(ob, o2, cmeta[c2 * 8 + 2]) 1393 o2 = ic_cat(ob, o2, ",\"max_sev\":" as *u8); o2 = ic_catn(ob, o2, cmeta[c2 * 8 + 3]) 1394 o2 = ic_cat(ob, o2, ",\"scopes\":" as *u8); o2 = ic_catn(ob, o2, cmeta[c2 * 8 + 7]) 1395 var span: i64 = 0 1396 if cmeta[c2 * 8 + 5] > cmeta[c2 * 8 + 4] { span = (cmeta[c2 * 8 + 5] - cmeta[c2 * 8 + 4]) / IC_DAY } 1397 o2 = ic_cat(ob, o2, ",\"span_days\":" as *u8); o2 = ic_catn(ob, o2, span) 1398 o2 = ic_cat(ob, o2, ",\"priority\":" as *u8); o2 = ic_catn(ob, o2, cmeta[c2 * 8 + 6]) 1399 o2 = ic_cat(ob, o2, ",\"idx\":[" as *u8) 1400 var shown: i64 = 0 1401 var mv: i64 = 0 1402 while mv < rows { 1403 if ic_in_fam(assign, assign2, mv, c2) == 1 { if shown < IC_IDS_PER_CLUSTER { 1404 if shown > 0 { o2 = ic_cat(ob, o2, "," as *u8) } 1405 o2 = ic_catn(ob, o2, mv) 1406 shown = shown + 1 1407 } } 1408 mv = mv + 1 1409 } 1410 o2 = ic_cat(ob, o2, "],\"idx_listed\":" as *u8); o2 = ic_catn(ob, o2, shown) 1411 o2 = ic_cat(ob, o2, "}" as *u8) 1412 if minpri < 0 { minpri = cmeta[c2 * 8 + 6] } 1413 if cmeta[c2 * 8 + 6] < minpri { minpri = cmeta[c2 * 8 + 6] } 1414 emitted = emitted + 1 1415 } } 1416 } 1417 ei = ei + 1 1418 } 1419 if minpri < 0 { minpri = 0 } 1420 1421 o2 = ic_cat(ob, o2, "],\"rows\":" as *u8); o2 = ic_catn(ob, o2, rows) 1422 o2 = ic_cat(ob, o2, ",\"rows_capped\":" as *u8); o2 = ic_catn(ob, o2, st[1]) 1423 o2 = ic_cat(ob, o2, ",\"open_rows\":" as *u8); o2 = ic_catn(ob, o2, st[2]) 1424 o2 = ic_cat(ob, o2, ",\"families\":" as *u8); o2 = ic_catn(ob, o2, families) 1425 o2 = ic_cat(ob, o2, ",\"clustered_rows\":" as *u8); o2 = ic_catn(ob, o2, clustered) 1426 o2 = ic_cat(ob, o2, ",\"noise_rows\":" as *u8); o2 = ic_catn(ob, o2, noise) 1427 o2 = ic_cat(ob, o2, ",\"emitted\":" as *u8); o2 = ic_catn(ob, o2, emitted) 1428 o2 = ic_cat(ob, o2, ",\"headline_min_priority\":" as *u8); o2 = ic_catn(ob, o2, minpri) 1429 o2 = ic_cat(ob, o2, ",\"tokens_distinct\":" as *u8); o2 = ic_catn(ob, o2, st[3]) 1430 o2 = ic_cat(ob, o2, ",\"tokens_df_skipped\":" as *u8); o2 = ic_catn(ob, o2, st[4]) 1431 o2 = ic_cat(ob, o2, ",\"rows_shape_alt\":" as *u8); o2 = ic_catn(ob, o2, st[5]) 1432 o2 = ic_cat(ob, o2, ",\"linker\":\"anchor-scope-then-organ\",\"dense_model_loaded\":" as *u8); o2 = ic_catn(ob, o2, dense_model) 1433 o2 = ic_cat(ob, o2, ",\"overlap_added\":" as *u8); o2 = ic_catn(ob, o2, overlap_added) 1434 o2 = ic_cat(ob, o2, ",\"dense_placed\":" as *u8); o2 = ic_catn(ob, o2, dense_placed) 1435 o2 = ic_cat(ob, o2, ",\"dense_min\":" as *u8); o2 = ic_catn(ob, o2, IC_DENSE_MIN) 1436 o2 = ic_cat(ob, o2, ",\"emitted_organ_anchored\":" as *u8); o2 = ic_catn(ob, o2, emitted_organ) 1437 o2 = ic_cat(ob, o2, ",\"anchors_considered\":" as *u8); o2 = ic_catn(ob, o2, ncand) 1438 // An anchor rejected for breadth is a THEME, not a family. Counted so the exclusion is never silent. 1439 o2 = ic_cat(ob, o2, ",\"anchors_too_broad\":" as *u8); o2 = ic_catn(ob, o2, anchors_too_broad) 1440 o2 = ic_cat(ob, o2, ",\"anchor_df_max\":" as *u8); o2 = ic_catn(ob, o2, anchor_dfmax) 1441 o2 = ic_cat(ob, o2, ",\"sim_join_unused\":" as *u8); o2 = ic_catn(ob, o2, simjoin) 1442 o2 = ic_cat(ob, o2, ",\"minpts\":" as *u8); o2 = ic_catn(ob, o2, IC_MINPTS) 1443 o2 = ic_cat(ob, o2, ",\"pairs_ge250\":" as *u8); o2 = ic_catn(ob, o2, st[6]) 1444 o2 = ic_cat(ob, o2, ",\"pairs_ge_join\":" as *u8); o2 = ic_catn(ob, o2, st[7]) 1445 o2 = ic_cat(ob, o2, ",\"pairs_ge500\":" as *u8); o2 = ic_catn(ob, o2, st[8]) 1446 o2 = ic_cat(ob, o2, ",\"effort_measured\":-1" as *u8) 1447 o2 = ic_cat(ob, o2, ",\"plane_declared\":" as *u8); o2 = ic_catn(ob, o2, dbf[0]) 1448 o2 = ic_cat(ob, o2, ",\"plane_found\":" as *u8); o2 = ic_catn(ob, o2, dbf[1]) 1449 var complete: i64 = 1 1450 if dbf[1] < dbf[0] { complete = 0 } 1451 o2 = ic_cat(ob, o2, ",\"coverage_complete\":" as *u8); o2 = ic_catn(ob, o2, complete) 1452 o2 = ic_cat(ob, o2, ",\"tier\":\"AICL-L2-evidence-capped\",\"tier_note\":\"L1 correlation + L2 derived summary. NOT L3+: no diagnosis is produced and none is claimed. Dense-embedding fusion (nx_recall_dense PPMI / nx_recall_fuse RRF 7/7) is a DECLARED GAP, not wired here.\"" as *u8) 1453 1454 // ── SELFTEST VERDICT. The teeth run against the clusters the real pipeline just produced. 1455 if is_self == 1 { 1456 var pass: i64 = 0 1457 var total: i64 = 0 1458 // T1 rows 0,1 share nx_mgmt_api_core -> MUST be the same family 1459 total = total + 1 1460 var t1: i64 = 0 1461 if assign[0] >= 0 { if assign[0] == assign[1] { t1 = 1; pass = pass + 1 } } 1462 // T2 ★NEGATIVE CONTROL: rows 0/1 vs 2/3 are near-identical SYMPTOM TEXT in DISJOINT organs. 1463 // They MUST NOT merge. Under anchor clustering this is structural rather than threshold-dependent: 1464 // row 2 does not CONTAIN nx_mgmt_api_core, so no setting of any parameter can put it in that 1465 // family. The tooth is kept anyway -- a guard is re-aimed, never deleted, and it now also catches 1466 // an anchor accidentally being built from a non-identifier token. 1467 total = total + 1 1468 var t2: i64 = 0 1469 // NON-VACUOUS BY CONSTRUCTION: it requires row 0 to actually BE in a family first. Written as a 1470 // bare inequality it read -1 != -1 as false when nothing clustered, i.e. the control's result was 1471 // decided by whether the clusterer ran at all rather than by what it decided. 1472 if assign[0] >= 0 { if assign[0] != assign[2] { t2 = 1; pass = pass + 1 } } 1473 // T3 rows 2,3 share nx_vault_walker_svc -> MUST be the same family 1474 total = total + 1 1475 var t3: i64 = 0 1476 if assign[2] >= 0 { if assign[2] == assign[3] { t3 = 1; pass = pass + 1 } } 1477 // T4 the unrelated row MUST be noise: it shares no identifier with anything, so it is UNASSIGNED 1478 total = total + 1 1479 var t4: i64 = 0 1480 if assign[4] < 0 { t4 = 1; pass = pass + 1 } 1481 // T5 the noise class is non-empty -- a clusterer that absorbs everything has no noise class at all 1482 total = total + 1 1483 var t5: i64 = 0 1484 if noise >= 1 { t5 = 1; pass = pass + 1 } 1485 // T6 IDF is monotonic: a token in fewer rows must outrank a token in more 1486 total = total + 1 1487 var t6: i64 = 0 1488 if ic_log2q(2) < ic_log2q(9) { if ic_log2q(9) < ic_log2q(40) { t6 = 1; pass = pass + 1 } } 1489 // T7 exactly two families were formed (mgmt pair + vault pair), not one blob and not four singletons 1490 total = total + 1 1491 var t7: i64 = 0 1492 if families == 2 { t7 = 1; pass = pass + 1 } 1493 o2 = ic_cat(ob, o2, ",\"selftest\":{\"T1_same_organ_joins\":" as *u8); o2 = ic_catn(ob, o2, t1) 1494 o2 = ic_cat(ob, o2, ",\"T2_NEGATIVE_CONTROL_disjoint_organs_stay_apart\":" as *u8); o2 = ic_catn(ob, o2, t2) 1495 o2 = ic_cat(ob, o2, ",\"T3_second_family_joins\":" as *u8); o2 = ic_catn(ob, o2, t3) 1496 o2 = ic_cat(ob, o2, ",\"T4_unrelated_row_is_noise\":" as *u8); o2 = ic_catn(ob, o2, t4) 1497 o2 = ic_cat(ob, o2, ",\"T5_noise_class_nonempty\":" as *u8); o2 = ic_catn(ob, o2, t5) 1498 o2 = ic_cat(ob, o2, ",\"T6_idf_monotonic\":" as *u8); o2 = ic_catn(ob, o2, t6) 1499 o2 = ic_cat(ob, o2, ",\"T7_exactly_two_families\":" as *u8); o2 = ic_catn(ob, o2, t7) 1500 o2 = ic_cat(ob, o2, ",\"pass\":" as *u8); o2 = ic_catn(ob, o2, pass) 1501 o2 = ic_cat(ob, o2, ",\"total\":" as *u8); o2 = ic_catn(ob, o2, total) 1502 o2 = ic_cat(ob, o2, "}" as *u8) 1503 if pass == total { o2 = ic_cat(ob, o2, ",\"verdict\":\"GREEN\"}\n" as *u8) } 1504 else { o2 = ic_cat(ob, o2, ",\"verdict\":\"RED\"}\n" as *u8) } 1505 sys_write(1, ob, o2) 1506 if pass == total { sys_exit(0); return 0 } 1507 sys_exit(1); return 1 1508 } 1509 1510 if complete == 1 { o2 = ic_cat(ob, o2, ",\"verdict\":\"COMPLETE\"" as *u8) } 1511 else { o2 = ic_cat(ob, o2, ",\"verdict\":\"PARTIAL\",\"reason\":\"plane load returned fewer rows than the plane declares -- some rows were never clustered\"" as *u8) } 1512 // EVAL. B-cubed against the hand-verified ground truth, restricted to the labeled rows -- the standard 1513 // way to score a PARTIAL gold set: judge the system's clustering AS RESTRICTED TO the items you labelled. 1514 var b3p: i64 = 0 - 1 1515 var b3r: i64 = 0 - 1 1516 var b3f: i64 = 0 - 1 1517 var gt_n: i64 = 0 1518 var gt_f: i64 = 0 1519 var gt_c: i64 = 0 1520 var do_eval: i64 = 0 1521 if mode == IC_MODE_EVAL { do_eval = 1 } 1522 // The PAGE scores itself too: a dashboard that cannot state its own accuracy is asking to be trusted 1523 // rather than checked, and this one is published where nobody can see the gate. 1524 if mode == IC_MODE_PAGE { do_eval = 1 } 1525 if do_eval == 1 { 1526 let gtid: *i64 = sys_mmap(8 * IC_GTMAX) as *i64 1527 let gtfam: *i64 = sys_mmap(8 * IC_GTMAX) as *i64 1528 let gtrow: *i64 = sys_mmap(8 * IC_GTMAX) as *i64 1529 let gtn: i64 = ic_gt_load(gtid, gtfam) 1530 var gfound: i64 = 0 1531 var gi: i64 = 0 1532 while gi < gtn { 1533 gtrow[gi] = 0 - 1 1534 var rr2: i64 = 0 1535 while rr2 < rows { 1536 if rmeta[rr2 * 8 + 0] == gtid[gi] { gtrow[gi] = rr2; rr2 = rows } else { rr2 = rr2 + 1 } 1537 } 1538 if gtrow[gi] >= 0 { gfound = gfound + 1 } 1539 gi = gi + 1 1540 } 1541 // NON-VACUITY: a ruler that resolved none of its own gold rows must NOT report a score. Without this 1542 // a re-seeded plane would silently yield 0/0 and print a confident-looking zero. 1543 if gfound == 0 { if mode == IC_MODE_EVAL { 1544 ic_puts("{\"organ\":\"nx_debtcluster\",\"verb\":\"eval\",\"verdict\":\"INSTRUMENT-BLIND\",\"reason\":\"ZERO ground-truth rows resolved in the plane -- nothing was compared, so NO quality score is emitted\"}\n" as *u8) 1545 sys_exit(2); return 2 1546 } } 1547 var psum: i64 = 0 1548 var rsum: i64 = 0 1549 var counted: i64 = 0 1550 var ai: i64 = 0 1551 while ai < gtn { 1552 if gtrow[ai] >= 0 { 1553 let ra: i64 = gtrow[ai] 1554 var inter: i64 = 0 1555 var ssize: i64 = 0 1556 var gsize: i64 = 0 1557 var bj: i64 = 0 1558 while bj < gtn { 1559 if gtrow[bj] >= 0 { 1560 let rb: i64 = gtrow[bj] 1561 var same_sys: i64 = 0 1562 if ic_share_fam(assign, assign2, ra, rb) == 1 { same_sys = 1 } 1563 if ra == rb { same_sys = 1 } 1564 var same_gt: i64 = 0 1565 if gtfam[ai] == gtfam[bj] { same_gt = 1 } 1566 if same_sys == 1 { ssize = ssize + 1 } 1567 if same_gt == 1 { gsize = gsize + 1 } 1568 if same_sys == 1 { if same_gt == 1 { inter = inter + 1 } } 1569 } 1570 bj = bj + 1 1571 } 1572 if ssize > 0 { psum = psum + (inter * 1000) / ssize } 1573 if gsize > 0 { rsum = rsum + (inter * 1000) / gsize } 1574 counted = counted + 1 1575 } 1576 ai = ai + 1 1577 } 1578 // b3p/b3r stay -1 when nothing was counted, so an unscored run can never print a confident zero 1579 if counted > 0 { b3p = psum / counted } 1580 if counted > 0 { b3r = rsum / counted } 1581 if b3p + b3r > 0 { b3f = (2 * b3p * b3r) / (b3p + b3r) } 1582 // SPLIT DIAGNOSTIC -- the loss is here, not in the noise pile (dense arm proved that). 1583 // NOTE: rows left unassigned all share assign=-1, so a family fully in noise counts as ONE 1584 // bucket, not N. That UNDERSTATES splitting; it can never overstate it. 1585 var fams_split: i64 = 0 1586 var worst_split: i64 = 0 1587 var gi2: i64 = 0 1588 while gi2 < gtn { 1589 var firstf: i64 = 1 1590 var gk: i64 = 0 1591 while gk < gi2 { if gtfam[gk] == gtfam[gi2] { firstf = 0 } gk = gk + 1 } 1592 if firstf == 1 { 1593 var dsys: i64 = 0 1594 var gm: i64 = 0 1595 while gm < gtn { 1596 if gtfam[gm] == gtfam[gi2] { if gtrow[gm] >= 0 { 1597 let sv: i64 = assign[gtrow[gm]] 1598 var seen2: i64 = 0 1599 var gn2: i64 = 0 1600 while gn2 < gm { 1601 if gtfam[gn2] == gtfam[gi2] { if gtrow[gn2] >= 0 { if assign[gtrow[gn2]] == sv { seen2 = 1 } } } 1602 gn2 = gn2 + 1 1603 } 1604 if seen2 == 0 { dsys = dsys + 1 } 1605 } } 1606 gm = gm + 1 1607 } 1608 if dsys > 1 { fams_split = fams_split + 1 } 1609 if dsys > worst_split { worst_split = dsys } 1610 } 1611 gi2 = gi2 + 1 1612 } 1613 gt_n = gtn 1614 gt_f = gfound 1615 // distinct gold families, COUNTED not declared -- the page states this number publicly and a 1616 // hardcoded one silently goes stale the moment the gold set grows (it did: said 5, was 12). 1617 var dz: i64 = 0 1618 while dz < gtn { 1619 if gtrow[dz] >= 0 { 1620 var seenf: i64 = 0 1621 var dw: i64 = 0 1622 while dw < dz { 1623 if gtrow[dw] >= 0 { if gtfam[dw] == gtfam[dz] { seenf = 1 } } 1624 dw = dw + 1 1625 } 1626 if seenf == 0 { gt_c = gt_c + 1 } 1627 } 1628 dz = dz + 1 1629 } 1630 o2 = ic_cat(ob, o2, ",\"eval\":{\"metric\":\"B-cubed per-element, chosen over ARI/NMI because family sizes span 152 rows down to 2 and pair-counting metrics are dominated by the largest cluster\",\"gt_declared\":" as *u8) 1631 o2 = ic_catn(ob, o2, gtn) 1632 o2 = ic_cat(ob, o2, ",\"gt_found\":" as *u8); o2 = ic_catn(ob, o2, gfound) 1633 o2 = ic_cat(ob, o2, ",\"gt_families\":" as *u8); o2 = ic_catn(ob, o2, gt_c) 1634 o2 = ic_cat(ob, o2, ",\"b3_precision_permil\":" as *u8); o2 = ic_catn(ob, o2, b3p) 1635 o2 = ic_cat(ob, o2, ",\"b3_recall_permil\":" as *u8); o2 = ic_catn(ob, o2, b3r) 1636 o2 = ic_cat(ob, o2, ",\"b3_f1_permil\":" as *u8); o2 = ic_catn(ob, o2, b3f) 1637 o2 = ic_cat(ob, o2, ",\"gold_families_split\":" as *u8); o2 = ic_catn(ob, o2, fams_split) 1638 o2 = ic_cat(ob, o2, ",\"worst_split_buckets\":" as *u8); o2 = ic_catn(ob, o2, worst_split) 1639 o2 = ic_cat(ob, o2, ",\"note\":\"A FLOOR, NOT A CORPUS SCORE -- see gt_declared/gt_found above for the exact denominator, against the whole plane. PRECISION asks whether rows grouped together really belong together; RECALL asks how much of each known family was found. HIGH PRECISION WITH LOW RECALL IS THE EXPECTED SIGNATURE of identifier-anchored grouping and is the EVIDENCE for adding the dense/PPMI layer, not a number to tune away.\"}" as *u8) 1640 } 1641 o2 = ic_cat(ob, o2, ",\"note\":\"A FAMILY IS A CLAIM, NOT A VERDICT. It says these rows describe one problem and that one act may close several. Confirm by reading the members before scheduling. noise_rows is the count that REFUSED to cluster and is never folded into a family. effort_measured=-1 because the plane records file-time, not close-time -- it is NOT zero and is NOT in the priority product.\"}\n" as *u8) 1642 1643 // ── PAGE. Written temp-then-rename so a reader never sees a half-page, and so a failed render leaves 1644 // the previous good page in place rather than truncating it to nothing. 1645 if mode == IC_MODE_PAGE { 1646 let pbuf: *u8 = sys_mmap(IC_MAGIC_16384) 1647 let tmpp: *u8 = sys_mmap(IC_MAGIC_1024) 1648 var tp: i64 = ic_cat(tmpp, 0, outfile) 1649 tp = ic_cat(tmpp, tp, ".tmp" as *u8) 1650 tmpp[tp] = 0 as u8 1651 let fd: i64 = sys_openat_wr(tmpp, IC_WRMODE) 1652 if fd < 0 { 1653 ic_puts("{\"organ\":\"nx_debtcluster\",\"verdict\":\"PAGE-WRITE-FAILED\",\"reason\":\"cannot open the temp page for write -- the PREVIOUS page is left intact, nothing was truncated\",\"path\":\"" as *u8) 1654 ic_puts(tmpp) 1655 ic_puts("\"}\n" as *u8) 1656 sys_exit(3); return 3 1657 } 1658 // A FULL document, not a fragment. The /code pages are complete self-contained HTML with an inline 1659 // design system; a fragment dropped into that tree serves 200 and renders UNSTYLED, which reads as 1660 // broken rather than as new. Variables mirror nx_codewiki's palette (light + dark) so this page is 1661 // native to the wiki rather than bolted onto it. 1662 var po: i64 = 0 1663 po = ic_cat(pbuf, po, "<!doctype html>\n<html lang='en'><head><meta charset='utf-8'><meta name='viewport' content='width=device-width,initial-scale=1'>\n<title>work families - nishi code wiki</title>\n<style>\n" as *u8) 1664 po = ic_cat(pbuf, po, ":root{--bg:#ffffff;--sf:#f6f8fb;--sf2:#eef1f6;--br:#dfe4ec;--ink:#232936;--mut:#68748a;--acc:#1a5dc8;--acc2:#2e8555;--hd:#111726;--hov:#f0f4fa}\n" as *u8) 1665 po = ic_cat(pbuf, po, "@media(prefers-color-scheme:dark){:root{--bg:#0d1017;--sf:#131826;--sf2:#1a2132;--br:#252d40;--ink:#dbe2ec;--mut:#8a94a8;--acc:#5aa7f0;--acc2:#86c793;--hd:#eef2f8;--hov:#141a28}}\n" as *u8) 1666 po = ic_cat(pbuf, po, "*{box-sizing:border-box}body{font-family:system-ui,'Segoe UI',sans-serif;margin:0;background:var(--bg);color:var(--ink);display:grid;grid-template-columns:252px minmax(0,1fr);min-height:100vh}\n" as *u8) 1667 po = ic_cat(pbuf, po, "a{color:var(--acc);text-decoration:none}a:hover{text-decoration:underline}\n" as *u8) 1668 po = ic_cat(pbuf, po, "nav.sb{background:var(--sf);border-right:1px solid var(--br);padding:14px 12px;position:sticky;top:0;height:100vh;overflow-y:auto;font-size:.9em}\n" as *u8) 1669 po = ic_cat(pbuf, po, ".sb .brand{display:block;font-weight:700;color:var(--hd);font-size:1.02em;margin:2px 6px 12px}\n" as *u8) 1670 po = ic_cat(pbuf, po, ".sb a.it{display:block;padding:4px 8px;border-radius:6px;color:var(--ink)}.sb a.it:hover{background:var(--sf2);text-decoration:none;color:var(--hd)}\n" as *u8) 1671 po = ic_cat(pbuf, po, ".sb .sec{margin:14px 6px 4px;font-size:.74em;letter-spacing:.09em;text-transform:uppercase;color:var(--mut)}\n" as *u8) 1672 po = ic_cat(pbuf, po, "main{padding:22px 30px 40px;max-width:1080px;min-width:0}\n" as *u8) 1673 po = ic_cat(pbuf, po, ".crumb{font-size:.85em;color:var(--mut);margin-bottom:10px}.crumb a{color:var(--mut)}\n" as *u8) 1674 po = ic_cat(pbuf, po, "h1{font-size:1.55em;color:var(--hd);margin:2px 0 6px;letter-spacing:-.01em}\n" as *u8) 1675 po = ic_cat(pbuf, po, "code{font-family:ui-monospace,'Cascadia Code',Consolas,monospace;font-size:.86em}\n" as *u8) 1676 po = ic_cat(pbuf, po, "p.muted,.muted{color:var(--mut)}p{line-height:1.5;max-width:78ch}\n" as *u8) 1677 po = ic_cat(pbuf, po, "table{border-collapse:collapse;width:100%;margin:14px 0;font-size:.9em}\n" as *u8) 1678 po = ic_cat(pbuf, po, "th{text-align:left;color:var(--mut);font-weight:600;font-size:.82em;letter-spacing:.04em;text-transform:uppercase;border-bottom:1px solid var(--br);padding:6px 9px}\n" as *u8) 1679 po = ic_cat(pbuf, po, "td{border-bottom:1px solid var(--br);padding:7px 9px;vertical-align:top}tr:hover td{background:var(--hov)}\n" as *u8) 1680 po = ic_cat(pbuf, po, "th.n,td.n{text-align:right;font-variant-numeric:tabular-nums;white-space:nowrap}\n" as *u8) 1681 po = ic_cat(pbuf, po, "td.idx{color:var(--mut);font-family:ui-monospace,Consolas,monospace;font-size:.82em}\n" as *u8) 1682 po = ic_cat(pbuf, po, "tr:first-child td{}\ntbody tr:nth-child(1) td{font-weight:500}\n" as *u8) 1683 po = ic_cat(pbuf, po, "@media(max-width:820px){body{display:block}nav.sb{position:static;height:auto;border-right:0;border-bottom:1px solid var(--br)}}\n" as *u8) 1684 po = ic_cat(pbuf, po, "</style></head><body>\n" as *u8) 1685 po = ic_cat(pbuf, po, "<nav class='sb'><a class='brand' href='/code/'>nishi code wiki</a>\n<div class='sec'>backlog</div>\n" as *u8) 1686 po = ic_cat(pbuf, po, "<a class='it' href='/code/families'>Work families</a>\n<a class='it' href='/code/findings'>Findings</a>\n<a class='it' href='/code/debt'>Debt taxonomy</a>\n<a class='it' href='/code/learnings'>Learnings</a>\n" as *u8) 1687 po = ic_cat(pbuf, po, "<div class='sec'>wiki</div>\n<a class='it' href='/code/'>Overview</a>\n<a class='it' href='/code/search'>Search</a>\n<a class='it' href='/code/ask'>Ask</a>\n</nav>\n<main>\n" as *u8) 1688 po = ic_cat(pbuf, po, "<div class='crumb'><a href='/code/'>code wiki</a> / work families</div>\n" as *u8) 1689 po = ic_cat(pbuf, po, "<h1>work families &mdash; what the debt ledger is actually asking for</h1>\n" as *u8) 1690 po = ic_cat(pbuf, po, "<p class='muted'>The ledger is a flat, append-only list sorted by severity, so it answers <em>what screams loudest</em> and never <em>what should I build</em>. This page groups rows that describe ONE problem, so a seat can pick the act that closes the most rows instead of the loudest single row. A family is the set of rows that all literally name the same thing &mdash; the <em>anchor</em>. Membership is a property each row holds on its own, so unrelated rows can never be chained together through a third, and the family's name is the anchor rather than a guess about it.</p>\n" as *u8) 1691 po = ic_cat(pbuf, po, "<table><tr><th class='n'>#</th><th>family</th><th class='n'>rows</th><th class='n'>open</th><th class='n'>sev</th><th class='n'>scopes</th><th class='n'>days</th><th class='n'>priority</th><th>row idx</th></tr>\n" as *u8) 1692 sys_write(fd, pbuf, po) 1693 sys_write(fd, hb, ho) 1694 po = 0 1695 po = ic_cat(pbuf, po, "</table>\n<p class='muted'>priority = max_sev &times; open_rows. Both terms are measured from the plane. distinct_scopes was removed from the product because under anchor grouping it rewards a GENERIC anchor touching many lanes over a specific one, which is backwards for deciding what to build; it is shown as context instead. <strong>Effort is deliberately absent</strong>: the plane records when a row was FILED, never when it was CLOSED, so an effort term here would be invented rather than measured.</p>\n" as *u8) 1696 po = ic_cat(pbuf, po, "<p class='muted'>Coverage: <strong>" as *u8); po = ic_catn(pbuf, po, rows) 1697 po = ic_cat(pbuf, po, "</strong> rows scanned, <strong>" as *u8); po = ic_catn(pbuf, po, st[2]) 1698 po = ic_cat(pbuf, po, "</strong> open, <strong>" as *u8); po = ic_catn(pbuf, po, families) 1699 po = ic_cat(pbuf, po, "</strong> families found covering <strong>" as *u8); po = ic_catn(pbuf, po, clustered) 1700 po = ic_cat(pbuf, po, "</strong> rows, and <strong>" as *u8); po = ic_catn(pbuf, po, noise) 1701 po = ic_cat(pbuf, po, "</strong> rows that REFUSED to cluster (reported, never folded into a family &mdash; most debt genuinely is singletons). Showing the top " as *u8); po = ic_catn(pbuf, po, emitted) 1702 po = ic_cat(pbuf, po, "; a recommendation list longer than 5&ndash;8 is a list again. Grouped by shared anchor over " as *u8); po = ic_catn(pbuf, po, ncand) 1703 po = ic_cat(pbuf, po, " candidate anchors, of which " as *u8); po = ic_catn(pbuf, po, anchors_too_broad) 1704 po = ic_cat(pbuf, po, " were rejected as too broad to name a work item" as *u8) 1705 po = ic_cat(pbuf, po, ".</p>\n" as *u8) 1706 if b3p >= 0 { 1707 po = ic_cat(pbuf, po, "<p class='muted'><strong>Measured quality of this grouping.</strong> B-cubed against " as *u8); po = ic_catn(pbuf, po, gt_f) 1708 po = ic_cat(pbuf, po, " hand-read rows across " as *u8); po = ic_catn(pbuf, po, gt_c) 1709 po = ic_cat(pbuf, po, " known families: precision <strong>" as *u8); po = ic_catn(pbuf, po, b3p) 1710 po = ic_cat(pbuf, po, "&permil;</strong>, recall <strong>" as *u8); po = ic_catn(pbuf, po, b3r) 1711 po = ic_cat(pbuf, po, "&permil;</strong>, F1 <strong>" as *u8); po = ic_catn(pbuf, po, b3f) 1712 po = ic_cat(pbuf, po, "&permil;</strong>. This is a FLOOR, not a corpus score &mdash; " as *u8); po = ic_catn(pbuf, po, gt_n) 1713 po = ic_cat(pbuf, po, " labelled rows against the whole plane. High precision with lower recall is the expected signature of grouping by shared identifier: what it groups does belong together, but it misses members that describe the same problem in different words.</p>\n" as *u8) 1714 } 1715 if complete == 1 { po = ic_cat(pbuf, po, "<p class='muted'>plane_declared == plane_found, so every row the plane declares was examined.</p>\n" as *u8) } 1716 else { po = ic_cat(pbuf, po, "<p><strong>PARTIAL:</strong> the plane load returned fewer rows than the plane declares &mdash; some rows were never clustered, and this page is a sample, not the set.</p>\n" as *u8) } 1717 po = ic_cat(pbuf, po, "<p class='muted'><strong>A family is a claim, not a verdict.</strong> It says these rows look like one problem and that one act may close several &mdash; confirm by reading the members before scheduling. Generated by <code>nx_debtcluster page</code> (AICL L2, evidence-capped: correlation and summary only, no diagnosis is produced and none is claimed).</p>\n" as *u8) 1718 po = ic_cat(pbuf, po, "</main></body></html>\n" as *u8) 1719 sys_write(fd, pbuf, po) 1720 sys_close(fd) 1721 sys_renameat(tmpp, outfile) 1722 } 1723 1724 sys_write(1, ob, o2) 1725 sys_exit(0); return 0 1726}