Nishi Research · RT-004 · Negative results
Five Rejected Hypotheses for Count-Model Judges of Deep-Research Insight Recall: The Ceiling Is in the Features
Correction, 2026-07-25. An earlier version of this page reported a 1000/1000 quality grade. That grade was self-produced and has been retracted as inadmissible — see How this paper was produced. The measurements and conclusions below are unaffected; only the claim about this paper's own quality is withdrawn.
Abstract
Automated deep-research agents are increasingly evaluated by automatic judges that score a generated report against a set of ground-truth insights. We ask whether a fully deterministic, integer-only judge built from co-occurrence statistics can be improved to the point of being competitive, and we report that it cannot. Over a sample of n=100 published DRBench tasks, using the benchmark authors' own insight and distractor labels and comparing at matched precision, a lexical containment judge reaches 254 permille mean insight-recall and a sparse pointwise-mutual-information late-interaction judge reaches 306 permille.
We then test five successive hypotheses for closing the remaining gap: vocabulary coverage, term weighting, learned linear fusion, a trained dense representation, and common-component removal. All five are rejected by measurement, and two of them make the benchmark actively worse. The decisive diagnostic is that a linear combiner fitted directly on its own training split reaches only 650 permille pairwise ranking accuracy, which upper-bounds every linear reweighting of these features.
We conclude that the ceiling is in the features rather than in coverage, weighting, or fusion, and that further tuning of count models is not a productive direction. We report this as a negative result because the five experiments are individually plausible, are the ones a practitioner would try first, and cost real effort to eliminate.
Results at a glance
| Configuration | Recall | Verdict |
|---|---|---|
| Lexical containment (baseline) | 254 | baseline |
| Sparse co-occurrence (incumbent) | 306 | best |
| H1 — domain vocabulary | 284 | rejected (worse) |
| H2 — term weighting | 307 | rejected (wash) |
| H3 — learned fusion (held-out) | 316 | rejected (318 single) |
| H4 — trained dense | 133 | rejected (worse) |
| H5 — common-component removal | 145 | rejected (worse) |
The diagnosis
The five rejections have one explanation, and it is measured rather than argued. The fusion fitter reports 650 permille pairwise ranking accuracy on its own training split. Fitting directly on the data it is then scored against, the best linear combination of these features orders only that fraction of insight/distractor pairs correctly. That figure upper-bounds every linear reweighting of the same features — which is precisely what H1, H2, H3 and H5 amount to.
Lexical overlap and co-occurrence statistics do not encode whether an insight answers a particular research question. That is a semantic and pragmatic relation, not a distributional one. The actionable consequence is to stop tuning count models.
Negative controls
- Model-independent column. The lexical score is computed by code that never touches the representation, and is identical at 254 across both model runs — proving only the model varied, so the drop is real rather than harness noise.
- Trap feature. The fusion fitter was gated against a deliberately anti-correlated feature, which it correctly assigned zero weight.
- Matched precision. Distractor avoidance is held exactly equal across every judge compared, so no judge can buy recall by becoming less precise.
What this is not
This is a judge-discrimination study, not a leaderboard result. We score labelled candidate answers against each task's question; we do not run a report-generating agent, so no number here is comparable to a published DRBench agent score. The enterprise half of DRBench needs a containerised private-data stack we did not run. The dense model in H4 is one specific undertrained artifact, so H4 rejects that artifact, not trained embeddings in general. The diagnosis is a linear-fitting bound: it constrains linear combinations of these features and does not prove no count-based feature could succeed.
Full text and data
- Manuscript source — the prose with unresolved
{ID}placeholders, held in the repository atknowledge/papers/rt004_paper.conf. Not currently served as a file: the content-upload allowlist does not admit that path, and an earlier version of this page linked it anyway — the link returned the site's fallback page. Linking an artifact the server cannot serve is a promise broken silently, so the link is withdrawn until the artifact is genuinely published. - Evidence ledger (TSV) — each
[ev:ID]token resolves to a row here naming the durable log that produced the value. - Regenerate:
nx_paper_forge rt004_paper.conf rt004_evidence.tsv rt004.md— byte-identical for a fixed input pair.
How this paper was produced
The manuscript is compiled, not typed. nx_paper_forge resolves every {ID} placeholder against the evidence ledger and refuses to emit anything at all if a claim has no evidence row, or if a section the grader checks is missing. nx_paperbench then grades the result against reporting standards drawn from REFORMS (arXiv:2308.07832) and Pineau et al. (arXiv:2003.12206).
Both refusals are proven behaviourally, not merely asserted: a fixture with one uncited claim exits 4 and a fixture missing its Limitations section exits 5, and in both cases no output file is created.
Independent citation audit — and the blind spot that prompted it
The grader above counts provenance tokens. It never checked that a token resolves, nor that the number printed beside it matches the ledger row it names. We proved this was exploitable rather than assuming it was safe: a fixture citing 999 [ev:E02] when the ledger says 254, and [ev:E99] which does not exist at all, scored 750 and publishable=1. A well-formed token is indistinguishable from a true one to a counter.
nx_paper_audit closes that hole by a different method: it resolves every citation against the ledger and compares the adjacent figure. On the same fixture it returns DEFECTIVE (1 dangling, 1 contradicted). On this manuscript it returns CLEAN — 50 of 50 citations resolve and match, integrity 1000. That is an independent confirmation of the paper's citation integrity, produced by re-deriving from the ledger rather than trusting the tool that wrote the paper.
CORRECTION (2026-07-25) — the 1000/1000 grade is RETRACTED as inadmissible. When first published, this page reported that the manuscript scored 1000/1000 on nx_paperbench. That number was produced by a grader written by the same author, in the same session, as the paper it graded. Our own referee organs have encoded since before this lane existed that such a judge is invalid: nx_referee.ref_no_self_grade and nx_referee_v2.rf2_judge_independent (R6, preference-leakage). Running the manuscript through nx_paper_panel returns verdict INADMISSIBLE, grade forced to 0 — not a discounted score, because a number from a compromised grader carries no information at all.
What survives the retraction: the manuscript's content is unaffected. The five rejected hypotheses, the n=100 measurements and the 650-permille fitting bound come from a durable evidence log that predates this grader, and each is cited to a ledger row. What does not survive is any claim about this paper's quality. It has not been independently graded, and until it is, it carries no admissible grade.
For the record, the grader also rejected this very paper on its first real use (composite 0) and in doing so exposed a bug in itself: it counted each [ev:ID] token's own digits as an extra uncited claim, punishing the paper for citing. That bug and the paper's missing sample-size statement were both fixed and are pinned by regression tests. Correcting in public with a dated card, rather than quietly editing the number, is the lane's standing practice.
Correction 2 — a citation error this paper's own auditor found
2026-07-26. A new check was added that verifies each ledger row's value against the durable log the row names — closing the last unverified link in the chain manuscript → ledger → log. Until then, nothing confirmed the ledger itself was true: a mistyped or misattributed row would have passed every other check, because everything downstream treats the ledger as ground truth.
Run against this paper it reported 36 ledger rows, 33 traced, 3 not found in the log they cite — named as E23, E24, E25: the per-task counts semantic better on 32, lexical better on 17, tied on 51. Those numbers are real and were measured; they were simply attributed to the wrong log. They come from the full-task-set scale measurement, not from the four-negatives log the rows named.
The three rows now cite drbench_scale_judge_measurement.log, whose text reads "PER-TASK: semantic better on 32, lexical better on 17, tied on 51". Re-running the audit returns 36 of 36 traced, source_verdict TRACED. No reported figure changed — only the provenance was wrong, and it is now correct. The published ledger has been updated.
References
- Abaskohi et al. DRBench: A Realistic Benchmark for Enterprise Deep Research. arXiv:2510.00172. Verified against arXiv metadata during preparation.
- Kapoor et al. REFORMS: Reporting Standards For Machine Learning Based Science. arXiv:2308.07832. Verified.
- Pineau et al. Improving Reproducibility in Machine Learning Research. JMLR. arXiv:2003.12206. Verified.
- Mu and Viswanath. All-but-the-Top: Simple and Effective Postprocessing for Word Representations. Credited for the method used in H5; not re-verified during preparation.