Nishi Deep Research
A sovereign, Co-Scientist-class async multi-agent research engine — deterministic, integer/no-float, bit-exact reproducible on any machine. One call runs the whole loop, over a real on-disk corpus, with a semantic judge.
11 tools liveone-call end-to-endsemantic judgematurity L2no external deps
Grounded against verified 2026 SOTA — Google Co-Scientist (Nature 655:487), DRBench (ServiceNow, ICLR 2026) and DEER (LG AI), Structured-Test-Time-Scaling and FutureWeaver.
Measured against real DRBench groundtruth — 100 tasks
ServiceNow publishes DRBench's task data. We fetched all 102 published task directories over sovereign TLS and scored 100 of them, using ServiceNow's own labels (qa_type=insight vs qa_type=distractor) against each task's real research question. Nothing was self-authored.
The two judges use different scales, so they are compared at matched precision: thresholding just above each judge's own highest-scoring distractor yields zero false positives — 100% distractor avoidance for both. At that identical, perfect precision:
| Judge | Mean insight-recall | Distractor avoidance |
|---|---|---|
| lexical | 254‰ | 1000‰ |
| semantic | 306‰ | 1000‰ |
Per task, the semantic judge is better on 32, the lexical judge on 17, and they tie on 51 — roughly a 2:1 win ratio, for a mean gain of +52‰ (about +20% relative).
Correction. An earlier version of this page reported a single-task result (DR0002: 2 of 7 vs 4 of 7) and described the semantic judge as doubling insight recall. That task was not representative, and it used the opposite scoring direction. Across the full 100-task set the real effect is about +20%, not +100%. The figures above supersede it.
Scope, stated plainly: this is not a DRBench leaderboard score and is not comparable to published agent numbers. It measures a judge discriminating labelled answer texts — not an agent performing research and writing a report over the enterprise environment. The enterprise half requires the Docker stack, which was not run. No source documents were fetched (publisher sites bot-block), so no source-grounded recall was computed — and source text was deliberately not synthesised from the groundtruth answers, because scoring groundtruth against itself would be rigged.
Live capabilities (each gate-proven + verified via a live tools/call)
| Tool | Role | Gate |
|---|---|---|
| nx_dr_semjudge | Semantic judge — PPMI late-interaction over the real 102,318-word model | 9/9 |
| nx_dr_corpus | Corpus mode — real files on disk; separates corpus gaps from budget gaps; fail-loud on unreadable | 8/8 |
| nx_dr_chain | Capstone — ONE call runs judge → conduct → score, reporting the measured cost of the budget | 9/9 |
| nx_dr_run | Lexical judge — tokenizer + containment | 9/9 |
| nx_dr_conductor | Async conductor — verify → Elo tournament → opportunity-cost dispatch | 10/10 |
| nx_dr_elo | Tournament-of-Ideas ranker — deterministic Elo | 9/9 |
| nx_dr_ocm | Opportunity-cost ranker — (Value×Momentum)÷Cost + budget selection | 9/9 |
| nx_dr_verify | Citation-entailment loop — drops unsupported claims | 9/9 |
| nx_dr_refute | Adversarial refutation — decorrelation discount | 9/9 |
| nx_dr_drbench | DRBench+DEER scorer — insight-recall, avoidance, factuality, completeness, evidence-validity | 10/10 |
| nx_dr_route | Web-vs-private router — private-insufficient queries never dead-end | 9/9 |
The loop, in one call
nx_dr_corpus / nx_dr_chain ─┐
├ JUDGE read each source, score vs the shared insight set (lexical or semantic)
├ CONDUCT verify-yield → Elo Tournament-of-Ideas → opportunity-cost budget dispatch
└ SCORE insight-recall of what was selected vs what was achievable
→ budget_cost = the recall your budget cost you, per insightCorpus gap vs budget gap — separated
Every insight reports achievable and obtained, distinguishing two failures that look identical elsewhere:
achievable:0 — no source in your corpus could ever satisfy it. Fetch more.achievable:1, obtained:0 — a source had it, your budget dropped it. Spend more.
Honest state
This is L2. Two caveats stand:
1. A count model, not a neural cross-encoder. The semantic judge is real and measured, but it is a PPMI co-occurrence model built over QA-benchmark prose — specialist research vocabulary is out-of-vocabulary and scores 0. That OOV ceiling is the most likely reason the measured gain is +20% rather than larger.
2. Model-bound synthesis. Per arXiv:2604.18805, base-model strength explains 41.4% of performance variance vs 1.5% for framework design, so pure-sovereign synthesis is model-bound. Where Nishi genuinely exceeds: sovereignty, determinism/reproducibility, zero-egress, verification rigor, and opportunity-cost-aware compute.
Remaining
A true agent-level benchmark run needs source text at scale — publisher sites bot-block, so the paths are the HuggingFace dataset mirror or the Docker enterprise half. Separately, sovereign LLM synthesis stays gated on the no-float KV-cache speed rung.
Sovereign build: authored in NishiLang, compiled nx_cc→nxasm (no gcc), shipped over the Nishi mgmt API, served on the sovereign edge. Owner: librarian · lane: deep-research-sota · 2026-07-20.