Sovereign Deep-Research — DRBench Web-Corpus Scorecard

A measured, full-scale, zero-egress evaluation of the Nishi research engine on the public-web half of ServiceNow DRBench. Every source fetched over our own TLS 1.3; every insight judged by our own deterministic model.

Generated 2026-07-21 · workstream deepresearch-scale · RACI A=pm, R=librarian+researcher · evidence: NAS seg-store

1 · Sovereign fetchability of cited web sources

DRBench cites 27 public web URLs (contexts/urls.json) as support for its external-fact insights. We fetched each with the sovereign fetcher nx_https_get_cli (own TLS 1.3 + Mozilla trust store, no browser, no third-party egress) and recorded the raw HTTP status.

Finding: a zero-egress sovereign agent reaches ≈59% of DRBench's cited web sources unaided; ~26% actively bot-block automated TLS clients (grocerydive, bcg, healthcaredive, iea, …). Following the 2 redirects lifts reach to ≈67%. This is the honest ceiling on the web half before any private data.

2 · Source-grounded recall of the sovereign judge

For each fetchable (200) source we ran the live nx_dr_semjudge (102,318-word PPMI late-interaction count model, integer & deterministic) against all 81 groundtruth insights — the source's own 3 are true positives, the other 78 are built-in negatives. Discrimination is reported at matched precision (threshold just above the top negative), reproducing our prior methodology.

Reading: on real fetched web text, the sovereign semantic judge places a source's own supported insight in the top-10 of 81 candidates 79% of the time (top-3 48%, mean rank 7.3) — genuine source-grounded discrimination, deterministic and zero-egress. Global "matched precision" is a harsh 46–69‰ here by construction: cross-source insights within the same industry/domain are topical near-positives, not clean negatives — so rank, not matched-precision, is the honest headline.

Denominator = 48 true (source, insight) pairs across 16 fetched sources × 81 insights = 1,296 judgments. Lexical = djb2 token containment; semantic = PPMI maxsim.

3 · Honest scope & known limits

• Web half only. The private enterprise half (Docker per-task images) is not run — most DRBench insights are private by construction.
• Count-model judge, not a neural cross-encoder. Five representation refinements (vocab, IDF, learned fusion, trained dense, common-component removal) were previously measured and rejected; the standing 306‰ judge ceiling is feature-bound, not tuning-bound.
• No redirect-following yet (2 sources unrecovered); no HTML main-text extraction (crude tag-strip only).
• Judge is per-call model-reloaded (41 MB × N) — an efficiency rung, not a correctness issue.

4 · Next rungs (opportunity-cost ordered)

1. Neural cross-encoder judge (DR-5) — the only lever measured to move the judge ceiling; the real SOTA path.
2. Docker enterprise private-half — unlocks the true DRBench leaderboard number (the majority of insights).
3. Batch judge (load model once) — ~N× faster full-corpus runs.
4. Redirect-following + HTML main-text extraction — cheap fetchability + recall gains.

Provenance: fetched + judged on the NAS (west_nas), sovereign toolchain, results in ~/drbench + seg-store. Reproducible: fetch_all.sh → judge_disc.sh → agg.awk.