nishi code wiki / research / autonomous AI scientists

Autonomous AI scientists: the bar, and how to beat it

SOTA census · compiled 2026-08-01 · 5 axes · ~60 sourced claims · 14 declared gaps

lineage: forked from research_rtrender parent domain: method slug: research_autosci

Claims are labelled SHIPPING / RESEARCH / ANNOUNCED / DEPRECATED. Gaps are declared UNVERIFIED rather than guessed.

HEADLINE 1 — the multi-agent scaffolding is not yet earning its keep on end-to-end rediscovery. On ResearchClawBench (40 tasks, 10 domains, scored 0-100 where 50 = matching the target paper), the best autonomous agent scores 21.5 — and that agent is Claude Code, a general-purpose coding agent, not a science system. The purpose-built entrant EvoScientist v0.1.1 scores 18.8, below the best bare frontier LLM driven through a thin harness (Claude-Opus-4.7, 20.7). A per-task oracle over 17 bare LLMs reaches 26.5 — higher than every scaffolded system tested. (arXiv:2606.07591v5, submitted 2026-05-28, revised 2026-07-03.) Elaborate agent graphs are currently losing to a good model with a good prompt.

HEADLINE 2 — any leaderboard delta under about 10 points is inside the noise floor. An audit of 168 benchmarks across nine domains, 34,285 tasks, found 8,819 tasks (25.7%) with major issues — ambiguous task design, execution-environment conflicts, incorrect ground truths. Filtering the flagged tasks shifts model rankings and raises average performance on SWE-bench Verified by 9.9% and Terminal-Bench 2 by 9.6%. (Wang et al., Stanford/Duke, arXiv:2605.26079, 2026-05-25.) That correction is larger than most reported model gaps. Treat sub-10-point wins as unmeasured until the tasks are audited.

⚠ Provenance downgrade — read before trusting this brief's confidence level. This brief was commissioned as a merge of two independent prior sweeps, with agreement between them treated as high confidence. Both source transcripts were 0 bytes, as were all 17 task outputs in that directory. The two-sweep cross-check could not be performed. What follows is a single fresh sweep (~18 web calls, primary sources fetched for both headline claims and every failure-mode paper). Confidence is correspondingly lower and every unresolved item is listed at the foot. This is the same failure the rtrender brief recorded: parallel subagents exhausted a shared search cap, were killed, and wrote 0-byte outputs with no completion notification. Silence is not progress, and an empty file is not a finding.

1. Google Co-Scientist — the reference architecture

SHIPPING to vetted labs only. A Supervisor agent allocates compute across a worker pool of six specialised agents operating on a shared context memory:

Agent	Role
Supervisor	Parses the goal into a research plan; queues and allocates resources to the workers
Generation	Literature exploration and self-play simulated scientific debate to produce initial hypotheses
Reflection	Acts as peer reviewer — correctness, novelty, quality critique
Ranking	Elo-rated tournament; hypotheses compete in pairwise simulated debates
Evolution	Refines top-ranked hypotheses: simplification, analogy, inconsistency repair
Proximity	Builds a similarity graph to cluster and de-duplicate the hypothesis pool
Meta-review	Synthesises tournament output into a research overview; feeds critique back into the next round

The loop is generate → debate → rank → evolve, iterated. The design claim is test-time compute scaling: quality rises monotonically with compute spent, and the Elo rating correlates with accuracy on GPQA diamond questions, which is what licenses Elo as a self-evaluation proxy. Built on Gemini 2.0. (Google Research blog, 2025-02-19, research.google/blog/accelerating-scientific-breakthroughs-with-an-ai-co-scientist/; preprint Towards an AI co-scientist, arXiv:2502.18864.)

Three wet-lab validated results

Domain	Result	Status
AML drug repurposing	KIRA6 (an IRE1α inhibitor), a candidate with no prior AML literature, inhibited KG-1 viability at clinically relevant concentrations; 18-fold selectivity for the KG-1a AML line over the non-malignant TK6 control	RESEARCH — in vitro only
Liver fibrosis	Proposed epigenetic targets; all suggested drugs showed significant anti-fibrotic activity in human hepatic organoids (p < 0.01). Vorinostat, already FDA-approved in oncology, cut a key TGFβ-induced response by 91%	RESEARCH — organoid, not in vivo
Antimicrobial resistance	Independently proposed that cf-PICIs (capsid-forming phage-inducible chromosomal islands) interact with diverse phage tails to spread across species — matching a collaborator's unpublished, experimentally validated finding	RESEARCH — recapitulation, not prospective

Publication: Accelerating scientific discovery with Co-Scientist, Nature, May 2026, DOI 10.1038/s41586-026-10644-y. Access model: no public API, no open weights — a Trusted Tester Program gated by application from research organisations. You cannot buy this, and you cannot run it. That matters more than the architecture: the architecture is published and reproducible, the access is not, and the reproducible part is the part that benchmarks poorly.

The honest read on the three results. All three are preclinical; none has entered a human trial. The cf-PICI result is a recapitulation of a finding the collaborators already had in hand — impressive as a validation protocol, but it is not a prospective discovery. The strongest claim the evidence supports is “generates hypotheses a domain expert judges worth the bench time, and sometimes they work” — which is genuinely valuable and much weaker than “AI made a discovery”.

2. The peer field

System	What it is	Status	Source (date)
Sakana AI Scientist v1 / v2	End-to-end idea → experiment → paper. v2 adds agentic tree search and drops per-template human scaffolding. v1 cost ~$15/paper	SHIPPING (open source)	arXiv:2504.08066 (2025-04); sakana.ai/ai-scientist/
FutureHouse Robin	Composes Crow/Falcon (literature) with Finch (data analysis); humans run the bench work. Proposed ripasudil (a ROCK inhibitor licensed for glaucoma) for dry AMD	SHIPPING (open source)	arXiv:2505.13400 (2025-05); Nature s41586-026-10652-y (2026)
Edison Scientific Kosmos	Successor to Robin built on a structured world model — a shared machine-readable state the agents read and write, rather than passing prose. Reads ~1,500 papers and executes analysis code per run; every conclusion traceable to code or citation. $200 per run	SHIPPING (commercial)	arXiv:2511.02824 (2025-11); edisonscientific.com/news/announcing-kosmos
Stanford Virtual Lab	An LLM Principal Investigator agent chairs a team of specialist agents through structured research meetings; human sets direction. Designed 92 candidate nanobodies for SARS-CoV-2 variants; several bound experimentally, two bound both new variants and the ancestral virus	RESEARCH	Nature s41586-025-09442-9 (2025)
Microsoft Discovery	Enterprise agentic R&D platform over Azure HPC + simulation. Produced a PFAS-free data-centre coolant prototype in ~200 hours	SHIPPING — GA at Build 2026 (2026-06); private preview Build 2025	azure.microsoft.com/en-us/blog/transforming-rd-with-agentic-ai-introducing-microsoft-discovery/
Anthropic Claude Science	An AI workbench rather than an autonomous scientist: connectors to lab systems, code execution, auditable artifacts and preserved research history. Follows Claude for Life Sciences (2025-10-20); NVIDIA BioNeMo toolkit integration	SHIPPING	anthropic.com/news/claude-science-ai-workbench (2026)
2026 open-source entrants	EvoScientist (self-evolving multi-agent with persistent memory — the only purpose-built system scored on ResearchClawBench), plus Agent Laboratory, URSA, MARVEL	RESEARCH	github.com/AI4Scientist/awesome-autoresearch

The architectural split worth noticing. The field has forked. Co-Scientist, Sakana and EvoScientist chase autonomy (the machine runs the loop). Kosmos, Virtual Lab and Claude Science chase auditability (the machine does the labour, a human holds the judgement, and every claim carries a traceable provenance link). Kosmos's structured world model and Claude Science's auditable artifacts are the same idea arrived at independently — shared verifiable state beats prose passed between agents. Given the benchmark evidence in §3 and the failure modes in §4, the auditability fork is the one with evidence behind it.

3. Benchmarks and current best scores

Every score below is the best reported for that benchmark as of the date given. Read them against Headline 2: differences under ~10 points are not established.

Benchmark	What it measures	Best score	Holder / date
ResearchClawBench	End-to-end rediscovery: 40 tasks, 10 domains, real papers hidden during eval; LLM peer-reviewer rubric, 0-100, 50 = paper parity	21.5	Claude Code (agent); Claude-Opus-4.7 20.7 (bare LLM); frontier oracle 26.5 — 2026-07-03
AstaBench	11 benchmarks, 2,400+ problems across the whole research process; controls for model cost and tool access — the methodological gold standard	see note	57 agents / 22 classes; arXiv:2510.21652 (2025-10-24), ICLR 2026
PaperBench	Replicate 20 ICML 2024 Spotlight/Oral papers from scratch; 8,316 individually gradable rubric criteria	21.0%	Claude 3.5 Sonnet (New) + open-source scaffold; below the human PhD baseline — 2025-04-02
MLE-bench	Kaggle-style ML engineering, medal rate	48.5%	ML-Master + DeepSeek-R1, any-medal, Lite split — leaderboard (secondary)
CORE-Bench	Reproduce a paper's results given its repository — strictly easier than PaperBench	26.7%	CORE-Agent + o4-mini High — 2025-04 (secondary)
LitQA2	Literature QA requiring full-text retrieval; part of AstaBench	see note	PaperQA2 reported superhuman precision, human-level accuracy; arXiv:2412.21154
BixBench	53+ real bioinformatics scenarios, ~300 open-answer questions	17%	Claude 3.5 Sonnet open-answer (GPT-4o 9%) — arXiv:2503.00096 (2025-03-01)

The pattern is the finding. Across five independently constructed benchmarks the best systems cluster at 17% to 27%. On the two benchmarks with an explicit human bar — PaperBench and ResearchClawBench — no system reaches it. AstaBench's authors put it plainly: “AI remains far from solving the challenge of science research assistance.” Nothing in this field is near parity with a competent graduate student working end to end. AstaBench is also the only suite that controls for cost, which is why an expensive agent beating a cheap one there means something it does not mean elsewhere.

4. Failure modes — the real content

4.1 Citation hallucination, and its sharp rise

Fabricated references in the published literature, per a Lancet-published analysis: 1 in 2,828 papers in 2023 → 1 in 458 in 2025 → 1 in 277 during the first seven weeks of 2026. That is roughly a 10x increase in frequency in about three years. (STAT News, 2026-05-07, statnews.com/2026/05/07/lancet-study-finds-steep-rise-fraudulent-citations-academic-papers/)

The mechanism matters for defence design: a model has learned that a citation is a shape — author, year, journal, DOI. Under uncertainty it fills the slot rather than refusing, so invented DOIs are well-formed, invented authors have plausible names, invented journals follow naming conventions. A fabricated citation is therefore invisible to any check short of resolution against a real index. In a controlled bibliographic-retrieval test of 8 chatbots, fabrication rates spanned 0% to 38% depending on system. (arXiv:2505.18059)

Design consequence. Plausibility review cannot catch this — the failure is engineered to be plausible. Only resolution against an external index catches it. This single fact is the strongest argument for the citation gate in §6.

4.2 AI ideation NARROWS exploration — it does not widen it

The largest study of this to date analysed 219,655 valid AI-generated research ideas across 155 research areas in 12 scientific fields, comparing them to human papers in the same areas. (Tang & Yang, HKUST, arXiv:2605.27905v2, revised 2026-07-11.)

Measure	AI ideas	Human papers	Deficit
Exploration breadth	0.554	0.599	-7.5%
Exploration distance from seed literature	0.322	0.410	-21.5%
Coverage of next-year frontier keywords	28.5%	36.5%	-21.9%
Rated potential scientific impact	0.387	0.492	-21.3%

The decomposition explains why. Of AI-generated ideas, 90.4% introduce a new method absent from the seed papers, but only 10.5% pose a research question absent from the seed literature. AI novelty is method recombination; human novelty is problem selection. Scaling idea generation therefore amplifies homogenisation and burns experimental budget on locally-perturbed variants of what was already being asked.

This inverts the naive build plan. The intuition is to use AI to widen the hypothesis space. The measurement says AI contracts it toward the seed corpus. The defensible division of labour: humans select the problem, AI expands and organises the evidence. Any system that auto-generates research questions at scale is optimising the metric that measured worst.

4.3 Robin: a 4x numerical error inside a Nature result

Robin's analysis agent Finch reported that ripasudil increased RPE-cell phagocytosis 7.5-fold versus DMSO control. Human re-analysis of the same data returned 1.75-fold — a discrepancy of more than 4x, inside the flagship result of a system whose paper reached Nature. (arXiv:2505.13400; Nature s41586-026-10652-y.)

The direction is the alarming part. The error ran toward the desired conclusion, and the qualitative claim (ripasudil increases phagocytosis) survived re-analysis — so the error was invisible to conclusion-level review. Only recomputing the number caught it. Robin compressed an estimated 872-937 hours of human cognitive labour into under two hours; the compression is real and so is the error rate.

4.4 Kosmos: random-gene negative controls caught an outright false result

An independent evaluation ran Kosmos on three problems in radiation biology, scoring each against simple random-gene null models. Outcome: one well-supported discovery, one plausible but uncertain, one outright false. The false one — a proposed DDR/p53 relationship — showed Spearman ρ = -0.40, p = 0.76, i.e. weakly negatively correlated and statistically indistinguishable from randomly chosen five-gene scores. (arXiv:2511.13825.)

The most transferable finding in this brief. A 1-in-3 false-hypothesis rate in the best-instrumented commercial system, and the thing that caught it was not a better model, a bigger context or a review agent — it was a random-gene negative control, one of the cheapest instruments in statistics. An AI scientist without negative controls is a hypothesis generator with the error bars removed. Note the resonance with our own house law: a neg-control is only evidence if it can fail.

5. What a one-operator sovereign stack should build, ranked

Constraint filter: one operator, no wet lab, no frontier-scale training budget, 16GB VRAM. Rank is by (measured failure mode addressed) x (tractability solo). Note that items 1-4 are all verification, because that is where the measured failures are — and verification is the half of science that scales without a lab.

#	Build	Payoff / cost	Exit criterion
1	Citation resolution gate — no claim enters the corpus until every reference resolves against OpenAlex / Crossref / Semantic Scholar	Kills the fastest-growing measured failure (§4.1, ~10x in 3 years) / days, pure API work	A known-fabricated DOI with valid syntax is rejected; a real one passes; both proven in a gate
2	Negative-control harness — every hypothesis scored against permutation / random-feature nulls before it may be reported	This is precisely what caught Kosmos's false result (§4.4) / 1-2 weeks	A deliberately spurious hypothesis is caught by the null and refused; the harness must be able to fail
3	Independent numerical re-computation of every headline figure, by a separate path from the one that produced it	Directly targets the Robin 7.5x-vs-1.75x class (§4.3), which conclusion-level review cannot see / 1-2 weeks	A seeded 4x arithmetic divergence is detected and localised
4	Append-only negative-result ledger — refuted hypotheses recorded permanently, never deleted	Nobody else ships this; prevents re-proposal and is the substrate for measuring our own false-hypothesis rate / days	A previously refuted hypothesis is auto-flagged on re-proposal
5	Mutation testing on every gate (non-vacuity)	A gate that cannot fail banks an unearned GREEN; we have measured this in our own fleet / ongoing	Surviving-mutant count trends to zero on scored gates
6	Problem-selection discipline — human picks the question; AI expands evidence. Measure distance-from-seed on anything auto-generated	Counters the 10.5%-vs-90.4% narrowing (§4.2) / policy, ~free	Auto-generated ideas carry a measured seed-distance score
7	Local benchmark audit before trusting any leaderboard delta	25.7% of tasks are defective (Headline 2) / per-use	Any adopted claim of <10 points improvement is marked unmeasured

What NOT to build: a full end-to-end autonomous paper generator. Three independent lines of evidence converge. (1) Purpose-built scaffolding loses to a bare frontier LLM on end-to-end rediscovery (Headline 1). (2) The whole field sits at 17-27% with no system at the human bar (§3). (3) The generation half is exactly where the narrowing and hallucination failures live (§4.1, §4.2). The verification half is unclaimed, cheap, and where every measured failure actually is. Build the referee, not the author.

6. The sovereign differentiators — all three are governance, not scale

We cannot win on model scale, wet-lab access or compute. Every advantage below is a governance advantage: a rule about what is allowed to enter the corpus. That is an honest description, not a modest one — the measured failures in §4 are all governance failures, and governance is the one axis where a one-operator stack can actually be strictest.

Differentiator	Why it is a real edge
Verified citations — resolved against OpenAlex / Crossref / Semantic Scholar before a claim may enter	Fabrication is engineered to look plausible, so resolution is the only defence (§4.1). Most systems cite; few resolve; almost none gate admission on resolution.
Append-only negative-result ledger	Nobody in §2 ships this. The literature's central bias is that negative results vanish. An append-only refutation log is cheap for us, structurally unavailable to a publish-driven lab, and is what makes a measured false-hypothesis rate possible at all.
Mutation-proven non-vacuous gates	A gate that cannot fail is detected as such. Every system in §2 reports scores from harnesses whose ability to fail is unproven. We have already measured surviving mutants in our own fleet, which is why this is a claim we can support rather than a slogan.

Registry presence is not a capability — these are candidates to VERIFY, not things we have. Four of the five named organs appear in this session's own tool registry: nx_atlas_citeverify (and nx_atlas_citeverify2), nx_dr_elo, nx_dr_refute, nx_paperbench. nx_paper_gen does not appear at all — the registry carries nx_paper_forge, nx_paper_audit, nx_paper_panel and nx_papers_index instead, so the name in the build request is wrong or the organ was never registered. More importantly, we have previously established in this ecosystem that registry presence does NOT imply a built, working artifact — a documented case found 8 tools doc-paged and never built, including one prescribed as the remedy by an enforcement gate. None of these five is claimed as a working capability here. Each needs a build check and a non-vacuous gate before it may be cited as one.

Declared UNVERIFIED — do not treat as measured

The two-sweep merge did not happen. Both commissioned source transcripts were 0 bytes; all 17 task outputs in that directory were 0 bytes. No agreement/disagreement analysis was possible. This is a single sweep.
Both Nature papers are paywalled and were not read. Co-Scientist (s41586-026-10644-y) redirected to an auth gate; Robin (s41586-026-10652-y) was not fetched. Every Nature-attributed number here comes from the publisher blog, the preprint, or secondary coverage — not the paper of record.
Exact Nature publication date for Co-Scientist. Secondary sources give 2026-05-20 and 2026-05-30. Stated as “May 2026”.
ResearchClawBench scoring scale above 50. One source states >50 = matching the original paper and >70 = surpassing it; the paper fetch stated >50 suggests new-discovery capability. Conflicting; unresolved. The 50 = paper-parity anchor is consistent across both.
AstaBench best overall agent score. Not obtained — the abstract and landing page do not carry it and the leaderboard was not fetched. Only the suite's scale and its authors' conclusion are cited.
LitQA2 exact best score. “Superhuman precision, human-level accuracy” is the authors' own framing for PaperQA2; no independent percentage obtained.
MLE-bench 48.5% and CORE-Bench 26.7% come from leaderboard aggregators, not primary papers. Split and date conventions are unconfirmed; the MLE-bench figure is the Lite split.
Sakana v2 cost of ~$50-200 per research cycle is from a secondary blog. Only v1's ~$15/paper is well attested. Whether Sakana withdrew the accepted workshop paper before publication was not verified.
Frontier hallucination range of 3.1-19.1% is from a statistics-aggregator site, not peer-reviewed; treat as indicative only. The Lancet/STAT trend figures and the arXiv:2505.18059 spread are the defensible numbers.
Kosmos's “7 discoveries, 4 novel” is company-reported and not independently audited. The independent evaluation (arXiv:2511.13825) covered 3 problems in one domain — a 1-in-3 false rate on n=3 is a warning, not a precise rate.
Microsoft's Majorana-2 and coolant claims are vendor-reported with no external replication found.
2026 open-source entrants (EvoScientist, URSA, MARVEL, Agent Laboratory) were identified from a single curated listing. Maintenance status, licences and working condition are unverified — except EvoScientist, which carries a third-party ResearchClawBench score.
The 18-fold KIRA6 figure and the 91% vorinostat figure are reported via secondary coverage of the Nature paper; the cell lines and assay endpoints are stated as found and were not checked against the primary methods.
Our own five organs are unverified as built (§6). No build check, byte count or gate run was performed on any of them in this sweep.

Method. Single sweep, 2026-08-01, ~18 web calls. Primary sources fetched directly for both headline claims (arXiv:2606.07591, arXiv:2605.26079), the narrowing study (arXiv:2605.27905v2), PaperBench (arXiv:2504.01848), AstaBench (arXiv:2510.21652) and the Co-Scientist architecture (Google Research blog). Everything else is search-summary level and labelled where it matters. What would change the conclusions: reading the two Nature papers in full; an AstaBench leaderboard pull with cost controls; and any replication of the ResearchClawBench result on a second end-to-end suite — Headline 1 currently rests on one benchmark, and one benchmark is not a field.