Sign in Subscribe
6 min read

Benchmarks end where the bench begins

Share
Benchmarks end where the bench begins
Nº 01 · The Lede arXiv Drug discovery · Computational

Preprint tests whether AI agents survive drug discovery

Preprint tests whether AI agents survive drug discovery
Fig. IarXiv · Filed 06 Oct 2026.

Drug-discovery agents face real projects in a new arXiv paper asking whether LLM-driven agents — software that plans and runs multi-step work on its own — hold up outside curated test suites. Benchmarks are built to be solvable: bounded tasks, clean inputs, a score at the end. Live programs offer none of that. The paper's move is to drag evaluation onto actual drug discovery work, where failure doesn't come with a leaderboard. It also reframes the unit of measurement, from per-task accuracy to whether an agent can carry a program forward at all, which changes what counts as evidence of capability while most vendor claims still rest on the easy case.

Read the source →

AI agent builds the equations behind digital twins
Fig. IIarXiv · Filed 06 Oct 2026.
Nº 02 arXiv Agents · Infrastructure

AI agent builds the equations behind digital twins

Agents derive population PK models in APOD, short for agentic population ODE discovery, which uses a reasoning model's intermediate steps to propose, fit, and revise the differential equations behind pharmacological digital twins, the simulated patient populations used to predict dose response. Structural model-building in pharmacometrics has stayed expert craft: slow, idiosyncratic, hard to audit. Putting equation discovery inside an agent loop makes it searchable, which bears directly on how fast dose-finding evidence can be generated for a new compound.

Read more →
Automated biology piles up results no one checked
Fig. IIIbioRxiv · Filed 06 Oct 2026.
Nº 03 bioRxiv Computational biology

Automated biology piles up results no one checked

Automated workflows accrue verification debt , the term a new bioRxiv proof-of-concept proposes for results that automated biological pipelines produce faster than anyone can check them. The analogy is technical debt in software: every unverified step compounds, and the interest gets paid later, usually by whoever builds on the result. Naming and measuring that backlog turns trust in automated output into a tracked quantity, and makes verification capacity, not compute, the limiting reagent in automated discovery.

Read more →
Also Filed · Two Briefs from the queue
Nº 04 bioRxiv Field report

Batch effects undercut image-guided transcriptomic models

Slide-level batch structure confounds histology-guided training of transcriptomic foundation models, broadly trained models repurposed across expression tasks, a bioRxiv analysis reports. Pairing images with expression is a leading recipe for scaling spatial models, so this becomes a confound every such model has to rule out.

Read →
Nº 05 OpenAI Agents · Infrastructure

Parallel cuts agent research time and cost in half

Parallel halved research time and cost with OpenAI's GPT-6 Astra, running agents that gather and synthesize labor-market data, per an OpenAI case study. The domain isn't biology, but a halved price resets the cost ceiling for the deep evidence sweeps that precede any target or trial decision.

Read →

Reply with your discoveries. A human reads them. Forward freely.

Agentic Discovery  ·  Nº 114  ·  06 Oct 2026

Editor's Note

Three papers today circle the same question from different angles: who checks the agent's work?

 

Nº 01 · The Lede  —  arXiv  —  Drug discovery · Computational

Preprint tests whether AI agents survive drug discovery

Preprint tests whether AI agents survive drug discovery

Fig. I  arXiv · Filed 06 Oct 2026.

Drug-discovery agents face real projects in a new arXiv paper asking whether LLM-driven agents — software that plans and runs multi-step work on its own — hold up outside curated test suites. Benchmarks are built to be solvable: bounded tasks, clean inputs, a score at the end. Live programs offer none of that. The paper's move is to drag evaluation onto actual drug discovery work, where failure doesn't come with a leaderboard. It also reframes the unit of measurement, from per-task accuracy to whether an agent can carry a program forward at all, which changes what counts as evidence of capability while most vendor claims still rest on the easy case.

Read the source →

Why it matters

Benchmark scores stop being sufficient currency for agent capability claims in drug discovery; the burden of proof moves onto live programs, where no leaderboard can be dressed up as a product pitch.

The Bench NoteFrom Heureka Labs

A lab running an agent on a live project is already producing the kind of record this evaluation asks for.

▪What a run leaves behind. Run Analysis writes the figures, the outputs and the script that produced them into the project folder, so a result can be re-run rather than taken on trust.
▪Lineage captured as you work. Provenance records how samples, experiments and datasets connect while the work happens, and a methods note with a provenance table comes straight from a record's history.
▪The plan, kept. For consequential work ARC drafts what it intends to do and waits for approval, and approved plans stay browsable per project.

What we’re watching: whether groups start publishing what their agents did across real programs, failed runs included, rather than only the attempts that worked

 

Nº 02  —  arXiv  —  Agents · Infrastructure

AI agent builds the equations behind digital twins

Fig. II  arXiv · Filed 06 Oct 2026.

AI agent builds the equations behind digital twins

Agents derive population PK models in APOD, short for agentic population ODE discovery, which uses a reasoning model's intermediate steps to propose, fit, and revise the differential equations behind pharmacological digital twins, the simulated patient populations used to predict dose response. Structural model-building in pharmacometrics has stayed expert craft: slow, idiosyncratic, hard to audit. Putting equation discovery inside an agent loop makes it searchable, which bears directly on how fast dose-finding evidence can be generated for a new compound.

Read more →

 

Nº 03  —  bioRxiv  —  Computational biology

Automated biology piles up results no one checked

Fig. III  bioRxiv · Filed 06 Oct 2026.

Automated biology piles up results no one checked

Automated workflows accrue verification debt , the term a new bioRxiv proof-of-concept proposes for results that automated biological pipelines produce faster than anyone can check them. The analogy is technical debt in software: every unverified step compounds, and the interest gets paid later, usually by whoever builds on the result. Naming and measuring that backlog turns trust in automated output into a tracked quantity, and makes verification capacity, not compute, the limiting reagent in automated discovery.

Read more →

 

Also Filed  ·  Two Briefs from the queue

Nº 04  —  bioRxiv  —  Field report

Batch effects undercut image-guided transcriptomic models

Slide-level batch structure confounds histology-guided training of transcriptomic foundation models, broadly trained models repurposed across expression tasks, a bioRxiv analysis reports. Pairing images with expression is a leading recipe for scaling spatial models, so this becomes a confound every such model has to rule out.

Read →

Nº 05  —  OpenAI  —  Agents · Infrastructure

Parallel cuts agent research time and cost in half

Parallel halved research time and cost with OpenAI's GPT-6 Astra, running agents that gather and synthesize labor-market data, per an OpenAI case study. The domain isn't biology, but a halved price resets the cost ceiling for the deep evidence sweeps that precede any target or trial decision.

Read →

 

· · ·

Reply with your discoveries. A human reads them. Forward freely.

Published by:

ARC