Weekly Arc: Analysts followed the agent 86% of the time
Last week's issue described checks moving earlier in the pipeline: a monitor that flags research-integrity violations while an agent is still running, a gate that blocks a model update before it reaches the clinic. The question under those stories was when a check should fire. This week asks a harder one. Every check compares something against a reference, and the reference is almost never reported. A benchmark score is compared against an answer key somebody chose. A reviewed workflow is compared against a person's judgment. An automated lab steers by a model of how noisy its own measurements are. Four results this week showed those references moving.
A randomized trial on the human in the loop
Anthropic ran a trial in which data analysts worked alongside an AI coding agent, and the agent was assigned at random to favor one of the available analytical options. The analysts picked whichever option the agent favored 86% of the time. Because the preference was assigned by coin flip rather than earned by being right, the agreement cannot be explained by the agent having the better answer. The analysts moved with it.
That result lands on the design of nearly every safeguard this column has tracked since the summer. Post-hoc audits, release gates, and in-run monitors all end at a person who reads what the system did and decides whether it is acceptable. The person is treated as a fixed reference, independent of the system and able to overrule it. The trial measures that assumption in conditions that favor the human, since these were trained analysts doing familiar work on code they could read line by line. Biology offers less to read. A scientist reviewing an agent's spatial transcriptomics analysis that names ICAM-1 as a fibrosis target has far fewer ways to inspect the intermediate steps than an analyst reviewing a script.
The answer key is a choice, and the score follows it
Two other results this week took the same shape elsewhere in the pipeline. An arXiv study of a breast-lesion system that looks up supporting evidence before answering found that its retrieval scores shift depending on which ground truth the graders used. Same system, same cases, different answer key, different number. The authors propose grading that accounts for what evidence was actually available, which amounts to conceding that the score is uninterpretable without knowing what it was scored against.
The week's most-read story made the point inside the lab. A bioRxiv preprint on self-driving laboratories argues the measurement model should be fixed before anyone tunes the optimizer. A self-driving lab pairs automated experiments with an optimizer, the algorithm that reads each result and picks the next experiment to run. That algorithm needs an assumption about how noisy each readout is. Get the noise assumption wrong on living cells and the system does not fail loudly. It keeps proposing experiments with confidence, in the wrong direction, and nothing in the loop reports that the thing it is steering by is off.
Grading the decision instead of the result
The response taking shape is to stop grading the final answer and start grading the path to it. OpenGray scores radiotherapy-planning agents on the choices they make while building a treatment plan rather than only on the finished plan. M4Bench tests whether agents pulling data from electronic health records follow the written procedure they were given. A third preprint compares several language models across complete transcriptomic analysis workflows instead of single questions.
The advantage of grading the path is worth stating plainly. A final answer must be compared against a key somebody chose, and arguments about the key are unresolvable from the outside. A recorded sequence of decisions can be re-read by anyone, and disagreement about it is specific: which step, which call, which alternative was skipped.
Two items push that further by making the run itself repeatable by outsiders. A bioRxiv preprint sets out agent-run science on public infrastructure, so the compute, data and steps are available to a second party rather than summarized in a methods section. OpenAI published model results on the Kakeya conjecture and related problems alongside instructions for academic researchers to verify them. Neither is an audit. Both reduce the audit to something a stranger can carry out without asking permission.
What to ask of any claim that something was checked
The practical version of this week is a question to put to any system described as verified. What was the check measured against, and could someone else reconstruct that reference? A reviewer's judgment fails the test when the reviewer reads the agent's recommendation first. An answer key fails it when the graders picked the key. A noise model fails it when nothing downstream would notice the model being wrong.
The 86% figure was measured on analysts reading code, the most inspectable material there is. Nobody has measured it on scientists reading agent-written biology, where the intermediate steps are harder to see and the result cannot be run to check whether it compiles. Until someone runs that study, a human reviewing the output is a design intention rather than a safeguard, and the trials that would turn it into one are the ones to watch for.
- The infrastructure nobody owns: Anthropic started drug discovery programs of its own, including rare childhood muscular dystrophy, added 20 partner institutions to its AI for Science program, and gave its Agent Skills format to the Agentic AI Foundation in the same week, so the open question is whether donating the file format changes anything about owning the programs that run on it.
- Agents as bench scientists: the output is now the nomination itself, with an agent naming ICAM-1 as a lung fibrosis target, Agent K assigning genes to biosynthetic pathways, and agents deriving bioprocess kinetic models unprompted — watch for the first agent-nominated target that enters a funded program under its own name.
- The open stack and its liabilities: Anthropic set out why it tests its own models for biological misuse while an arXiv preprint proposed clinical agents that improve themselves with no central hub, leaving the same unanswered question from two directions: who inspects the party doing the gatekeeping, and where does a check sit when there is no hub to put it in.
Reply with what you're seeing. A human reads them. Forward freely.
|