Sign in Subscribe
7 min read

When the same query returns three answers

Share
When the same query returns three answers
AGENTIC ARC
Nº IV  ·  week of 08 Jun 2026  ·  from Agentic Discovery
When the same query returns three answers

Two weeks ago this column argued the agent stack had matured faster than the science it produced. Last week the arc turned toward the protocol layer underneath — LAP, MCP-native planners, instrument control — on the assumption that wire protocols would become the durable, reproducible substrate. This week complicates that assumption in the most direct way possible: Anthropic published a research note arguing biology is the hardest agentic domain frontier models have encountered, paired with a demonstration on NCBI retrieval where the same query returned 106 viral sequences, then 15, then 5, across three runs. A protocol assumes the thing it carries is reproducible. On the evidence of the past week, it isn't.

What the NCBI result actually shows

The retrieval task in question is the kind a graduate student would consider closed: query a public database, return matching sequences. There is a correct answer. Three runs of a frontier agent produced three answers — none of them, on inspection, defensible as the right one. This is not a hallucination in the familiar sense. The agent constructed three different plans, called the database with three different query shapes, and reported three different result sets, each internally consistent. The nondeterminism lives in the trajectory, not the model output token-by-token. Which is the harder problem. A model that gives different answers can be averaged, ensembled, or sampled at low temperature. A planner that takes different paths through the same tool surface cannot be cleaned up after the fact — the action history is the answer, and the action history changed.

Three diagnoses, one failure

What makes this week unusual is that three separate proposals for fixing the problem surfaced in parallel, each locating the bottleneck somewhere different. EurekAgent argues the bottleneck is the environment — give the agent the right tools, schemas, and feedback channels, and the planner stabilizes. Crownlands open-sourced a 4M-cell tissue atlas on the explicit thesis that frontier biology AI is data-starved at the tissue level, not model-starved. Anthropic's own framing points at the model: biology is harder than coding because the action space is wider and the verification signal is weaker. Three diagnoses for the same observation — and each implies a different industry. The environment diagnosis funds protocol work and tool standardization. The data diagnosis funds proprietary atlases. The model diagnosis funds frontier training runs with chemistry and biology experts in the loop, which is what Anthropic also announced for chemistry this week.

What the protocol layer cannot fix on its own

The arc came into this week leaning on a tidy story: standardize the wire format between agent and instrument, and the trajectory becomes auditable, comparable, reproducible. The protocol work was real and remains real. But a protocol describes what crossed the wire; it does not constrain what the planner decided to send. If three runs of the same task generate three different query shapes, the audit log faithfully records three different sessions. The protocol does its job. The science still doesn't reproduce. This is the part worth naming: the protocol layer makes agent behavior observable, not deterministic. Observability is a precondition for accountability, not a substitute for it. A separate mechanism has to constrain the planner — either by narrowing the action space (the environment diagnosis), by training the model on enough domain examples to converge on stable plans (the model diagnosis), or by grading the trajectory itself rather than the final answer (the audit diagnosis, which ABC-Bench and EpiBench both moved toward this week).

Where this leaves the arc

The cleanest finding of the week is also the most uncomfortable: the field has been building protocols, benchmarks, and atlases on the assumption of an underlying property — trajectory determinism — that frontier agents do not yet have on routine biological retrieval. The infrastructure is not wrong to build. It is, however, ahead of the substrate. Which makes the next question concrete and testable: of the three diagnoses now on the table, which one moves the NCBI number first? A repeat of the same query, by the same agent, on the same database, converging on the same answer across three runs would be the smallest possible signal that one of these diagnoses is correct. Until then, the agent that ran the analysis is not yet the accountable unit. The trajectory is.

Still tracking

Reply with what you're seeing. A human reads them. Forward freely.

AGENTIC ARC

Nº IV  ·  week of 08 Jun 2026  ·  from Agentic Discovery

When the same query returns three answers

Two weeks ago this column argued the agent stack had matured faster than the science it produced. Last week the arc turned toward the protocol layer underneath — LAP, MCP-native planners, instrument control — on the assumption that wire protocols would become the durable, reproducible substrate. This week complicates that assumption in the most direct way possible: Anthropic published a research note arguing biology is the hardest agentic domain frontier models have encountered, paired with a demonstration on NCBI retrieval where the same query returned 106 viral sequences, then 15, then 5, across three runs. A protocol assumes the thing it carries is reproducible. On the evidence of the past week, it isn't.

 

What the NCBI result actually shows

The retrieval task in question is the kind a graduate student would consider closed: query a public database, return matching sequences. There is a correct answer. Three runs of a frontier agent produced three answers — none of them, on inspection, defensible as the right one. This is not a hallucination in the familiar sense. The agent constructed three different plans, called the database with three different query shapes, and reported three different result sets, each internally consistent. The nondeterminism lives in the trajectory, not the model output token-by-token. Which is the harder problem. A model that gives different answers can be averaged, ensembled, or sampled at low temperature. A planner that takes different paths through the same tool surface cannot be cleaned up after the fact — the action history is the answer, and the action history changed.

 

Three diagnoses, one failure

What makes this week unusual is that three separate proposals for fixing the problem surfaced in parallel, each locating the bottleneck somewhere different. EurekAgent argues the bottleneck is the environment — give the agent the right tools, schemas, and feedback channels, and the planner stabilizes. Crownlands open-sourced a 4M-cell tissue atlas on the explicit thesis that frontier biology AI is data-starved at the tissue level, not model-starved. Anthropic's own framing points at the model: biology is harder than coding because the action space is wider and the verification signal is weaker. Three diagnoses for the same observation — and each implies a different industry. The environment diagnosis funds protocol work and tool standardization. The data diagnosis funds proprietary atlases. The model diagnosis funds frontier training runs with chemistry and biology experts in the loop, which is what Anthropic also announced for chemistry this week.

 

What the protocol layer cannot fix on its own

The arc came into this week leaning on a tidy story: standardize the wire format between agent and instrument, and the trajectory becomes auditable, comparable, reproducible. The protocol work was real and remains real. But a protocol describes what crossed the wire; it does not constrain what the planner decided to send. If three runs of the same task generate three different query shapes, the audit log faithfully records three different sessions. The protocol does its job. The science still doesn't reproduce. This is the part worth naming: the protocol layer makes agent behavior observable, not deterministic. Observability is a precondition for accountability, not a substitute for it. A separate mechanism has to constrain the planner — either by narrowing the action space (the environment diagnosis), by training the model on enough domain examples to converge on stable plans (the model diagnosis), or by grading the trajectory itself rather than the final answer (the audit diagnosis, which ABC-Bench and EpiBench both moved toward this week).

 

Where this leaves the arc

The cleanest finding of the week is also the most uncomfortable: the field has been building protocols, benchmarks, and atlases on the assumption of an underlying property — trajectory determinism — that frontier agents do not yet have on routine biological retrieval. The infrastructure is not wrong to build. It is, however, ahead of the substrate. Which makes the next question concrete and testable: of the three diagnoses now on the table, which one moves the NCBI number first? A repeat of the same query, by the same agent, on the same database, converging on the same answer across three runs would be the smallest possible signal that one of these diagnoses is correct. Until then, the agent that ran the analysis is not yet the accountable unit. The trajectory is.

 

Still tracking

 

· · ·

Reply with what you're seeing. A human reads them. Forward freely.

Published by:

ARC