Weekly Arc: Pretraining fails the single-cell audit

Last week's issue traced two answers to the same problem of who checks an agent's work: formal proof, where a machine supplies the verdict, and provenance, where a record shows how a result was produced. Both assume the thing under review is a particular result. This week the evaluation layer did something it had not done before in this arc. A bioRxiv preprint reported that simple models outperform single-cell foundation models on downstream benchmarks, a verdict that applies to a class of models rather than to one analysis. Alongside it, OpenAI released a test of routine wet-lab competence, and an agent arrived that checks whether a dataset meets the assumptions of the method about to be run on it.
An audit that can come back negative
Foundation models in single-cell biology follow the pattern that made language models useful. Train one large model on tens of millions of cells, then reuse it for whatever comes next: labeling cell types, correcting batch effects, predicting what a gene knockout does. The appeal is that the expensive step happens once. The bioRxiv preprint ran the comparison the whole pattern depends on and found that small, task-specific models beat the pretrained ones across downstream tests. The baselines here are the kind a graduate student fits in an afternoon.
This is not the first such report. A test of the same model class on aging-specific biology, covered here in late September, found the headline scores did not survive contact with a harder question. What makes this week's version matter is not novelty but what it costs to accept. Nearly every check this arc has tracked grades a system against a target: how close to the right answer, how well the path was documented, how much the score moves when the graders change. Those checks return a number. A comparison against a cheap baseline can return something harder, which is that an approach is not worth its price. A check that cannot produce that answer is a scoreboard rather than an audit.
The cost is concrete. Public money is already committed to generating biological data at the scale needed to train cell models of exactly this kind, through the Biohub, Google and federal effort announced last month. Labs are building pipelines on pretrained single-cell representations now.
The check moves in front of the analysis
DE-Rigor-Agent does something small and worth copying. Before running differential expression on bulk RNA-seq, it tests whether the dataset actually satisfies the conditions the chosen statistical model requires, then adapts the analysis when it does not. Every statistical method carries assumptions about the data it is handed, and verifying them is tedious enough that in practice it is often skipped. The agent does the part a careful statistician would do and most pipelines skip.
The placement is what makes it interesting. Last month RIDAR caught research-integrity violations while agents were still running, and BRIDGE blocked clinical model updates that only looked like improvements. Both inspect work in progress. This one inspects the premise before any work happens. RIVER applies the same discipline to evidence, sending several agents to search in parallel for enzyme-reaction pairings and requiring each proposed pairing to be backed by a retrieved source instead of asserted. Public enzyme annotation is thin, which is precisely the condition under which a confident model invents plausible pairs.
A review published this week explains why front-end checks earn their keep. General-purpose models have specific gaps in biology, and an agent that chains ten tool calls carries the error from each one forward. By the end of the chain, inspecting the final output cannot tell you which step went wrong. When checking the answer cannot localize the fault, the useful place to intervene moves to the inputs and the assumptions.
Who writes the floor test
BioBasicsBench is 100 tasks on everyday wet-lab science, released by OpenAI to score how well agents handle routine work. Set it against the tests of the past month. TrialDesignBench grades models on phase I trial design against established clinical knowledge. NPCBench scores an entire nasopharyngeal carcinoma care pathway as a sequence of decisions a trainee would face. BioXArena-Pro grades agents doing biomedical machine learning at research level, Pyrite builds docking-pose-ranking agents and then scores them, and a protein-engineering benchmark posted to bioRxiv measures how performance decays as tasks get deeper. Those test the ceiling. BioBasicsBench tests the floor.
That direction matters for adoption. A ceiling score says what is possible when a system is driven by people who know how to drive it. A floor score says what can be handed off and left alone, which is the number that actually governs whether a lab delegates anything. Scientists do not adopt a tool because it once solved a hard problem. They adopt it when the routine case stops needing supervision.
The authorship matters too. OpenAI argued last month that existing benchmarks have been gamed hard enough to stop measuring progress, and then published a replacement. Now it defines basic laboratory competence. There is nothing improper in a vendor building a test; tests exist because somebody builds them. But the floor test sets the threshold for what is safe to hand off, and the company with the most to gain from handoff is holding the pen.
What acting on it would look like
Which returns the arc to the question it keeps asking. A check binds only when someone is obliged to respond to it, and the single-cell result is a usable test case for whether this field's measurement layer binds at all. If the finding holds, the honest reading is that substantial compute, data collection and institutional commitment went into an approach a simple model matches. Replication is the first thing to watch, because one preprint is a claim and the audit only exists once a second group re-runs the comparison and gets the same answer. After that, watch for the response that costs something: a grant redirected, a methods section rewritten, a pipeline rebuilt on the cheaper model. If none of those appear, the field has built an evaluation layer that reports and does not oblige.
- Agents as bench scientists: SciMARS now lets agents revise their own methods between rounds, one spatial transcriptomics agent calls cell types before drawing any cell boundaries, and ANTIOCH-VS runs screening from protein sequence to ranked hits. Watch for the first paper whose methods section cannot state which method was used, because the agent changed it mid-run.
- The infrastructure nobody owns: Stanford and MIT labs running genome-wide studies on Claude drew the week's heaviest reader traffic, while an anti-IgE antibody designed with Claude was billed best-in-class and three Claude-designed electronics prototypes worked without circuit fixes. The open question is what a university group does when the model its published pipeline depends on is deprecated.
- The open stack and its liabilities: quiet this week, and now dormant. The question left standing from the adversarial tests of synthesis-planning agents two weeks ago is whether any party other than the vendor ever re-runs a vendor's safety evaluation.
Reply with what you're seeing. A human reads them. Forward freely.
|